Shift summary for Monday Aug 23 EVENING

Richard Kessler (rkessler@fnal.gov)
Tue, 24 Aug 1999 00:06:29 -0500 (CDT)

Shift Summary for EVENING shift, Monday Aug 23, 1999

On Shift: Rick Kessler and Rick Ford

Hours of beam delivered at all intensities: 8.0 hrs
Hours of beam delivered at nominal intensity: 8.0
Hours of E832 data (special runs not included): 6.0
Hours of special runs (laser scans, muon runs...): 0.0

Accounting of lost time:
1hr 50 minutes: Debugging crate 53, swapping AZLAT in slot 8.
10 minutes : kfrending

========================================================================
Runs ended this shift:
Run 14305 BEAM_832 - 2350 vac 2pi0
Run 14306 BEAM_832 - 8481 vac 2pi0 [ended with cr53 problems]
Run 14307 BEAM_832 - debugging cr53; it's all garbage.
Run 14308 BEAM_832 -
========================================================================
Accesses during shift:

========================================================================
Overview of shift:
RC10 trigger source was fixed by unplugging and plugging in cable
at patch panel.

Everything was fine until 18:15 when cr53 gave incessant ktevana
errors despite every known trick known to the ktev crew. We then
removed everything except DYC and CTI, which worked. Then we added
one module at a time until the trigger mask AZLAT was found to be
the culprit. The debugging was slowed a bit when the pipeline gave
ktevana errors, and we had to reboot two crates before the pipeline
could be re-initialized. The 2-hour loss of beam coincided with
a thunder storm during which some beam-line magnets tripped and our
beam stop went in by accident; this was only a 5-10 minute loss.
The 2 hour loss also means that we do not have to change tapes
during this shift !

A few hours later we noticed that B14 [DAL 3PI0] trigger is not
processed in L3 for run 14308. The hardware L2 mask is fine as
checked on the scalers and by the hardware trigger mask via scope.
It seems that the new AZLAT has added Bits 3,15,16 when B14 bit
is set [perhaps an internal short ??] ...
this makes B14 look like a calibration trigger in L3 and is therefore
not processed. I overlaid the beam and calib trigger masks for runs
14308/14306 [XXI.80] and all triggers are OK except B14 in run 14308.
Decide to wait 11 hours for the TeV shutdown to investigate further
[since B14 is not critical].

Mean E/p=0.991 ... time to update CsI Q/E ??

Inspired by Vivian and Taku's brilliant on-line plot surveys,
RF carefully inspects all of the online plots and finds many
discrepancies that nobody cares about. Perhaps an updated
set of plots at high intensity would be useful.


========================================================================
Description of Shift:

run spill time comment
------------------------------------------------------------------------
14305 15:30 Inherit BEAM_832 run from day shift

24 Beam down for BOOSTER access.

40 16:00 RC10 trigger source working again after
unplugging and plugging cables at L1 patch panel.
There was no beam during this exercise.

16:01 Rick Ford arrives delinquent.

48 16:10 Decided to stop run to make clean break with RC10
and since there is no beam.

16:11 Crappy beam returns, but some magnets are off ?
MCR contacted.
[By the way, "crappy" is not a valid word according
to the kpasa spell-checker. However, the Merriam
Webster Dictionary defines "crappy" as
"inferior in quality".
Crappier and crappiest are also valid words.

14306 16:12 Init BEAM_832
01 16:15 START run with no ktevana errors !!!!

16:20 Lost monitor screen 6; use fix_disp.
Need to fresh_start next run.

20 16:30 Large bag of Sun Chips opened.

35 17:00 Gas checklist is done.

88 18:15 ktevana errors. After kfrend, DYC FULL cr 53 hangs
up after few thousand triggers. Massaging the REQ
line and kfrending does not help, so stop run and
re-init so that FERA scalers are initialized.

14307 18:25 Begin new run for debugging.
18:25 Doesn't work ... crate 53 still gives ktevana errors.
Begin tedious search for bad module in cr53.

19:00 Pipeline errors. Cannot re-init before rebooting
crates 5 and 15.

19:40 Finally identify bad AZLAT for trigger mask;
RickK and J.Hamm do the swap; no ktevana errors.
Drt_Fresh_start.

20:00 Large bag of Sun Chips is empty.

14308 20:00 Init BEAM_832 after drt_fresh_start.
20:07 Start. ktevana errors ==> kfrend.
Now no ktevana errors and no events !!!
Oops; L2->AZLAT was not plugged back in.
We plug it in and it works !!
04 20:12 Real/live data starts at spill 4.

13 20:25 Cr 53 DYC FULL gets stuck again! AH MERDE !!!!!
kfrend fixes it.

20:35 Rick #1 prepares/eats dinner from Family Foods.
20:55 Rick #2 prepares/eats dinner from Chopsticks.
21:01 Sydney #1 prepares/eats dinner from Chopsticks.
21:16 Rick #1 eats leftover scraps from Chopsticks.
23:20 Rick #2 complains about really dirty coffee pot.

174 24:00 End of shift

========================================================================
Follow up Notes on Previous Shift

========================================================================
Items to Follow up on Next Shift

- need to debug B14 trigger bit in AZLAT