owl shift 10/13/99

John Shields (jshields@klak)
Wed, 13 Oct 1999 07:05:37 -0500

Shift Summary for owl shift, 10/13/99

On Shift: J. Shields, J. Whitmore

Hours of beam available at all intensities: 3.25/7.0
Hours of beam available at nominal intensity: 3.2
Hours of E799 data (special runs not included): 3.0
Hours of special runs (laser scans, muon runs...): 0.0

Accounting of lost time:

new run/drt_fresh_start, tevatron down due to power supply failure
========================================================================
Runs ended this shift:
Run 14830 - 7.6M events - BEAM_799

Tapes brought to FCC this shift:
(including tapes for online split)

none
========================================================================
Accesses during shift:

none

========================================================================
Overview of shift:

E799 datataking at ~9e12, a small problem with L3 acceptance,
more E799 datataking, and then beam dies due to a power supply failure.

========================================================================
Description of Shift:

run spill time comment
------------------------------------------------------------------------
14830 78 00:45 DAQ Dead 9.1 sec. Note that beam intensity was
only 9.07e12. Ok in next spill.

01:10 Level 3 kumac shows processor # 2 on ktev4 is
taking much less CPU than other 6 processors.
Verdict is to drt_fresh_start at end of run

105 01:20 DAQ dead 3.3 sec. This time SEM only 8.7e12!
Again, no problem on the following spill.

131 01:55 ktevana errors--unknown crate stream 5. kfrended

140 02:08 CSI chan 455 has 31.9% RMS. Ok at next spill.

02:10 On-split request: tape UPF017 on ktev2.

150 02:20 Sydney alarm warns us to check the L3 accept fraction
Examination of histograms reveals a level 3
problem on ktev4. Decide to end run & see if a
drt_fresh_start will solve the problem

02:27 end run 14830 & drt_fresh_start

14831 1 02:40 begin run 14831 & almost immediately we are treated
to an alarm saying that TRD3 cathode currents are
out of limits. SEM=9.2e12

3 02:42 TRD 3 trips off. SEM=9.3e12. Sasha paged.

7 02:50 level 3 processes look normal for this run

19 03:06 Sasha resets TRD3 from home.

21 03:09 Alarm: scalar live-time out of limits. Trigger
hung because crate 53 Azlat busy. Kfrended.

24 03:16 Beam goes away. MCR message: " Mubend quench,
Investigating. No estimate for beam".

04:14 "updated" message: " Mubend caused by Musep
power-supply failing. Consulting experts."

04:20 Rebooted main alarm screen after fixing
terminal setup parameters

04:44 another *extremely* useful update: "Musep
power supply failed. experts enroute."
(TRANSLATION: "after half an hour of whining &
dragging their feet, the experts reluctantly
agree to get up & go fix the tev.")

06:13 Yet *another* thoroughly unhelpful message update:
"Mubends quench caused by the Musep power supply
failing. Experts investigating."
(TRANSLATION: "bleary-eyed and irrate experts
finally stumble into work & stare blankly at
smoking Musep power supply.")

06:36 AT LAST! MCR is finally ready to divulge some
useful intelligence: "Tunnel access in progress
to investigate Musep problem. Beam estimate
is 30-45 minutes."

========================================================================
Follow up Notes on Previous Shift

========================================================================
Items to Follow up on Next Shift

argon ethane west needs a bottle change.