Shift summary, Evening 12/25/99

Tony Barker (tonyb@cuhep.Colorado.EDU)
Sun, 26 Dec 1999 00:09:48 -0700 (MST)

Shift Summary for Evening Shift, 25 December 1999

On Shift: J. Hamm, T. Barker

Helping with numerous problems: Arun, JasonL, SashaL, Valeri

Hours of beam delivered at all intensities: 8.5
Hours of beam delivered at nominal intensity: 8.5
Hours of E799 data (special runs not included): 2.8
Hours of special runs (laser scans, muon runs...): 0

Accounting of lost time: ET overtemp trip took about an hour to reset,
including access.
Trip was caused by a failure of the water
chiller, which led to FASTBUS racks being
turned off when hall temp reached 90 degF.
Several hours were then lost trying to get
the FASTBUS racks turned back on, which we
were NOT able to do. This is followed by
readout problems which cause further lost time.

========================================================================
Runs this shift:

Run 15289 - 7.0M events - ONGOING - BEAM_799
Run 15290 - JUNK
Run 15291 - Test trigger w/out fruit

Tapes brought to FCC this shift: none

========================================================================
Accesses during shift: One, to reset ET power supplies.

========================================================================
Overview of shift: Water chiller failures cause ET power trips.
These were followed by a Liebert alarm for
high temperature in the control room (85 to 90
degrees). All the FASTBUS racks then had
to be powered off to avoid building meltdown.
Duty mechanic called by MCR at 19:40 to see
about resetting the chiller, which requires
access to exposed 480V power circuits.
Duty mechanic shows up and resets the chiller.

Next we attempt to turn the FASTBUS racks
back on. This does not work. The main circuit
breakers on RR176 and RR177 WILL NOT RESET no
matter what we try; they immediately trip.
We call Jason LaDue, who makes various
suggestions, none of which work. After over
an hour of trying, these two racks still will
not turn on.

Eventually we find that the rack-monitor-disable
switch on the chamber-3 rack works BACKWARDS
from the switches for chambers 1 and 4. Once
this is determined, we are able to get the
chamber 3 rack turned back on.

However, repeated attempt trying the switch both
ways meet with no success for rack 2. Nobody
can get the rack2 main breaker to turn on.

Eventually we decide to run without the
chamber-2 bananas. This means that STT and
YTF cannot be used, and removing the chamber 2
hitcounting requirements from everywhere they
appear.

At the end of the shift, we reconnect the
readout for the chamber 1,3,4 fruits. Then the
readout does not work and we get KTEVANA errors,
coming now from the 3Y Kumquats. Tony massages
cables and reseats the relevant Kumquat_Blossom.

========================================================================
Description of Shift:

run spill time comment
------------------------------------------------------------------------
15289 016 15:30 Shift begins shortly after start of 15289.
031 15:52 Errors from pipeline: 'No heartbeat from
crate 03' message in log file. Try hardware
reboot.
040 16:03 Stuck busy; KFREND.
120 17:52 Stuck busy; KFREND.
140 18:20 ET power supplies all trip off.
Call to request access to reset them.

15291 001 22:40 After several hours down trying to get
FASTBUS crates turned on, we start a
test run not using any Banana/Kumquat
stuff in the triggers. Deadtime is over 50%.

030 23:15 Arun sets up a new trigger configuration
to restore the chamber 1,3,4 signals,
but not using those from chamber two,
whose rack is still off.

00:00 end of disastrous shift.

========================================================================
Follow up Notes on Previous Shifts

========================================================================
Items to Follow up on Next Shift

Don't worry about the "heater" temp. Just be sure the "CsI-Bot" and
CsI-Top" readings are about 15 and 19 degrees, respectively.

-inefficient HCC bit 1767. Bad connection in ET that will be worked on
next long down time.
-latches are out of the readout so some of the monitoring histograms look
funny.

All fruits are out of trigger. Chamber 2 bananas are off because the
&*$%*! rack main breaker will not turn on. Chamber 3Y kumquats are
out of readout because of readout errors from modules 3Y-7. Fruits
are generally rotten and will require much work when JasonL returns,