KTeV Shift Summary, Evening of 12/26/1999

Tony Barker (tonyb@cuhep.Colorado.EDU)
Sun, 26 Dec 1999 23:11:07 -0700 (MST)

Shift Summary for Evening Shift, 26 December 1999

On Shift: J. Hamm, T. Barker

Hours of beam delivered at all intensities: 4.0
Hours of beam delivered at nominal intensity: 4.0
Hours of E799 data (special runs not included): 1.0
Hours of special runs (laser scans, muon runs...): 0

Accounting of lost time: Main Injector power supply trip.
Investigration of ground fault.
Many fresh starts, pipeline problems,
fruit power problems, reconfiguration,
and reloading.

========================================================================
Runs this shift:

Run 15295 - JUNK - BEAM_799
Run 15302 - 0.3M events - BEAM_799 [No fruit]
Run 15303 - 0.5M events - BEAM_799 [Normal, but readout errors/changes]
Run 15304 - ONGOING - BEAM_799 [Everything in nominal configuration]

Tapes brought to FCC this shift: UP(A,B,C)259-264, UPS0087-088,
UPK034, UPE018, UPLA12, UPT021,
UPF065, UPH047

========================================================================
Accesses during shift: none.

========================================================================
Overview of shift: Shift begins with beam off due to Main Injector
power supply trip. Experts are called to
investigate a ground fault.

We call MCR to ask if an electrician can look
at the problem FASTBUS rack circuit breaker box;
on-call electrician arrives 17:30, but is not
sure if this box is in his 'territory'.

Leonard Nelson is consulted. He says that these
breaker boxes were worked on by T&M electricians,
and are not nominally the responsibility of the
duty electrician. However, under the holiday
circumstances, Leonard has spoken to MCR and will
try to secure permission for the duty electrician
to make some basic checks, e.g. whether the box
is live.

The duty electrician shows up and looks at the
breaker boxes. He is displeased to find that
collaborators have been accessing their interior.
He notifies his superiors, who notify theirs.

PLEASE NOTE THE SAFETY NOTICE IN THE LOGBOOK
AND IN THE SPECIAL E-MAIL EXPLAINING THE
RESTRICTIONS ON ACCESSING DANGEROUS ELECTRICAL
EQUIPMENT!

While we are testing different relays that go
inside the breaker boxes, the chamber 3 rack
spontaneously trips off again, and we cannot
reset it, so we are back to two racks on and
two off :-(

Eventually we decide that the failed component
in the RR176 and RR177 circuit breaker boxes
can only be the MAIN shunt/breaker switches
themselves. Unfortunately, these are not
standard FNAL equipment. Fortunately, they
are not unusual, and the electrician is able to
locate two breaker switches in a contractor supply
room. He replaces the MAIN breakers, and (after
some more confusion and several tries) ALL THE
FASTBUS RACKS COME BACK ON.

Meanwhile, JasonH has been trying to get a run
going with no fruit. This has been stymied
because of hangups in the INIT process,
possibly connected to the deactivated FASTBUS
racks. Some config files are changed to
eliminate attempts to reset modules that are off,
and after five fresh starts, an INIT finally works.

THEN, when the run is started, there are pipeline
errors. An attempt to reset failed at crate 7
which he then tried to reboot. The reboot hung,
after which JasonH did several pipe_cleanup_fulls,
and then crate 7 rebooted. Finally, the run
got started, but shortly afterward we stopped
to try to put the fruits back in the nominal
configuration.

Tony reloads the Banana maps at this point, which
takes about 50 minutes to do all four crates.
When a new run is started, we again have KTEVANA
errors resulting from a hot bit in the 3Y Kumquat
crate. We eventually bypass the readout of the
3Y Kumquats, as was done yesterday.

At 23:40 we begin a run in the standard E799
configuration with the nominal triggers. At
23:55, we call MCR to request the usual 9.5E12
intensity, rather than the reduced 7.0E12 we have
been requesting since yesterday. L2BEAM/L1BEAM
ratio is back down to about 16%.

========================================================================
Description of Shift:

run spill time comment
------------------------------------------------------------------------
15295 011 15:30 Shift begins with beam off (MI power supply trip)
038 16:06 End run 15295; no beam triggers.

Replace trigger db with new ktrig_e799_noban2,
which still uses Ch 1 Bananas + Kumquats, but
not Ch 2 Bananas, YTF. or STT.

15297 001 16:30 Run with above trigger config.
011 16:44 KTEVANA errors; KFREND. Still no beam.

19:45 Beam returns
Trying to get a run going with
no fruits again, many problems (see above).

15302 001 20:50 No-fruit run starts, but triggers
are stopped soon after to try to get fruits
back in once rack power problems are fixed.
031 21:30 End run 15302.

Loading Banana maps...

15303 001 22:40 Start run with nominal E799 trigger/readout.
Various KTEVANA errors and changes to readout
as this run starts. SeveraL pipeline resets
and KFRENDS. End run when things seem stable.

15304 001 23:40 Start run with 3Y Kumquats out of readout.
No KTEVANA errors. Bananas and Kumquats
seem to be working mostly ok, but there are
some 'neutral faces' indicating possible
low-level inefficiencies or readout problems
that JasonL should investigate.

014 23:55 Call MCR to request normal 9.5E12 intensity.

00:00 end of shift.

========================================================================
Follow up Notes on Previous Shifts

========================================================================
Items to Follow up on Next Shift

Don't worry about the "heater" temp. Just be sure the "CsI-Bot" and
CsI-Top" readings are about 15 and 19 degrees, respectively.

-inefficient HCC bit 1767. Bad connection in ET that will be worked on
next long down time.
-latches are out of the readout so some of the monitoring histograms look
funny.
-Some 'neutral faces' in Banana monitoring. Bananas may have been
traumatized by recent power problems. JasonL should check these
when he returns Monday or Tuesday to see if they can be fixed.
-Chamber 3Y KQ's are out of readout, but in trigger, leading to screwy
monitoring histograms.