11/13/99 Evening shift summary

Valeri Jejer (jejer@uvahea.phys.virginia.edu)
Sun, 14 Nov 1999 01:13:14 -0500 (EST)

Shift Summary for evening shift, 13/Nov/99

On Shift: Jim H., Valeri.

Hours of beam available at all intensities: 7.4
Hours of beam available at nominal intensity: 7.4
Hours of E832 data (special runs not included): 7.15
Hours of special runs (laser scans, muon runs...): 0.0

Accounting of lost time:

Accelerator problems and beam aborts - 1.1 hr
Pipeline resets, ET reload, kfrends - 15 min

========================================================================
Runs ended this shift:
Run 14961 - in progress - BEAM_799

Tapes brought to FCC this shift:
(including tapes for online split) none

========================================================================
Accesses during shift:
None

========================================================================
Overview of shift:

Smooth datataking with good beam quality and stable intensity at 9e12
until 18:50, then intermittent beam due to pre-accelerator problems with
many empty spills and aborts and intensity 8-8.5e12 for about an hour,
then smooth running at 9e12 with some aborts. Pipeline problems twice.
========================================================================
Description of Shift:

run spill time comment
------------------------------------------------------------------------
14961 15:35 kfrend to fix VV' latch drift (no beam lost)
16:20 Dave N. put 2 new ArEth bottles on west side line
17:45 kfrend to fix VV' latch drift (no beam lost, but
decided to give up at this point, they are out of
in-time slice in about 5 min. VV' latches are not
usable starting run 14955. Trigger latches are OK,
no information about other latches.)
250 18:24 Beam aborts, decide to reset pipeline (we were
getting "Message from pipeline()" every 5-10 min and
some CSI channels (1957, 1325) appeared in the
Comparator Problems list for short time). Reset
stuck in crate 14. Killed pipe.tcl, run pipe_clean
after which pipeline was reset without errors,
although Cr15 button was red. Lost 4 spills.
18:50 Beam disappears for a while.
19:02 Message on Ch.13:"Trouble with pre-accelerator,
ops investigating, beam will be intermittent"
21:45 Mounted UPH022 on tps3d2.
22:16 HCC is inefficient, there was a vertical stripe,
reload current ET maps -> kfrend -> illegal module
number errors from a lot of crates -> kfrend ->
pipeline errors and ordinal number mismatch ->
kfrend -> illegal module number errors -> kfrend.
lost 6 spills.
23:18 Disk ktev3a is 91% full, delete two files from
/ktev3a/DATA/CSI area (for run 14923 on Nov. 5
and before)
========================================================================
Follow up Notes on Previous Shift
> - Monitor timing L1 timing shift

L1 timing was stable during the shift.

> - Monitor CPU usage on KTeV 4; process 37 on ktev4 appears not to be
> processing data.

Monitored ktev4 CPUs when L3 CPU time summary showed that CPU 37 is dead.
No. of processed events plot was OK at the same time and top showed that
all 7 L3 processes are running and using about the same CPU fractions, so
the problem is in monitoring,
========================================================================
Items to Follow up on Next Shift