12-Jan-00 DAY shift summary

Subject: 12-Jan-00 DAY shift summary
From: Vivian O'Dell (odell@fnal.gov)
Date: Fri Feb 04 2000 - 15:21:15 CST

              Shift Summary for DAY shift, 12-JAN-00

On Shift: Herman W., Sasha L.
Also: many other people

Hours of beam available at all intensities: 7.33 h
Hours of beam available at nominal intensity: 7.00 h
Hours of E799 data (special runs not included): 0.0
Hours of special runs (laser scans, muon runs...): 2.50 h

Accounting of lost time: Tape failure aborts, DAQ errors,
online cluster reboots.

Runs ended this shift:

Tapes brought to FCC this shift:

Accesses during shift:
Overview of shift: In few words: The theater of absurd.

Three tape failure aborts, lots of drt_fresh_starts, ktevana errors,
pipeline errors, Ordinary number mismatch errors, DART errors, You
name it - we had it. Despite everything we took some short regenerator

Inventory of blank tapes left in KTeV Hall: 147 unlabeled and 186
                                            labeled tapes

Description of Shift:

run spill time comment
15485 7:00 Shift starts.

        210 7:27 "HCC is inefficient" for 2 spills. Reload Etot.
                8:11 "Abort!" Tape failure in ktev1/tps2d2
                        Run does not end nicely. l3#save
                        Cleaning the tape drive
                        Changing tapes (~30% full)
                        There are not enough labeled tapes to load.
                        Label some tapes. Do drt_fresh_start

                8:20 Beam is gone. A1 fridge problems

                8:30 Run 15486 started. Beam est 20 min.
                8:55 Beam est 20 min
                9:04 Beam est 10 min

                9:20 Beam is back. Spill structure is terrible.

                        HCC ch 1952 is inefficient very often. JasonH
                        walks in just in time: "1952 dynode is toast".
                        We keep taking data.

                9:57 Beam is gone. Est 40 min
                 10:05 Beam returns

                10:44 "Abort!". Tape failure in ktev3/tps2d1.
                        Cleaning all tape drives. Changing tapes.

                10:52 Meanwhile, Main Injector is having a problem.
                        Intensity is ~1.2E12
15488 11:20 Start run 15488.
                        It started with LOTs of "Ordinal number mismatch",
                        "Missing DYC word" and pipeline errors
                        KFRENDs don't help. At some point we turned off
                        all triggers and started turning them on one by one.
                        All triggers are on and there is no ktevana errors.

        24 11:50 Everything is running, finally

                11:52 "Abort!" Tape failure in ktev1/tps2d2 (again)
                        reconfiguring tape drive destination to avoid this
                        drive. Changing tapes. Labeling more tapes.

15490 Starting new run, 15490. LOTs of ktevana errors
                        and such (see begining of run 15488). KFRENDs don't
                        help. At some point we turned off
                        all triggers and started turning them on one by one.
                        All triggers are on and there is no ktevana errors.
                        (yes, same as in 15488)

        27 Everything is running smoothly.

                13:40 DDD veto 3. Stop the run
15491 14:09 initialize run 15491. LOTs of "dfm_schedule_list"
                        errors on ktev1. DO drt_fresh_start again

15492 14:22 initialize 15492 but same errors as in 14591. Page
                        DART. Things got out of control with rebooting
                        all machines. Problems with rebooting.

                15:20 All machines are up (we are not sure)
                        Doing fresh start for dart and sdaq.

15493 15:29 Started new run 15493. Triggers are running,
                        no errors. Shift ends.
Follow up Notes on Previous Shift:

Items to Follow up on Next Shift:

This archive was generated by hypermail 2b27 : Fri Feb 04 2000 - 15:21:17 CST