Shift Summary 8/28/99 DAY shift

Vivian O'Dell (odell@fnal.gov)
Sat, 28 Aug 1999 16:03:41 -0500

Shift summary for DAY shift, 8/28/99

On Shift: S. Amasha, V. O'Dell

Hours of beam available at all intensities: 7.4h
Hours of beam available at nominal intensity:7.4h
Hours of E832 data (special runs not included): 6h
Hours of special runs (laser scans, muon runs...):0h

Accounting of lost time: mainly plane 4 problems with some complications
from damp_fwdr, diskmon and lastly a bad tape,
========================================================================
Runs ended this shift:

14354 - 17.0 K vac 2pi0 stopped by plane 4 stream 4 crash
14356 - ended and overwritten stopped by plane 4 stream 4 crash
14358 - 17.4 K vac 2pi0 stopped by plane 4 stream 4 crash

Tapes brought to FCC this shift: QK(S,C,K)336-344, QKB117,QKB118,QKB119
Accesses during shift:
========================================================================
Overview of shift: Bad start, fairly smooth middle, lousy ending.

========================================================================
Description of Shift:

run spill time comment
------------------------------------------------------------------------
14354 07:00 Day shift turns up to insult spokesperson. Spokesperson
leaves, muttering "I'd rather be home making pancakes."

14354 last 08:20 Run crashes. plane 4 is the culprit. Groggy shift
crew misdiagnosis stream but surmises it was quite
likely stream 4 after the next crash (coming up).

14356 08:52 Swap out tapes (they were about 1/3 full) and start a new run.

14358 08:55 Crash again. This time shift crew is more awake and
sees it is definitely plane 4 stream 4.

14358 09:35 After several drt_fresh_starts with red messages babbling
about damp_fwdr, finally page Ed to ask what to do. He calls
back, swinging music in the background. Our respect for him
skyrocketed, thinking he'd been in a bar all night. Imagine
our disappointment to learn he was in a yuppie health club.
At any rate, he says to fix the damp_fwdr problem,
"log out and in again". We did it and it worked. It looks like
if you do the wrong thing (ie anything) in the open run control
window, you could set up a different version of tcl than what
damp uses, which then screws up damp_fwdr. Put a message on
the RC screen to this effect, which made Ed feel his expertise
is no longer needed.

Overwrite the tapes from the first fiasco (there were only
a few spills of data on it).

14358 10:00 Run manager offers magnanimously to sit in for one of the
shift crew members so that she can go to a meeting. What a
terrific run manager we have!

14358 10:13 SDAQ + processes buttons are red.
SDAQ: "Vacuum is not running in atev1"
processes: process "vacuum.tcl" in machines "atev1" is NOT
running.

Paged Theo/Ashkan.

14358 10:30 restarted vacuum.tcl process on atev1.

14358 10:53 SDAQ button red:"You have to create new swics DB file". Call/page
Theo/Ashkan. Send email, too.

14358 11:55 Shift crew member returns from meeting. Run manager does not
look so happy although he does offer to take the rest of the
shift (I believe "might as well" figured into the offer).
All efforts to appease him, short of offering him a banana, fail.

14358 12:14 MI aborts: losing spills.

14358 12:15 Offer run manager the banana. At this point he has decided to
go out to lunch so he sneers at the banana. Decide to page Rick
K. just to hear a friendly voice. Trigger 5 L3 accepts is much
higher than it was, due to the missing trigger latch crate. Rick
said "Don't worry about it". He's very annoyed to be paged while
eating breakfast in a restaurant. He would not divulge what
he had ordered for breakfast.

14358 12:21 beam returns... for 1 spill.

14358 126 12:25 beam back (again).

14358 128 12:28 beam gone.

14358 132 12:35 beam back

14358 142 12:48 beam leaves

14358 147 12:55 beam back

14358 163 13:15 Theo calls in, promising all SDAQ buttons will be green in 5
minutes.

14358 13:20 All SDAQ buttons are green.

14358 last 13:50 ktevana errors. Ran kfrend.
plane 4, stream 4 crashes again. Coincidence?
reseat all the h/w in stream 4 in the hopes of doing
some good... change tapes, drt_fresh_start. You know
the drill.

13:58 Beam dies, a convenient death.

14:20 Beam back, we're not.

14:25 start fails, diskmon blah blah blah. Sydney says he's
never seen it before. Yet another drt_fresh_start.

14:35 init hangs on tape in ktev2.
eventually decide to replace the tape. Sydney runs
ftt test program on new tape and it works. We muddle
on.

14:42 lose beam. 15 minute estimate.

14:48 beam back. Viv beings cursing heavily.

14:58 beam gone. Viv relaxes.

15:05 plane 4 dead again. Turns out that no power was on
any of the dc2's, even though the power switch was
on. Flaky power supply? power cycle and things come
back up. We also swapped the 1st memory module for
stream 4 between planes 4 & 2. Will this have any
discernable (or desirable) effect?

15:05 beam back. Viv curses more.

15:15 beam gone. Viv relaxes.
MCI calls to say beam may be intermittant. We are
all very surprised. Imagine that!

15:20 finally get new run (14366) started. Everyone sighs in
relief.

14366 2ish 15:23 beam back.

15:30 new shift turns up. We head for the hills before
things can break again.

========================================================================
Follow up Notes on Previous Shift:
o kept our eys on channel 450. It was reasonably stable during the
time we were.
========================================================================
Items to Follow up on Next Shift:

o if we lose beam for >= 40 minutes, and the DA works, do a laser scan.

o if no beam for > 2 hours, page Julie for divider swap.

o keep your eyes on channel 450 (see log book p. 103). If it remains stable,
and we do a laser calibration, maybe we don't need to swap the divider
(ask Ed).

o give Theo fresh batteries for his pager.
========================================================================