PASS1 Instructions

The duties of the Pass1 shift are the following:

• Prepare for the shift at least one day in advance
• Submit jobs to production system queues
• Ensure each queue has enough jobs
• Ensure each production system is processing data
• Inform Pass1 Czars of any problems
• Follow additional instructions of Pass1 Czars

Before taking your PASS1 shift, please do the following:

• Check the PASS1 shift schedule so you can contact the previous shift personnel and the Pass1 czars. Find out the day before your shift (Monday) whether there is anything specific that you should know.

• Make sure you know what periods of runs that you should be processing. The Czars should tell you this, but may need to be reminded.

• Notify the previous shift personnel that you are ready for your shift.

• If you are in the U.S.A. make sure you get a Pass1 pager

• If there are problems causing you to miss your assigned shift you should contact your own institution personnel to sort this out. The PASS1 shifts are considered institutional responsibilities. If your institution cannot do your assigned shift you need to contact the FOCUS PASS1 czars as soon as possible (Eric Vaandering (ewv@fnal.gov), Jon Link (link@fnal.gov) or just focus_pass1_czars@fnal.gov)

• Since problems do arise, it is recommended that one or two people do their shifts for several days in a row as we find that this is more efficient. Also they can become expert enough to train other people in their institution or the next shift crew.

• Look through the PASS1 WEB pages before your shift to familiarize yourself with the PASS1 pages, especially the MECCA access page and the PASS1 shift instructions page

• Read the whole of this WEB page and make sure you understand what you need to do. As well as checking out the WEB pages, you should log onto the farm systems to use some of the monitoring commands available there.

This section is intended to give a very basic introduction to the Fermilab Farms. It is by no means complete, but it is all the shift should need to know. More information is available, of course.

• There are currently 8 Pass1 farms (properly "production systems") that we are using to process raw (FDxxxx) data tapes. We call these systems a, b, c, d, e, f, g, and h. They are actually labelled as "ssse831x" where "sss" is either sgi or ibm depending on the system, and x is either a, b, c, d, e, f, g, or h.

• Each farm has a host; the SGI farms have fnsfu.fnal.gov as the host and the IBM farms have fnckm.fnal.gov as the host. Farm f is its own host fnio.fnal.gov. For each farm, some actions MUST be performed from the host machine. Below are the farm systems, full names and hosts.

  FARM SYSTEM	NAME	HOST	APPROX. PROC. TIME
  a	sgie831a	fnsfu.fnal.gov	7-8 hours
  b	sgie831b	fnsfu.fnal.gov	7-8 hours
  c	ibme831c	fnckm.fnal.gov	6-7 hours
  d	ibme831d	fnckm.fnal.gov	6-7 hours
  e	sgie831e	fnsfu.fnal.gov	7-8 hours
  f	sgie831f	fnio.fnal.gov	4-8 hours
  g	ibme831g	fnckm.fnal.gov	7-8 hours
  h	ibme831h	fnckm.fnal.gov	7-8 hours

• The tape drives are actually attached to the hosts, so the input and output processes are performed on the host node. The actual analysis process is done on the farm nodes. The different farms have different computing power; shown above is the typical time a job takes on each of the different farm systems for a typical full raw data tape.

• In order to submit a job to a particular farm system you must log onto and submit from the host node as user e831p1. You should know the password. For example to submit a job to system e, you need to log onto fnsfu.fnal.gov as user e831p1. See next section for submission command.

• In order to run any of the cps viewing commands to looking at the farm jobs from the host, the host must know about the system. At present system f is only known to host fnio. Host fnio almost only knows about system f (sgie831f). So viewing of systems a-e,g,h is possible from either fnsfu or fnckm, however system f can only be viewed from fnio. See next section for viewing commands or see the summary of useful cps commands.

• Each system has a job queue. At any time, one job can be "On-deck" waiting to run, one job can be running, and a third job can be in the rundown phase. Any number of jobs can be in the queue waiting to be on-deck. ("On-deck" is a baseball term for anyone who doesn't know).

Check if you need to submit a job on any of the farm systems. You can do this by checking the WEB page or by logging onto the farm host. In general you should keep five jobs running/queued on each system. Always make sure you have at least one job per queue which has not started to do anything. If you will not be checking the systems for an extended time like 8-10 hours (e.g. overnight) you should make sure there are at least five jobs on each system. To check the number of jobs you can do the following:

• Check the MECCA access page to see how many jobs are in the queue for each system. If there are four or less you should submit more so that you have at least five. There are two Warnings in only looking at the WEB page.
  1. Jobs submitted more than 3 days ago disappear from the Runs in Progress table. For this reason, we recommend keeping the number of submitted jobs to 7 or less.
  2. There may be problems that cause the MECCA page to show jobs in a queue when in fact the system is idle. To avoid these problems, pay attention to the times and phases given in the table.
• Each phase of Pass1 has a time it normally takes to execute
  • Submitted: No time limit
  • Waiting: A few minutes for staged jobs, 6-12 hours for unstaged jobs
  • Input Staging: 2.5-6 hours
  • Ready: Waits until previous job finishes
  • Analysis: 6-12 hours (see table above)
  • Output Staging: 2.5-6 hours
  • Done: A few minutes until results from that run are processed
• If a stage takes much longer than this it should be viewed with suspicion. The status code will turn orange and then red if the job takes too long at a given stage. The shift should attempt to determine if the Pass1 system is working correctly or if things are frozen. If there is any doubt, contact a czar.

To submit a job on the farm you should

1. Reload any monitoring pages and check your e-mail for special instructions from the czars.

2. Find the next tape number you want to submit. Note that you can get this from the MECCA access page by clicking on the latest run number. A table will show the run and tape numbers for the relevant range of runs with the status of each. Select the next one which is not yet submitted or one that failed previously and is not yet resubmitted. Note the TAPE NUMBER, not the run number.

   The table also shows the expected number of events. If there are runs missing between tapes you can click on the tape label to see if multiple runs were stored on the tape. This link will give you a better estimate of the number of events on the tape. Do not submit tapes with less than 800,000 events. Submitting short runs will disrupt the timing of the farms and cause CPU time to be wasted. E-mail a Pass1 Czar (when you e-mail your usual report) and ask for special instructions regarding these tapes.

3. The Pass1 status code gives the status of a run. D means the job has completed successfully, S means it has been submitted, P means the job has completed with some minor error, and E means that an error may have occurred which requires the attention of the Czars. Combinations are possible, so ES means the job had an error it first time and has been resubmitted.

4. Log onto the correct host (from the table above). Login as user e831p1.

5. You should be in the home (top) area of user e831p1 (i.e. ~e831p1). To submit a job use the command "p1submit # xxxx", where # is the farm system (a-f,t) and xxxx is the 4 digit TAPE NUMBER.

   Enter the username (e831p1) and password when prompted. If you make a mistake typing in the system or tape number type in the wrong password. (Please do not hit CTRL-C). If you type in the username or password incorrectly just try again.

   See the section on staging below for one minor modification to the p1submit command usage.

6. You can check if the job was submitted properly with the correct tape numbers by viewing the Mecca Access page or using the monitoring commands below.

If you find that you need to cancel a job, e.g. one submitted with the wrong tape or system number, issue the command:

e.g. p1cancel a 25 This command can also be used to abort a running job.

If you must cancel a running job or one that is on-deck, let a Pass1 czar know that you did so as there may need to be special cleanup and accounting modifications. In general, the shift should never have to do this.

PROBLEMS

Jobs occassionally fail, or at least seem to. These are the jobs that appear with an E in the table of jobs over the last 36 hours. At least once per day, but not more than twice per day, you should send e-mail to the czars with a list of these runs. The Czars will invesitigate and tell you whether to resubmit these jobs or if they are OK. If the jobs are OK, the czars will modify the database so that the E no longer appears. This may take some time as the czars are often quite busy. Single failures are not considered critical to progress since there are thousands of tapes left to analyze.

Typically Pass1 czars take one week shifts in which they deal with these non-critical problems.

More infrequently something will happen that will either cause a queue to freeze or cause all the jobs in the queue to die in rapid succession. This is a situation which requires the immediate attention of the czars. You may try e-mail first, but if you don't hear anything back in a few minutes, you should call or page a czar. If there is any doubt as to whether a problem is serious or not, make sure you get in contact with a czar (not just the primary czar). See below for phone and pager numbers.

One exception to this division is Output Staging errors. If you notice one of these, e-mail the czars immediately, but you needn't call or page anyone.

Name	E-mail	Office (Work)	Home	Pager
Irwin Gaines	gaines@fnal.gov	(630) 840-4022	(630) 420-1452*	(888) 390-9193
Jon Link	link@fnal.gov	(630) 840-2183	(630) 584-9613	(800) 241-9016*
Alberto Sánchez	asanchez@fnal.gov	+55 (21) 541 0337 x197	+55 (21) 542 2602
Eric Vaandering	ewv@fnal.gov	(303) 492-4821	(303) 543-8924	(800) 241-9165*
*Preferred after hours contact method

SHIFT CONCLUSION

On the last day of your shift (Monday) you should do the following:

• Check the PASS1 shift schedule so you can contact the next person on shift and make sure they are ready for their shift.

• Give the next person on shift any specific information that they may need, e.g. known tape or tape drive problems, etc.

• Make sure the next person on shift knows the run/tape number range for the farm systems. If the next person is in your institution you should train him/her to help make running PASS1 easier and more efficient.

• If you have a Pass1 pager, send it back to Jon Link.

---

More detailed information is available from these sources:

PASS1 shift schedule

FOCUS phone list page

Summary of FOCUS Pass1 commands

Description of staging software

Send comments to: FOCUS Pass1 Czars