FIFE Notes - October, 2015 News for Distributed Computing at Fermilab

Screen Shot 2015-10-14 at 1.10.21 PM.png

Best in Class


This newsletter is brought to you by:

  • Dave Dykstra
  • Gabriele Garzoglio
  • Bo Jayatilaka
  • Mike Kirby
  • Katherine Lato
  • Tanya Levshina
  • Anna Mazzacane
  • Kevin Retzke

This is the first of a series of bi-monthly newsletters to the community.  We welcome articles you might want to submit. Please email

Mu2e Offsite Simulation Campaign:
Utilizing the Open Science Grid to increase sensitivity to new physics

Mu2e was able to utilize more than 20 million CPU hours in only 5 months, averaging more than 4000 concurrent jobs with peak usage as high as 20,000 simultaneous jobs. This was without any direct capital expenditures from Mu2e.


If your experiment, proposal, or idea needs more computing resources from the OSG, please contact the FIFE group to learn how you too can access all this “free” computing. Click here for more information.


Introducing OPOS 

Offline Production Operations Service (OPOS) helps transform the raw data collected by detectors into a format that scientists can interpret.

Screen Shot 2015-10-06 at 2.48.00 PM.png 

Initial team, photo courtesy of Luis Contreras

The OPOS team contains skilled computing scientists and physicists from different regions of the world. Their combined expertise assists Fermilab experiments  in running their large-scale production data. Click here for more information.


News from the OSG - FIFE CVMFS repositories now distributed worldwide

If other FIFE VOs want to make their repositories available worldwide, they can be fast-tracked through the process. Click here for more information.

Helpful hints -- A SeaQuest tale: the Quest for Open Science Grid

SeaQuest was in a difficult situation this summer: a looming deadline for two fall conferences, a big crunch for data processing in preparation for them, the need for final tweaks to the tracker code with pressure on the few experts that could make it all happen. It became apparent that FermiGrid would not have enough opportunistic cycles to finish this computation. SeaQuest was running out of time and resources to deliver on key physics results. FIFE helped SeaQuest use resources available on the OSG.


With FIFE support, SeaQuest was successful in running on OSG. Overall, the experiment ran 1,100,000 CPU hours for 185,000 jobs, transferring 7.5 TB of input data and 9.5 TB of output. About 10% of the computing was executed at 10 OSG sites, running for the last 3 days of the campaign at the level of 500-1,000 CPU slots continuously, with an overall 90% efficiency.

Click here for more information.


Service updates monitoring: Why are my jobs not running?

Fifemon, the near-real-time monitoring system for Fifebatch, enables users to see at a glance the status of their jobs, their experiment’s jobs and the status of the Fifebatch system to answer the question of why their job isn't running.


Click here for more information including two references to carrier pigeons.


To provide feedback on any of these articles, or the FIFE notes in general, please email

The complete material (for viewing offline) is available in the following formats: