The NOvA experiment is the largest running experiment at Fermilab. NOvA scientists presented their first neutrino oscillation results at the APS Division of Particles and Fields Meeting in August of 2015. Using a beam of neutrinos generated at Fermilab, NOvA studies the oscillation parameters that define the neutrinos transformation from one type to another. The experiment compares the neutrino flavor composition observed in the 300-ton near detector and the 14,000-ton far detector in Minnesota. The computing challenges faced by the experiment involve not just processing the data, but also transferring data from both the near and far detector, cataloging and storing the data, and then delivering that data to worker nodes.

Screen Shot 2015-12-08 at 3.30.28 PM.png 

Maximum: 1,221,329 GB, Minimum 10,249 GB, Average 395,130 GB

To address their data handling needs, the NOvA experiment is using the full FIFE data handling suite: IF Data Handling Client (IFDHC), File Transfer Service (FTS), SAM File Catalog and dCache/Enstore for data storage. The combination of both near and far detector transfer rate is approximately one terabyte per day, which pales in comparison to the rate at which files must be delivered to worker nodes during peak analysis periods. During the preparation for DPF, dCache was delivering files at a rate of approximately one terabyte per hour for analysis jobs. The utilization of IFDHC, FTS, SAM and dCache allowed for complete integration into all workflows without customization by analyzers.

While data-heavy processing was focused on worker nodes near (but not exclusively at) Fermilab, processing, such as Monte Carlo generation, was transitioned to off-site resources to increase the resources available for analyzers or processing requiring large input data sets. Three example sites were Institute of Physics of the Czech Academy of Sciences (Fyzikální Ústav AV ČR or FZU), Harvard University and Texas Tech University which provided the largest number of opportunistic jobs during the last six months to NOvA. All off-site opportunistic processing combined resulted in over five million CPU hours during that time and increased the average number of cores utilized by NOvA from 2,200 cores on-site to 3,250 cores total.



The number of NOvA opportunistic jobs run at Open Science Grid sites other than Fermilab during the last six months.

The three largest contributions based on the number of jobs comes from FZU in the Czech Republic, Harvard University, and Texas Tech University.

- Mike Kirby & Ken Herner