​​OPOS: The importance of collaboration and cooperation

Offline Production Operations Service (OPOS) assists experiments with running their large-scale production workflows and other large-scale off-line production activities. With three computer scientists and two physicists, the group expands the human resources of the experiment's off-line team while coordinating across multiple requests. The Service Level Agreement, cd-DocDB-5563, gives the details of what is offered to the experiments as well as what is required to be eligible to request this service. Basically, the experiment must have production code, scripts and configuration files with ownership limited to authorized people. The experiment must train the OPOS team and make a request via a Service Desk ticket to request service.


Once trained, OPOS team members schedule, according to the collaboration's deadlines, and run the large-scale production tasks. OPOS team members monitor the tasks and provide the experiment with status and progress information.


On Nov. 30, DUNE started using OPOS for MCC5.0 production, which had requests from several physics working groups such as oscillations, supernova, proton decay and cosmogenics. DUNE had a training session and gave the OPOS team instructions on submitting and monitoring jobs and doing SAM-related operations. An issue with account permission was quickly solved with help from SCD. By day two, OPOS was producing samples at full speed. Half the samples were finished after two-and-a-half days. As Tingjun​ Yang said, “We are extremely impressed by their dedication and cooperation on the MC production. I think the OPOS group is doing a fantastic job, and their contribution is very much appreciated by the DUNE collaboration.”


The OPOS group facilitates the transfer of tools and know-how among experiments by helping with the adoption of common tools. The goal is to help experiments increase their productivity while focusing on the final results: data analysis and papers.

- Anna Mazzacane & Katherine Lato