A Scalable Online Monitoring System Based on Elasticsearch for Distributed Data Acquisition in Cms
Name
epjconf_chep2018_01048.pdf
Description
Published version
Size
445.03 KB
Format
Unknown
Checksum (MD5)
859524380565dd6d13df17bf5c86cba9
Author(s)
Andre, Jean-Marc
Behrens, Ulf
Branson, James
Brummer, Philipp
Chaze, Olivier
Cittolin, Sergio
Gomes, Diego Da Silva
Darlea, Georgiana-Lavinia
Deldicque, Christian
Demiragli, Zeynep
Date Issued
2019
Journal
EPJ Web of Conferences
Publisher
EDP Sciences
Version
Final published version
Abstract
The part of the CMS Data Acquisition (DAQ) system responsible for data readout and event building is a complex network of interdependent distributed applications. To ensure successful data taking, these programs have to be constantly monitored in order to facilitate the timeliness of necessary corrections in case of any deviation from specified behaviour. A large number of diverse monitoring data samples are periodically collected from multiple sources across the network. Monitoring data are kept in memory for online operations and optionally stored on disk for post-mortem analysis. We present a generic, reusable solution based on an open source NoSQL database, Elasticsearch, which is fully compatible and non-intrusive with respect to the existing system. The motivation is to benefit from an offthe-shelf software to facilitate the development, maintenance and support efforts. Elasticsearch provides failover and data redundancy capabilities as well as a programming language independent JSON-over-HTTP interface. The possibility of horizontal scaling matches the requirements of a DAQ
monitoring system. The data load from all sources is balanced by redistribution over an Elasticsearch cluster that can be hosted on a computer cloud. In order to achieve the necessary robustness and to validate the scalability of the approach the above monitoring solution currently runs in parallel with an existing in-house developed DAQ monitoring system.
monitoring system. The data load from all sources is balanced by redistribution over an Elasticsearch cluster that can be hosted on a computer cloud. In order to achieve the necessary robustness and to validate the scalability of the approach the above monitoring solution currently runs in parallel with an existing in-house developed DAQ monitoring system.
MIT Department
Massachusetts Institute of Technology. Department of Physics
Massachusetts Institute of Technology. Laboratory for Nuclear Science
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1051/EPJCONF/201921401048
