MIT Libraries homeMIT Libraries logoDSpace@MIT

MIT
View Item 
  • DSpace@MIT Home
  • MIT Open Access Articles
  • MIT Open Access Articles
  • View Item
  • DSpace@MIT Home
  • MIT Open Access Articles
  • MIT Open Access Articles
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.

A scalable monitoring for the CMS Filter Farm based on elasticsearch

Author(s)
Andre, J-M; Andronidis, A; Behrens, U; Branson, J; Chaze, O; Cittolin, S; Deldicque, C; Dobson, M; A Dupont; Erhan, S; Gigi, D; Glege, F; Hegeman, J; Holzner, A; Jimenez-Estupinan, R; Masetti, L; Meijers, F; Meschi, E; Mommsen, R K; Morovic, S; Nunez-Barranco-Fernandez, C; O'Dell, V.; Orsini, L; Petrucci, A; Pieri, M; Racz, A; Roberts, P; Sakulin, H; Schwick, C; Stieger, B; Zaza, S; Zejdl, P; Gomez-Ceballos, Guillelmo; Paus, Christoph M. E.; Sumorok, Konstanty C; Veverka, Jan; Darlea, G. L.; ... Show more Show less
Thumbnail
DownloadA scalable monitoring.pdf (1.548Mb)
PUBLISHER_CC

Publisher with Creative Commons License

Creative Commons Attribution

Terms of use
Creative Commons Attribution 3.0 Unported license http://creativecommons.org/licenses/by/3.0/
Metadata
Show full item record
Abstract
A flexible monitoring system has been designed for the CMS File-based Filter Farm making use of modern data mining and analytics components. All the metadata and monitoring information concerning data flow and execution of the HLT are generated locally in the form of small documents using the JSON encoding. These documents are indexed into a hierarchy of elasticsearch (es) clusters along with process and system log information. Elasticsearch is a search server based on Apache Lucene. It provides a distributed, multitenant-capable search and aggregation engine. Since es is schema-free, any new information can be added seamlessly and the unstructured information can be queried in non-predetermined ways. The leaf es clusters consist of the very same nodes that form the Filter Farm thus providing natural horizontal scaling. A separate central" es cluster is used to collect and index aggregated information. The fine-grained information, all the way to individual processes, remains available in the leaf clusters. The central es cluster provides quasi-real-time high-level monitoring information to any kind of client. Historical data can be retrieved to analyse past problems or correlate them with external information. We discuss the design and performance of this system in the context of the CMS DAQ commissioning for LHC Run 2.
Date issued
2015-12
URI
http://hdl.handle.net/1721.1/108642
Department
Massachusetts Institute of Technology. Department of Physics; Massachusetts Institute of Technology. Laboratory for Nuclear Science
Journal
Journal of Physics: Conference Series
Publisher
IOP Publishing
Citation
Andre, J-M; Andronidis, A; Behrens, U; Branson, J; Chaze, O; Cittolin, S; Darlea, G-L et al. “A Scalable Monitoring for the CMS Filter Farm Based on Elasticsearch.” Journal of Physics: Conference Series 664, no. 8 (December 2015): 082036. © Copyright 2015 IOP Publishing
Version: Final published version
ISSN
1742-6588
1742-6596

Collections
  • MIT Open Access Articles

Browse

All of DSpaceCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

My Account

Login

Statistics

OA StatisticsStatistics by CountryStatistics by Department
MIT Libraries homeMIT Libraries logo

Find us on

Twitter Instagram YouTube

MIT Libraries navigation

SearchHours & locationsBorrow & requestResearch supportAbout us
PrivacyPermissionsAccessibility
MIT
Massachusetts Institute of Technology
Content created by the MIT Libraries, CC BY-NC unless otherwise noted. Notify us about copyright concerns.