A novel method for Multilevel Autonomous Clustering (MAC) for anomaly detection in distributed systems

Abstract

Anomaly detection in networks is crucial to detecting security threats. Network anomalies are often not localized to a single point, but spread over a range of nodes. In this case of distributed anomalies, the anomalies are typically too subtle to detect at an individual-node level, and so require examining groups of nodes together. But it is usually not known in advance on which subset of nodes to focus; and it is infeasible to check all 2N subsets of nodes in a network. This renders distributed anomaly detection extremely challenging. An emerging strategy for detecting such anomalies is to apply a detection technique to a hierarchy of clusters of nodes in the network. However, developing such a hierarchy is challenging in large, decentralized networks with no central controller. Here, we present Multilevel Autonomous Clustering (MAC), a novel local algorithm for self-organized, hierarchical clustering in distributed networks. MAC enables individual devices in a distributed system to determine their cluster membership at multiple levels, without centralized computation or information about the entire network. The result is an approach to hierarchical network clustering that is both practical to use in large, real-world systems, as well as effective for distributed anomaly detection. The algorithm is evaluated on both synthetic and real-world networks. Its effectiveness for anomaly detection is demonstrated on various test problems. In particular, we examine the MAC algorithm's effectiveness for anomaly detection in electric power systems. Utilizing power flow balance equations, we generate anomalies that satisfy power conservation laws (and are therefore difficult to detect by normal means). Using MAC to cluster these power networks, we apply hierarchical anomaly detection on the resultant clusters.