Evaluating Intrusion Detection Systems for Energy Diversion Attacks

Abstract

The widespread deployment of smart meters and ICT technologies is enabling continuous collection of high resolution data about consumption behavior and health of grid infrastructure. This has also spurred innovations in technological solutions using analytics/machine learning methods that aim to improve efficiency of grid operations, implement targeted demand management programs, and reduce distribution losses. One one hand, the technological innovations can potentially lead large-scale adoption of analytics driven tools for predictive maintenance and anomaly detection systems in electricity industry. On the other hand, private profit-maximizing firms (distribution utilities) need accurate assessment of the value of these tools to justify investment in collection and processing of significant amount of data and buy/implement analytics tools that exploit this data to provide actionable information (e.g. prediction of component failures, alerts regarding fraudulent customer behavior, etc.) In this thesis, the focus on the value assessment of intrusion/fraud detection systems, and study the tradeoff faced by distribution utilities in terms of gain from fraud investigations (and deterrence of fraudulent customer) versus cost of investigation and false alarms triggered due to probabilistic nature of IDS. Our main contribution is a Bayesian inspection game framework, which models the interactions between a profit-maximizing distribution utility and a population of strategic customers. In our framework, a fraction of customers are fraudulent - they consume same average quantity of electricity but report less by strategically manipulating their consumption data. We consider two sources of information incompleteness: first, the distribution utility does not know the identity of fraudulent customers but only knows the fraction of these consumers, and second, the distribution utility does not know the actual theft level but only knows its distribution. We first consider situation in which only the first source of information incompleteness is present, i.e., the distribution utility has complete information about the actual theft level. We present two simultaneous game models, which have same assumption about customer preferences and fraud, but differ in the way in which the distribution utility operates the IDS. In the first model, the distribution utility probabilistically chooses to use IDS with a default (fixed) configuration. In the second model, the distribution utility can configure/tune the IDS to achieve an optimal operating point (i.e. combination of detection probability and false alarm rate). Throughout, we assume that the theft level is greater than cost of attack. Our results show that for, the game with default IDS configuration, the distribution utility does not use the IDS in equilibrium if the fraction of fraudulent customers is less than a critical fraction. Also the distribution utility realizes a positive "value of IDS" only if one or both have the following conditions hold: (a) the ratio of detection probability and false alarm probability is greater than a critical ratio, (b) the fraction of fraudulent customers is greater than the critical fraction. For the tunable IDS game, we show that the distribution utility always uses an optimal configuration with non-zero false alarm probability. Furthermore, the distribution utility does not tune the false alarm probability when the fraction of fraudulent customers is greater than a critical fraction. In contrast to the game with fixed IDS, in the game of tunable IDS, the distribution utility realizes a positive value from IDS, and the value increases in fraction of fraudulent customers. Next, we consider the situation in which both sources of information incompleteness are present. Specifically, we present a sequential game in which the distribution utility first chooses the optimal configuration of the IDS based on its knowledge of theft level distribution (Stage 1), and then optimally uses the configured IDS in a simultaneous interaction with the customers (Stage 2). This sequential game naturally enables estimation of the "value of information" about theft level, which represents the additional monetary benefit the distribution utility can obtain if the exact value of average theft level is available in choosing optimal IDS configuration in Stage 1. Our results suggest that the optimal configuration under lack of full information on theft level lies between the optimal configurations corresponding to the high and low theft levels. Interestingly enough, our analysis also suggests that for certain technical (yet realistic) conditions on the ROC curve that characterizes achievable detection probability and false alarm probability configurations, the value of information about certain combination of theft levels can attain negligibly small values.