From Logs to Causal Analysis: A Guided User Interface for Causal Graph Discovery

Abstract

In a world full of digital systems, logs are found everywhere. From distributed systems logging network events to stock exchanges logging transactions, preserving information in logs is a widely-used practice. Our group’s hope is that logs can preserve events and system states at various points in time, which can later be leveraged to answer causal questions about the system. However, analyzing logs is currently far from a smooth experience. Some system dynamics might only be partially captured by log variables, while others are drowned out by the sheer volume of uninteresting, "common-case" log-lines. It is not always possible to require the logging format to match our analysis, since most systems rely on infrastructure code and libraries that cannot be altered directly. We would also be throwing away a considerable amount of existing logs. An existing system, Sawmill, is able to parse and process log data in order to answer causal questions. Sawmill’s main functionalities include presenting the user with candidate answers to causal questions, and relies on user input to accept or reject them. Doing this iteratively allows a user to build up a causal graph for a system’s logs. However, the user currently has no way to verify Sawmill’s answers. So if a user incorrectly accepts or rejects an edge representing a causal relationship based off of Sawmill’s answers on average treatment effect (ATE), this will be integrated into the user’s causal graph and can cause even more errors further down the line. In this master’s thesis, we extend Sawmill’s capability by identifying and presenting key assumptions which greatly impact Sawmill’s answer to a causal question. The existence or non-existence of these assumptions informs the user about possible different states of the causal graph, providing more context about the log and ultimately allowing the user to be more confident in drawing causal conclusions. This also mitigates cascading effect of a single error in the construction of a causal graph. Importantly, we continue to leverage the user’s knowledge about the log, relying on their ability to accept and reject assumptions.