Tracer: A Machine Learning Based Data Lineage Solver with Visualized Metadata Management

Abstract

In databases, many data do not come from scratch. They are derived from some other data and what describes this is called data lineage. Knowing the data lineage could help us do data validation, error detection, data debugging, and privacy and access control. Unfortunately, many databases do not have well documented data lineage information, and most existing works in this area heavily relies on extra input such as metadata, source code or annotations. In this paper, we build upon Tracer, a previously purposed machine learning approach to this problem, and make it more accurate, more general, and more intuitive.