Representation learning for non-sequential data

Abstract

In this thesis, we design and implement new models to learn representations for sets and graphs. Typically, data collections in machine learning problems are structured as arrays or sequences, with sequential relationships between successive elements. Sets and graphs both break this common mold of data collections that have been extensively studied in the machine learning community. First, we formulate a new method for performing diverse subset selection using a neural set function approximation method. This method relies on the deep sets idea, which says that any set function s(X) has a universal approximator of the form f([sigma]x[xi]X [phi](x)). Second, we design a new variational autoencoding model for highly structured, sparse graphs, such as chemical molecules. This method uses the graphon, a probabilistic graphical model from mathematics, as inspiration for the decoder. Furthermore, an adversary is employed to force the distribution of vertex encodings to follow a target distribution, so that new graphs can be generated by sampling from this target distribution. Finally, we develop a new framework for performing encoding of graphs in a hierarchical manner. This approach partitions an input graph into multiple connected subgraphs, and creates a new graph where each node represents one such subgraph. This allows the model to learn a higher level representation for graphs, and increases robustness of graphical encoding to varying graph input sizes.