Methodologies for Training Neural Networks from Small Models to Foundation Models on Time Series Data
Name
sun-fankeng-phd-eecs-2026-thesis.pdf
Size
9.73 MB
Format
Adobe PDF
Checksum (MD5)
a24d37c28e8c7331d0fffef0a2164b3a
Author(s)
Sun, Fan-Keng
Advisor(s)
Boning, Duane S.
Date Issued
May 2026
Publisher
Massachusetts Institute of Technology
Abstract
Time series data is ubiquitous because, at its core, it is simply information collected sequentially over time. From utility companies forecasting electricity demand, to logistics networks tracking traffic patterns, and to factories monitoring the heartbeat of assembly lines — time series drives decision-making across the modern world. Imagine a scenario where we could install infinite sensors across the globe, collect every data point, and model it with unlimited computing power. To realize such a vision, we would need a model architecture that is massively scalable. Neural networks (NNs) have proven to be the best candidate for this task. Yet, despite this potential, the adoption of NNs in time series analysis lags significantly behind other domains, such as Natural Language Processing (NLP) and Computer Vision (CV). There are good reasons for this disparity. First, compared to the massive corpora of text and images available today, most time series datasets remain relatively small. Despite this, the trend in research has been to adapt increasingly large and complex architectures—inspired by the success of the Transformer—to time series problems. However, when applied to smaller datasets, these parameter-heavy models frequently suffer from severe overfitting. Instead of chasing complexity, we propose a counter-intuitive approach: Simple Feed-forward Neural Networks (SFNNs) as extremely strong baselines. We demonstrate that SFNNs can achieve state-of-the-art performance while remaining easier to code, computationally cheaper, and more robust to hyperparameter changes. The primary drawback of this architecture is that the model size grows quadratically with the input look-back window. To solve this, we introduce a periodicity align ment method, which reduces model size by up to 99.5% while incurring only a minor 11% trade-off in performance on periodic time series. A second major complication in time series is concept drift [1]. Unlike images or text, where the definition of a “cat” or the grammar of a sentence remains stable over decades, time series data often reflects real-world systems that change dynamically. For example, a manufacturing machine experiences wear and tear, subtly altering its data signature over time and degrading NN model accuracy. To address this, we propose a Bayesian approach for drift adaptation in NN time series models. This formulation allows the model to continuously update its parameters as new batches of data arrive, ensuring that predictive performance remains consistent even as the underlying system evolves. While the previous contributions focus on forecasting, another critical application is anomaly detection. In this context, anomalies are rare and often unknown during the training phase. NNs tend to overfit the normal data, making them brittle when detecting novel outliers. To alleviate this, we propose a regularization method based on density matching. This technique theoretically and practically constrains the model, preventing overfitting. We validate this method across a wide range of base models and datasets, demonstrating its ability to universally improve baseline performance. Finally, we tackle the challenge of data diversity. In NLP and CV, researchers now train massive foundation models by treating all text or images as a single domain. This is difficult in time series because the data is fundamentally heterogeneous; traffic flow data has little in common with patient vital signs, making a unified model difficult to train. However, training separate models for every single dataset is inefficient. Bridging this gap, we explore training an anomaly detection foundation model within a single domain: semiconductor manufacturing. Here, diversity comes from different tools, recipes, and machines rather than entirely different physics. Our exploration yields promising results but highlights the difficulty time series foundation models face when extrapolating to out-of-sample distributions. Together, these methods provide fundamental and essential pathways to training time series NNs that perform robustly in the real world while avoiding common pitfalls. Future research can build upon this thesis to further extend the utility of NNs into the dynamic realm of time series.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
