Continual Learning for Engineering: Benchmarking and Exploring Strategies for 3D Engineering Problems

Abstract

Engineering applications of machine learning often involve high-dimensional, computationally intensive simulations paired with limited and evolving datasets. As new designs and constraints emerge, models must adapt to incoming data without frequent retraining, which is often infeasible due to the cost of generating engineering data. Continual learning (CL) offers a promising alternative by enabling models to incrementally learn from sequential data while mitigating catastrophic forgetting, in which there is a loss of performance on previously seen examples. This thesis investigates the application of continual learning to regression-based engineering tasks, with an emphasis on surrogate modeling. We begin by benchmarking several foundational CL strategies, including regularization-based and rehearsal-based methods, across five diverse engineering datasets. To support this analysis, we construct nine new regression-focused continual learning benchmarks designed to reflect practical engineering scenarios. Results show that Experience Replay, a simple rehearsal method, consistently achieves strong performance, approaching "joint training" performance baseline of retraining from scratch, while substantially reducing computational cost. To further explore how rehearsal strategies can be made more efficient and effective, we propose two adaptive replay methods that prioritize memory samples based on forgetting dynamics. These methods extend previous adaptive replay strategies by using input clustering and representations from TabPFN, a foundation model for tabular data, to guide more informed sample selection without knowledge of experience boundaries. We evaluate their performance on both complex engineering datasets and controlled synthetic tasks. In scenarios where forgetting is unevenly distributed, the adaptive methods offer clear advantages, highlighting the potential for more intelligent replay under constrained resources. This work positions continual learning as a practical and effective strategy for handling dynamic engineering datasets, and offers new insights into how adaptive replay can enhance efficiency in data-limited, high-cost learning environments.