Applying high performance computing to early fusion video action recognition

Abstract

Over the past few years, there has been significant interest in video action recognition systems and models. However, direct comparison of accuracy and computational performance results remain clouded by differing training environments, hardware specifications, hyperparameters, pipelines, and inference methods. Additionally, the literature demonstrates a fixedness on late fusion approaches to audio-video multimodal problems. This project provides a side-by-side comparison of several 2-Dimensional Convolutional Neural Network (2D-CNN) video action recognition approaches and investigates the effectiveness and efficiency of new audio-video early fusion, slicing, and sampling methods. Model accuracy is evaluated using standard Top-1 and Top-5 metrics in addition to novel p-ROC metrics, and this project demonstrates the usefulness of the latter. Computational performance is measured via total training time and training time per epoch on a variety of high-performance computing (HPC) training configurations.