Collaborative Research:SHF:Medium:Machine Learning on the Edge for Real-Time Microsecond State Estimation of High-Rate Dynamic Events (8/1/20 to 7/31/24)

Abstract

Computer control of dynamic systems from the manufacturing, robotics, and aviation fields traditionally operate on timescales of 10s or 100s of milliseconds. For example, an avionics system traveling at 1000 kilometers per hour and operating at 10 milliseconds per control decision will move three meters in the time allocated to each control decision. However, emerging hypersonic, space, and military systems require active control while operating at extreme velocities or while being subjected to accelerations or decelerations caused by explosions or high-speed collisions. These applications require control at timescales on the order of microseconds. Making control decisions for such systems often requires that the controller estimate the state of the system from indirect measurements such as vibration. Traditional methods for state prediction are based on first principles using finite element analysis (FEA), whose execution time scales as a square of the number of elements. This makes it impractical to evaluate FEA models at microsecond timescales. Models derived from machine learning can estimate the state of the system based on pre-curated datasets and require less workload as compared to an equivalent FEA model. Such models, when combined with domain-specific processors, could provide equivalent accuracy with higher throughput than FEA models, making microsecond-scale state modeling possible. However, there are currently no suitable development methodologies for systematic generation of machine-learning models at such extreme performance constraints. The objective of this research is to develop a structural model compiler that meets a given accuracy constraint, as well as a corresponding overlay generator on which the generated model meets a given microsecond-scale latency constraint. This research will advance the fundamental knowledge and skills required for the real-time decision-making and control of active structures that experience high-rate dynamic events.

This project addresses two distinct but synergistic problems: (1) technologies to enable real-time decision-making and control of active structures that experience dynamic events at the microsecond timescale and (2) development of tools for optimization and synthesis of domain-specific processors for trained models. Recent academic and industrial work focusing on development of specialized architectures for evaluating Long Short Term Memory (LSTM) models generally yield “one-off” designs tuned to a specific Field Programmable Gate Array (FPGA)–often a server class FPGA–and have rigid, ?baked in? design decisions. This makes it difficult to compare alternative or competing optimization techniques for a desired target FPGA platform. To solve this, this project is developing a generalized programmable processor architecture that incorporates a repertoire of optional features designed to accelerate specific aspects of LSTMs and support associated model optimizations. The architecture is both programmable and customizable, allowing it to serve as a common platform for evaluating different approaches for accelerating LSTM models. Concurrently, the investigators are developing a set of benchmark datasets for structural state estimation with accuracy and performance requirements. The project is also developing useful artifacts for subsequent research in edge-based machine learning, including a method for comparing different LSTM model-pruning and compression approaches and comparing different microarchitecture designs. Code and hardware designs developed from this project are open-source.

Link to NSF award

Code Repositories Developed for this Project

• dropbear_lstm: A collection of Matlab sandboxes to evaluate various models for infinite sequence-to-infinite sequence mapping and forecasting
• netscheduler: A tool for lowering multi-level perceptron DAGs into arithmetic DAGs, analyzing their fundamental latency bounds, and generating HLS code with concurrent inference and dynamic re-training
• Real-time-LSTM: LSTM model for the DROPBEAR dataset
• LEMP: Local Eigenvalue Modification Procedure-based prediction for the DROPBEAR dataset
• High-Rate Structural Model Updating with LEMP: Real-time FEA model updating using the local eiginvaluce modifaction procedue.
• Physics-Informed Machine LEarning non-stationary Temporal Forecasting (PIMENTO): Physics informed time-series forecasting for the univariate dataset with nonstationarity
• LabVIEW-LSTM: LabVIEW Library for deploying LSTMS in LabVIEW and LabVIEW-Real-Time.
• PART-MLC: Programming Architecture for Real-Time Machine Learning Control. Contains FFT-based forecasting for the univariate dataset with nonstationarity.
• Towards online structural state estimation with sub-millisecond latency: Data and models for paper preseted at SAVE 2022.
• UofSC-Walking-Bridges: FEA modeling of walking bridges undertaken by REU students working on the project.
• LSTM-on-FPGA: LSTM deployment for FPGA (used for Kabir et al, FPL 2023).
• PiCaSCO: A bit-serial overlay architecture (used for Kabir et al, FPL 2023).

Datasets Developed for this Project

• High-Rate-SHM-Working-Group

Related Publications

Journal Publications

• Emmanuel A. Ogunniyi, Claire Drnek, Seong Hyeon Hong, Austin R.J. Downey, Yi Wang, Jason D. Bakos, Peter Avitabile, and Jacob Dodson, “Real-time structural model updating using local eigenvalue modification procedure for applications in high-rate dynamic events,” Mechanical Systems and Signal Processing, 195:110318, jul 2023. doi:10.1016/j.ymssp.2023.110318 pdf

Peer-Reviewed Conference Publications

2023

• Ehsan Kabir, Daniel Coble, Joud N. Satme, Austin R.J. Downey, Jason D. Bakos, David Andrews, Miaoqing Huang, “Accelerating LSTM-based High-Rate Dynamic System Models,” Proc. 33rd International Conference on Field Programmable Logic and Applications (FPL 2023) pdf

• MD Arafat Kabir, Ehsan Kabir, Joshua Hollis, Eli Levy-Mackay, Atiyehsadat Panahi, Jason Bakos, Miaoqing Huang, David Andrews, “FPGA Processor In Memory Architectures (PIMs): Overlay or Overhaul?” Proc. 33rd International Conference on Field Programmable Logic and Applications (FPL 2023) pdf

• M. Kabir; J. Hollis; A. Panahi; J. Bakos; M. Huang; D. Andrews, “Making BRAMs Compute: Creating Scalable Computational Memory Fabric Overlays”, Proc. of the 31st IEEE International Symposium On Field-Programmable Custom Computing (FCCM 2023) doi:10.1109/FCCM57271.2023.00052. pdf

• Alexander B. Vereen, Emmanuel A. Ogunniyi, Austin R.J. Downey, Erik Blasch, Jason D. Bakos, Jacob Dodson, “Optimal Sampling Methodologies for High-rate Structural Twinning,” Proc. 26th International Conference on Information Fusion, Jun. 27-30, 2023 (FUSION 2023). pdf

2022

• Joud Satme, Daniel Coble, Braden Priddy, Austin R. J. Downey, Jason D. Bakos, and Gurcan Comert, “Progress towards data-driven high-rate structural state estimation on edge computing devices,”. In Volume 10: 34th Conference on Mechanical Vibration and Sound (VIB). American Society of Mechanical Engineers, aug 2022. doi:10.1115/detc2022-90118 pdf

• Atiyehsadat Panahi, Ehsan Kabir, Austin Downey, David Andrews, Miaoqing Huang, and Jason D. Bakos, “High-rate machine learning for forecasting time-series signals,” In 2022 IEEE 30th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM 2022). IEEE, may 2022. doi:10.1109/fccm53951.2022.9786127 pdf