By Sam Partee, Matthew Ellis
The scientific community has spent decades building specialized applications for scientific simulations, which are typically written in Fortran, C, and C++. These codes are computationally expensive and, as such, require large high-performance computers (HPC). Recently, there has been tremendous growth in AI, data science, and heterogeneous (GPU/CPU) computing. However, due to the technological anachronism of the scientific applications and the Python data science ecosystem, there is a wide chasm between these historical simulations and modern AI tools. This has presented us a grand challenge of developing a method to leverage modern AI/data science practices within the same frame of reference as these well-established simulations.
In this talk, we present SmartSim—a library that utilizes Redis and RedisAI to enable computational scientists to access modern data science and machine learning methods unencumbered and at scale. SmartSim includes RedisAI clients in four languages—Python, C, C++, and Fortran—that are all Redis cluster compatible. In addition, we show how SmartSim provides launching mechanisms for the utilization of Redis at HPC scale. Lastly, we demonstrate the effectiveness and scalability of our approach by using SmartSim + RedisAI to augment a computationally expensive global climate simulation with a PyTorch model to improve simulation realizations without impacting simulation runtime.