Fanny Lehmann

While AI weather models excel at short-to-medium range forecasts (up to 15 days), they frequently suffer from ill-defined "instabilities" when rolled out over longer horizons. This work addresses the lack of a formal taxonomy by categorizing these failures into three distinct regimes: blow-up, drift, and loss of seasonality, through year-long rollouts of nine state-of-the-art AI weather models. Our analysis reveals that stability hinges on the treatment of small spatio-temporal scales: unstable models amplify high-frequency energy, while stable models act as denoisers when noise is added to their inputs. Far from reducing these models to mere stochastic parrots, our findings highlight that stable models generate unique weather trajectories, conditioned on the initial state. We verify our findings through ablation studies on architectural design choices, conducted using state-of-the-art Vision Transformer (ViT) AI weather model architectures.

Recent advances in AI weather forecasting have led to the emergence of so-called "foundation models", typically defined by expensive pretraining and minimal fine-tuning for downstream tasks. However, in the natural sciences, a desirable foundation model should also encode meaningful statistical relationships between the underlying physical variables. This study evaluates the performance of the state-of-the-art Aurora foundation model in predicting hydrological variables, which were not considered during pretraining. We introduce a lightweight approach using shallow decoders trained on the latent representations of the pretrained model to predict these new variables. As a baseline, we compare this to fine-tuning the full model, which allows further optimization of the latent space while incorporating new variables into both inputs and outputs. The decoder-based approach requires 50% less training time and 35% less memory, while achieving strong accuracy across various hydrological variables and preserving desirable properties of the foundation model, such as autoregressive stability. Notably, decoder accuracy depends on the physical correlation between the new variables and those used during pretraining, indicating that Aurora’s latent space captures meaningful physical relationships. In this sense, we argue that an important quality metric for foundation models in Earth sciences is their ability to be extended to new variables without a full fine-tuning. This provides a new perspective for making foundation models more accessible to communities with limited computational resources, while supporting broader adoption in Earth sciences.

Numerical simulations are essential tools to evaluate the solution of the wave equation in complex settings, such as three-dimensional (3D) domains with heterogeneous properties. However, their application is limited by high computational costs and existing surrogate models lack the flexibility of numerical solvers. This work introduces the Multiple-Input Fourier Neural Operator (MIFNO) to deal with structured 3D fields representing material properties as well as vectors describing the source characteristics. The MIFNO is applied to the problem of elastic wave propagation in the Earth’s crust. It is trained on the HEMEWS-3D database containing Image 1 earthquake simulations in different heterogeneous domains with random source positions and orientations. Outputs are time- and space-dependent surface wavefields. The MIFNO predictions are assessed as good to excellent based on Goodness-Of-Fit (GOF) criteria. Wave arrival times and wave fronts propagation are very accurate since Image 2 of the predictions have an excellent phase GOF (larger than Image 3). The fluctuations amplitudes are good for Image 4 of the predictions (envelope GOF larger than Image 5) and excellent for Image 6. The envelope score is hindered by the small-scale fluctuations that are challenging to capture due to the spectral bias inherent to neural networks and the complex physical phenomena associated with high-frequency features. Nevertheless, the MIFNO can generalize to sources located outside the training domain and it shows good generalization ability to a real complex overthrust geology. When focusing on a region of interest, transfer learning is an efficient approach to improve the accuracy with limited additional costs, since GOF scores improved by more than 1 GOF unit with only Image 7 additional specific samples. The MIFNO is the first surrogate model offering the flexibility of an earthquake simulator with varying sources and material properties. Its good accuracy and massive speed-up offer new perspectives to replace numerical simulations in many-query problems.

Following their success in other areas, such as language processing, the idea of creating a foundation model for natural sciences has gained traction. A foundation model for the Earth system is one that has learned the statistical relationships between physical variables and can predict them. It is a versatile model that excels when finetuned to specific tasks, and not a task-specific weather forecasting model. To achieve this, foundational models are typically pre-trained on very large datasets. Here, we introduce Earth System Foundation Model (ESFM), a fully open model that builds on the 3D Swin UNet transformer backbone of the pioneering Aurora model. ESFM introduces a series of extensions that significantly increase its functionality, thereby fostering the adoption of foundation models in the climate sciences. Firstly, the encoding scheme and training protocols have been extended to handle diverse datasets, including those containing missing values across all spatio-temporal dimensions such as satellite data, as well as station data, all under one backbone. Axial attention is introduced to capture inter-variable dependencies. As a result, ESFM skillfully predicts variables in regions or on pressure levels where no data is present at the initial time, while preserving inter-variable relationships, for example between temperature, pressure, and humidity. Individual variable tokenization enables different sets of variables to be shuffled during training and simplifies the process of building extensions for new downstream tasks. Adaptive layer norm-based ensembles allow for a simple yet effective way to transform deterministic ESFM to a probabilistic FM. We present our findings using dense gridded data, such as ERA5 and CMIP6, as well as regionally masking of dense data, sparse gridded MODIS satellite data, and station data. Our results demonstrate competitive or superior performance relative to state-of-the-art benchmarks in each domain. Additional case studies of Super Typhoon Doksuri in July 2023 and the 2024 sudden stratospheric warming events show clear indications of accurate positional and magnitude estimations of extreme weather. The results show that ESFM retains all the strengths of previous foundation models, such as long term stability, but facilitates the application of foundation models on a variety of downstream tasks.