High-dimensional nonlinear data assimilation with non-Gaussian observation errors for the geosciences
Date
2023
Journal Title
Journal ISSN
Volume Title
Abstract
Data assimilation (DA) plays an indispensable role in modern weather forecasting. DA aims to provide better initial conditions for the model by combining the model forecast and the observations. However, modern DA methods for weather forecasting rely on linear and Gaussian assumptions to seek efficient solutions. These assumptions can be invalid, e.g., for problems associated with clouds, or for the assimilation of remotely-sensed observations. Some of these observations are either discarded, or not used properly due to these inappropriate assumptions in DA. Therefore, the goal of this dissertation is to seek solutions to tackle the issues arising from the linear and Gaussian assumptions in DA. This dissertation can be divided into two parts. In the first part, we explore the potential of the particle flow filter (PFF) in high dimensional systems. First, we tested the PFF in the 1000- dimensional Lorenz 96 model. The key innovation is we find that using a matrix kernel in the PFF can prevent the collapse of particles along the observed directions, for a sparsely observed and high-dimensional system with only a small number of particles. We also demonstrate that the PFF is able to represent a multi-modal posterior distribution in a high-dimensional space. Next, in order to apply the PFF for the atmospheric problem, we devise a parallel algorithm for PFF in the Data Assimilation Research Testbed (DART), called PFF-DART. A two-step PFF was developed that closely resembles the original PFF algorithm. A year-long cycling data assimilation experiment with a simplified atmospheric general circulation model shows PFF-DART is able to produce stable and comparable results to the Ensemble Adjustment Kalman Filter (EAKF) for linear and Gaussian observations. Moreover, PFF-DART can better assimilate the non-linear observations and reduce the errors of the ensemble, compared to the EAKF. In the second part, we shift our focus to the observation error in data assimilation. Traditionally, observation errors have been assumed to follow a Gaussian distribution mainly for two reasons: it is difficult to estimate observation error statistics beyond its second moment, and most of the DA methods assume a Gaussian observation error by construction. We developed the so-called Deconvolution-based Observation Error Estimation (DOEE), that can estimate the full distribution of the observation error. We apply DOEE to the all-sky microwave radiances and show that they indeed have non-Gaussian observation errors, especially in a cloudy and humid environment. Next, in order to incorporate the non-Gaussian observation errors into variational methods, we explore an evolving-Gaussian approach, that essentially uses a state dependent Gaussian observation error in each outer loop of the minimization. We demonstrate the merits of this method in an idealized experiment, and implemented it in the Integrated Forecasting System of the European Centre for Medium-Range Weather Forecasts. Preliminary results show improvement for the short-term forecast of lower-tropospheric humidity, cloud, and precipitation when the observation error models of a small set of microwave channels are replaced by the non-Gaussian error models. In all, this dissertation provides possible solutions for outstanding non-linear and non-Gaussian data assimilation problems in high-dimension systems. While there are still important remaining issues, we hope this dissertation lays a foundation for the future non-linear and non-Gaussian data assimilation research and practice.
Description
Rights Access
Subject
non-Gaussian data assimilation
non-linear data assimilation
weather forecast
non-Gaussian observation error
data assimilation
particle flow filter