Covariate-adjusted Expected Shortfall under Data Heterogeneity

The expected shortfall of a random variable is the tail average below or above a given threshold specified by a quantile, and it becomes a natural and useful summary statistic when low- or high-valued outcomes are of primary interest, as is often the case in risk assessment and treatment effect detection. Given the emerging importance of the expected shortfall as a summary measure, the recent literature in financial econometrics, statistics and operations research has focused on the expected shortfall regression, which enables one to evaluate the tail differences after adjusting for the covariates or possible confounding factors. The project will study the estimation of covariate-adjusted expected shortfall, identify new approaches for estimation, and study the statistical properties for its adaptation to data heterogeneity. The proposed research will provide toolkit for data-driven and evidence-based analysis in diverse fields, including concussion research, health disparity research, and climate studies. The project will also contribute to the training of a new generation of statisticians and data scientists. The project will develop a new approach to estimation of covariate-adjusted expected shortfall that is computationally feasible and flexible, adapts well to data heterogeneity, and allows effective statistical inference. The proposed approach is built on a characterization of the expected shortfall based on a quantile loss function, but without reliance on the quantile function itself. When the expected shortfall function takes a parametric form, the proposed approach will start with an initial estimator of the expected shortfall at possibly a sub-optimal rate of convergence and obtain a much better solution from convex optimization. The proposed method works under weak modeling assumptions and opens a new window of opportunities for better statistical inference for expected shortfall regression. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.