Collaborative Research: Statistical Inference for High Dimensional and High Frequency Data: Contiguity, Matrix Decompositions, Uncertainty Quantification

To pursue the promise of the big data revolution, the current project is concerned with a particular form of such data, high dimensional high frequency data (HD2), where series of high-dimensional observations can see new data updates in fractions of milliseconds. With technological advances in data collection, HD2 data occurs in medicine (from neuroscience to patient care), finance and economics, geosciences (such as earthquake data), marine science (fishing and shipping), and, of course, in internet data. This research project focuses on how to extract information from HD2 data, and how to turn this data into knowledge. As part of the process, the project develops cutting-edge mathematics and statistical methodology to uncover the dependence structure governing HD2 data. In addition to developing a general theory, the project is concerned with applications to financial data, including risk management, forecasting, and portfolio management. More precise estimators, with improved margins of error, will be useful in all these areas of finance. The results will be of interest to main-street investors, regulators and policymakers, and the results will be entirely in the public domain. The project will also provide research training opportunities for students. In more detail, the project will focus on four linked questions for HD2 data: contiguity, matrix decompositions, uncertainty quantification, and the estimation of spot quantities. The investigators will extend their contiguity theory to the common case where observations have noise, which also permits the use of longer local intervals. Under a contiguous probability, the structure of the observations is often more accessible (frequently Gaussian) in local neighborhoods, facilitating statistical analysis. This is achieved without altering the underlying models. Because the effect of the probability change is quite transparent, this approach also enables more direct uncertainty quantification. To model HD2 data, the investigators will explore time-varying matrix decompositions, including the development of a singular value decomposition (SVD) for high frequency data, as a more direct path to a factor model. Both SVD and principal component analysis (PCA) benefit from contiguity, which eases both the time-varying construction, and uncertainty quantification. The latter is of particular importance not only to set standard errors, but also to determine the trade-offs involved in estimation under longitudinal variation: for example, how many minutes or days are required to estimate a covariance matrix, or singular vectors? The investigators also plan to develop volatility matrices for the drift part of a financial process, and their PCAs. The work on matrix decompositions will also benefit from projected results on spot estimation, which also ties in with contiguity. It is expected that the consequences of the contiguity and the HD2 inference will be transformational, leading to more efficient estimators and better prediction, and that this approach will form a new paradigm for high frequency data. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.