CIF: Small: Coding for Reliable Distributed Learning: Revamping the Design Principles

Distributed computing is a vital paradigm that harnesses the collective power of multiple machines to tackle massive computations, such as training large-scale machine learning models. This project aims to push the reliability, resiliency, and efficiency of distributed computing platforms to their fundamental limits, in the presence of slow, faulty, or adversarial servers. Recent studies have shown that incorporating dependency and redundancy in computing, using coding theoretic techniques, is a promising approach to mitigate the impact of unreliable machines. Existing solutions, however, are built on a foundation that is inconsistent with the computational requirements of machine learning, limiting their effectiveness. This project seeks to redefine coded computing from its core principles, establishing a unified and concise foundation that is deeply connected and naturally applicable to large-scale, distributed machine learning algorithms. In addition to the technological novelty, the project features curricular development, graduate student mentoring, and dissemination of results through tutorials and summer courses. Inspired by the success of error correction codes in dealing with missing and corrupted symbols in communication channels, researchers have been adapting and extending those concepts, methods, and tools to introduce coded redundancy in computing. Traditional error correction codes, however, are designed for exact recovery of messages, often embedded in a finite field. Those designs do not align with the requirements of machine learning computations and impose unnecessarily stringent constraints on the application of coded computing. This research seeks to shift the foundational principles of the coded computing from a coding theory-centric approach to a multidisciplinary framework, incorporating approximation theory, non-parametric regression, and manifold learning. This new approach seamlessly integrates with machine learning applications, offering a more natural and effective solution. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.