Elements: Real-Time, Incremental, and Sustainable Sequence Search over SRA

The Sequence Read Archive (SRA) is a vast but underutilized repository of genomic and related data, containing the majority of publicly available sequencing experiments in raw, unassembled format. Scientists could leverage this resource to search for newly discovered genes across the entire collection of existing public experiments, enabling rapid functional characterization and enhanced biological insights that would otherwise require extensive individual dataset analysis. However, building a sustainable and scalable index over the SRA presents significant challenges due to its massive size, continuous daily growth, and diverse data types. This project addresses these issues by developing new indexing tools that will enable scientists to search the entire SRA in real time. To further enable broad usage, the project provides a hosted, web-accessible version of the calculated indices. This project develops a real-time, scalable sequence search index for SRA using innovative data structures, compression algorithms, and distributed indexing approaches designed for cost-effective deployment on commodity infrastructure. This project has three main thrusts. First, it extends the previously developed Mantis index to efficiently index abundance, positions and experiment metadata while maintaining the original performance and scalability. Second, it develops a dynamic and distributed version of Mantis to scale out and incrementally index newly deposited experiments and support real-time queries. Finally, it develops an easy-to-use Application Programming Interface (API) for both command-line and web usage to enable scientists to perform rapid and advanced biological analyses over SRA. This project seeks to (1) empower researchers to conduct large-scale analyses with flexible, easy-to-use tools; (2) improve biological analysis algorithms and tools to keep pace with the scale and growth of modern datasets; and (3) rapidly identify and quantify novel transcripts, genes, and viruses among raw sequencing datasets. This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Division of Biological Infrastructure in the Directorate of Biological Sciences. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.