Pelican: Advancing the Open Science Data Federation Platform

The Pelican Software Suite provides a platform that facilitates sharing of research datasets. By federating their datasets across institutional boundaries, research endeavors ranging from single Principal Investigator laboratories to international collaborations make their local datasets accessible to remote clients for viewing, caching, or processing. The flagship Pelican federation is the Open Science Data Federation (OSDF) which has the reach of the entire open science community. Utilizing storage embedded throughout national and international networks, the OSDF facilitates, independently or in conjunction with distributed computing resources, effective access to remote datasets as part of diverse scientific workflows. Following Translational Computer Science methodology, the team from three institutions develops the Pelican Software Suite so the OSDF can carry the responsibilities of a national-scale pillar for dataset sharing. The distributed computing principles of Pelican enable dataset providers to exercise data sovereignty, access control, protection of storage and network capacity, and observability of usage. Pelican clients can be embedded in web browsers, Jupyter notebooks, or computing infrastructure. As a Pelican federation the OSDF can be viewed as a “transport bus,” connecting clients to dataset providers that use a diverse set of backend storage technologies. With a focus on datasets relevant to the climate community and attention to the evolving role of AI in research, the project enhances the user experience and adds new backends to the Pelican Software Suite. This addresses the ever-growing need to broaden the reach of the national Cyberinfrastructure ecosystem to new users and a wider spectrum of colleges and universities. This effort is also supported by National Discovery Cloud for Climate (NDC-C) resources. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.