RI:Small: Modeling and Relating Visual Tasks

Deep networks trained on massive visual datasets are being used in an increasing number of applications in fields such as autonomous driving, robotics, manufacturing, e-commerce, and science and engineering disciplines. However, exploring the vast space of solutions for a new problem can be difficult as it demands significant computational resources. There is a pressing need for tools that improve understanding of the extent to which solutions from one task generalize to new tasks, along with methods for sharing the expertise needed to design these solutions. This project aims to tackle these challenges by developing a framework to model, relate, and visualize recognition tasks across a broad range of visual domains. Doing so will enable practitioners to identify closely related datasets for application across tasks and to select deep network architectures for pre-training. The project will examine practical applications of the framework and examine methods for detecting and adapting to statistical shifts that take place in long-term deployment of machine-learning models. Specifically, the project will look at efficient solutions for problems in Ecology and Civil Engineering domains. The educational impact of the project includes teaching, mentoring graduate and undergraduate students through research activities associated with the project, and mentoring underrepresented undergraduates in computing through the University's Early Research Scholars Program. This project aims to create a general framework for representing a variety of visual recognition tasks and their relationships. Specifically, the research team will develop: 1) A theoretical framework to embed tasks into Euclidean and hyperbolic vector spaces (creating “task embeddings”) by evaluating the importance of the parameters of deep networks employed to solve them; 2) Efficient methods for computing task embeddings for networks containing millions, or even billions, of parameters; 3) Techniques to leverage unlabeled data to enhance task embeddings when label availability is limited; 4) Techniques to compute task embeddings for dense visual prediction tasks such as object detection and image segmentation; 5) The application of task embeddings to address meta-tasks such as dataset selection, multi-tasking, and detecting task shifts; 6) Visualization of symmetric and asymmetric relationships between tasks represented by widely used computer vision datasets. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.