Collaborative Research: HCC: Medium: Tools for Thought: Augmenting Divergent, Convergent, and Cooperative Work with Large Language Models

A long-standing goal of computing research is to create "tools for thought", in which computers extend our abilities to think and communicate in both work and social contexts. Used carefully, large language models (LLMs) -- and their remarkable ability to process and generate text -- can contribute to this goal. Already, people use LLMs to generate and organize ideas, summarize documents, support writing, plan events or meals, generate computer programs, and analyze data. However, current LLM usage prioritizes conversational "chat" interactions involving a single person and one-at-a-time responses, whereas creative work requires considering a variety of possibilities and may include multiple collaborators. The goal of this project is to leverage and evaluate LLMs as "tools for thought" that support creative, open-ended, and collaborative work. The main aims are to (1) integrate LLMs into larger, interactive systems while safeguarding LLM output quality, (2) help people generate and consider diverse, relevant ideas, and (3) support collaborative work involving multiple people and LLMs interacting together. This project looks beyond current chat-based interactions to leverage LLMs to support people's everyday work in a reliable and effective manner. More specifically, this project develops novel methods, evaluations, and applications to better leverage LLMs as tools for thought in both single-user and cooperative scenarios. The main approach is to scaffold LLM-powered systems to provide higher control and reliability, while focusing on a key step of open-ended information work: "divergent" phases of generating diverse yet relevant candidate ideas, followed by "convergent" phases in which one navigates, selects, and synthesizes the most promising ideas. The first objective of this project is to develop a design space and guidance for building more reliable and controllable LLM workflows, drawing upon over a decade of crowdsourcing research and documenting the adaptations necessary to build effective workflows and evaluate LLM capabilities. The second objective is to enable cycles of divergent and convergent work: developing robust operations for generating diverse yet relevant candidates -- whether they be writing suggestions, brainstorming ideas, or salient quotes to extract from a text -- alongside methods for choosing among and combining responses. The third objective expands this focus to cooperative projects, enabling hybrid multi-user/LLM workflows and investigating how LLMs could improve awareness and coordination among collaborators. In support of these objectives, the project will develop and evaluate user-facing applications for tasks such as scientific writing, text analysis, and design ideation, providing practical examples of LLM-supported "tools for thought". This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.