HCC: Small: Direct Manipulation for Everyday Programming

Computer programming is an important skill for modern society. However, it is difficult for beginners to learn and even for experienced programmers, programming can be mentally demanding. This is because computer programs are typically represented in text-based code that can be difficult to read and understand. One way to help with understanding programs is to use visual representations of computer programs, data, and outputs. Using visual materials to explain code is common in educational materials and classrooms, and there are even some programming systems that are primarily visual. This project's key insight is that, instead of using visual materials to explain code or replace code, programming tools might allow programmers to work on linked visual and textual representations of programs, directly manipulating the graphics to understand and change the code. The goal is to fundamentally transform the nature of programming by integrating textual code with directly manipulable graphics, making coding more intuitive and accessible. Previous attempts to replace text-based programming with graphical interfaces have not met the goal of revolutionizing programming. These attempts failed because they discarded not only the weaknesses of textual code but also its strengths: textual code is economical, unambiguous, and supported by a robust ecosystem of industrial-strength packages and tooling. This project takes the view that direct manipulable graphics can augment text rather than replace it. To do this, the research team will develop a better understanding of visual representations of code, then design, build, and evaluate the first direct manipulation programming system for everyday code. Specifically, the research team will begin by investigating programmer-produced diagrams to understand what constitutes effective manipulable graphics for supporting programming. Then, the team will design a programming system that incorporates these identified qualities, and evaluate this new programming paradigm in scientific, educational, and AI interaction contexts, generating initial theories and empirical evidence regarding the suitability of direct manipulation-supported programming in these diverse settings. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.