CAREER: Interactive Program Synthesis for Web Automation

Process Automation continues to be a main driver of digital transformation. Yet, it is still technically very demanding to create such automation programs. This project aims to significantly lower the technical barrier of creating web automation programs. The project’s novelties are a suite of new rewrite-based program synthesis algorithms that can automatically generate programs from user demonstrations. The project’s impacts are to enable non-experts in need of performing a tedious but programmatic web-related task to create a program to automate the work, even if they have little or no background in web programming. At the core, this project is developing algorithms that take as input a user demonstration – in the form of a trace A of user actions (e.g., clicking buttons, scraping data) – and synthesize a parameterized program P with control-flow structures by rewriting A to P. The approach is based on finite tree automata and involves both neural and symbolic elements in the underlying web automation language, with the goal of being able to effectively reason about the webpage contents while still leveraging the underlying webpage structure. The project is curating a new suite of web automation tasks that are independently useful for future research beyond this project. The investigator is collaborating with partners at Michigan (e.g., Women in Science and Engineering (WISE), Michigan Louis Stokes Alliance for Minority Participation (MI-LSAMP) and Engineering Center for Academic Success (ECAS)) to increase participation of students from underrepresented groups in his research. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.