GOALI: A Community-Driven Benchmarking Framework for Trustworthy Generative AI in Architecture, Engineering and Construction

Generative artificial intelligence tools are expanding rapidly across the architecture, engineering, and construction industry to assist with engineering design, simulation, cost estimates, contract development and regulatory compliance. However, these systems can produce convincing but incorrect results that are difficult to detect, posing significant risks when used in safety-critical engineering applications. An incorrect structural calculation or a misinterpreted building code, for example, could compromise the safety of buildings and their occupants. This Grant Opportunity for Academic Liaison with Industry (GOALI) award supports research that will develop a community-driven framework for evaluating and validating generative AI tools before they are deployed in real-world practice. By establishing standardized methods to assess the reliability, accuracy, and engineering validity of AI-generated outputs, this research will help improve public safety, reduce engineering errors, and support more trustworthy use of AI in the built environment. The project will also strengthen the engineering workforce by preparing students and practicing engineers to critically evaluate results generated by artificial intelligence rather than accept them uncritically. The resulting methodologies are expected to be broadly applicable to other safety-critical domains where AI systems support engineering and operational decision making. The project develops a community-driven framework for evaluating trustworthy generative AI in architecture, engineering, and construction through three integrated research thrusts. The first develops domain-specific ontologies and engineering-aware evaluation metrics to assess AI performance across diverse tasks, output modalities, and engineering criteria. The second establishes scalable methodologies for generating, validating, and continuously expanding benchmark datasets using automated data generation, expert verification, and community contributions. The third validates the framework through industry-grounded benchmark applications involving engineering simulation and geometric modeling. Together, these efforts establish a generalizable methodology for evaluating generative AI in safety-critical engineering applications while providing an extensible benchmarking infrastructure that can be adapted to other engineering disciplines. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.