Choose, Don’t Label: Multiple-Choice Query Synthesis for Program Disambiguation
High-level specifications of code are inherently ambiguous, and prior systems have explored interactive techniques to help users clarify their intent and resolve such ambiguities. However, most existing approaches elicit supervision through labeled examples, which are often error-prone and may fail to capture user intent. This paper introduces a new active learning paradigm for program disambiguation based on multiple-choice queries. In this paradigm, the system presents a small set of high-level behaviors as multiple-choice options, and the user simply selects the intended one. Technically, each answer option corresponds to a Hoare triple that characterizes a cluster of semantically similar candidate programs. This formulation enables formal reasoning about the informativeness and interpretability of queries, and supports systematic construction of optimal queries. Building on this insight, we develop a new active learning algorithm and implement it in a tool called Socrates, which automatically synthesizes informative multiple-choice queries for program disambiguation. We evaluate Socrates across four domains spanning both symbolic and neurosymbolic settings and show that it produces intuitive, easy-to-answer queries and achieves efficient convergence. Most importantly, Socrates identifies the intended program more reliably than existing methods, while maintaining competitive runtime performance.
Wed 17 JunDisplayed time zone: Mountain Time (US & Canada) change
11:00 - 12:40 | Specification Synthesis and VerificationPLDI Research Papers at Flatirons 3 Chair(s): John Regehr University of Utah | ||
11:00 20mTalk | [SIGPLAN OOPSLA’25] Counterexample-Guided Inference of Modular Specifications PLDI Research Papers William Hallahan Binghamton, Ranjit Jhala University of California at San Diego, Ruzica Piskac Yale University | ||
11:20 20mTalk | Choose, Don’t Label: Multiple-Choice Query Synthesis for Program Disambiguation PLDI Research Papers Celeste Barnaby University of Texas at Austin, Danny Ding University of Texas at Austin, Osbert Bastani University of Pennsylvania, Işıl Dillig University of Texas at Austin DOI | ||
11:40 20mTalk | Presynthesis: Towards Scaling Up Program Synthesis with Finer-Grained Abstract Semantics PLDI Research Papers Rui Dong University of Michigan, Qingyue Wu University of Michigan, Danny Ding University of Texas at Austin, Zheng Guo University of Michigan, Ruyi Ji University of Michigan, Xinyu Wang University of Michigan DOI | ||
12:00 20mTalk | Verification Modulo Tested Library Contracts PLDI Research Papers Abhishek Uppar IISc Bangalore, Omar Muhammad IISc Bangalore, Sumanth Prabhu S Relyance AI, Deepak D'Souza IISc Bangalore, P. Madhusudan University of Illinois Urbana-Champaign, Adithya Murali University of Wisconsin-Madison DOI | ||
12:20 20mTalk | Expecto: Extracting Formal Specifications from Natural Language Description for Trustworthy Oracles PLDI Research Papers DOI | ||