Testing LLM-Generated Distributed Protocol Code
Large Language Models (LLMs) are increasingly used for code generation, including complex distributed systems. However, their reliability in stateful and failure-prone environments remains unclear. In this paper, we present a structured evaluation framework for assessing LLM-generated implementations of distributed protocols, including Two-Phase Commit, Ring Election, and Raft. We introduce a simulation environment that injects realistic network faults such as message loss, delay, and duplication. Our results show that while LLMs perform well on simpler protocols and code completion tasks, they struggle with complex consensus algorithms and exhibit inconsistent debugging behavior. We further analyze the relationship between model size and performance, finding that parameter count alone does not predict effectiveness. These findings highlight the need for systematic testing and verification in agentic code generation systems.
Mon 15 JunDisplayed time zone: Mountain Time (US & Canada) change
10:40 - 12:20 | |||
10:40 20mTalk | Agentic Code Reasoning PAgE Pre-print | ||
11:00 20mTalk | Towards Verified Code Reasoning by LLMs PAgE Meghana Aparna Sistla Google DeepMind, Gogul Balakrishnan Google, Patrick Rondon Google, José Pablo Cambronero Google, USA, Michele Tufano Google, Satish Chandra Meta Platforms, Inc. | ||
11:20 20mTalk | Testing, Credible Compilation, and Verification in the Axon Verified Compiler in Lean and Claude CodeRemote PAgE Martin C. Rinard Massachusetts Institute of Technology | ||
11:40 20mTalk | The Next Frontier for AI-Generated Kernels: Correctness PAgE DOI Pre-print | ||
12:00 20mTalk | Testing LLM-Generated Distributed Protocol Code PAgE | ||