Koyuncu, AnılOuedraogo, Wendkuuni C.Kabore, KaderTian, HaoyeSong, YeweiKlein, JacquesLo, DavidBissyande, Tegawende F.2025-02-212025-02-212024-11-01979-840071248-7https://hdl.handle.net/11693/116554Conference Name: Proceedings - 39th ACM/IEEE International Conference on Automated Software Engineering, ASE 2024Date of Conference: 28 October 2024 - 1 November 2024Unit testing, essential for identifying bugs, is often neglected due to time constraints. Automated test generation tools exist but typically lack readability and require developer intervention. Large Language Models (LLMs) like GPT and Mistral show potential in test generation, but their effectiveness remains unclear.This study evaluates four LLMs and five prompt engineering techniques, analyzing 216 300 tests for 690 Java classes from diverse datasets. We assess correctness, readability, coverage, and bug detection, comparing LLM-generated tests to EvoSuite. While LLMs show promise, improvements in correctness are needed. The study highlights both the strengths and limitations of LLMs, offering insights for future research. © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.EnglishCC BY 4.0 DEED (Attribution 4.0 International )https://creativecommons.org/licenses/by/4.0/Automatic test generationEmpirical evaluationLarge language modelsPrompt engineeringUnit testsConference Paper10.1145/3691620.3695330