Assessing the quality of GitHub copilot’s code generation

Simple item page
buir.contributor.authorYetiştiren, Burak
buir.contributor.authorÖzsoy, Işık
buir.contributor.authorTüzün, Eray
buir.contributor.orcidTüzün, Eray|0000-0002-5550-7816
dc.citation.epage71en_US
dc.citation.spage62en_US
dc.contributor.authorYetiştiren, Burak
dc.contributor.authorÖzsoy, Işık
dc.contributor.authorTüzün, Eray
dc.coverage.spatialSingapore Singaporeen_US
dc.date.accessioned2023-02-21T08:17:43Z
dc.date.available2023-02-21T08:17:43Z
dc.date.issued2022-11-09
dc.departmentDepartment of Computer Engineeringen_US
dc.descriptionConference Name: PROMISE 2022: Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineeringen_US
dc.descriptionConference of Date: 17 November 2022en_US
dc.description.abstractThe introduction of GitHub’s new code generation tool, GitHub Copilot, seems to be the first well-established instance of an AI pair-programmer. GitHub Copilot has access to a large number of open-source projects, enabling it to utilize more extensive code in various programming languages than other code generation tools. Although the initial and informal assessments are promising, a systematic evaluation is needed to explore the limits and benefits of GitHub Copilot. The main objective of this study is to assess the quality of generated code provided by GitHub Copilot. We also aim to evaluate the impact of the quality and variety of input parameters fed to GitHub Copilot. To achieve this aim, we created an experimental setup for evaluating the generated code in terms of validity, correctness, and efficiency. Our results suggest that GitHub Copilot was able to generate valid code with a 91.5% success rate. In terms of code correctness, out of 164 problems, 47 (28.7%) were correctly, while 84 (51.2%) were partially correctly, and 33 (20.1%) were incorrectly generated. Our empirical analysis shows that GitHub Copilot is a promising tool based on the results we obtained, however further and more comprehensive assessment is needed in the future.en_US
dc.identifier.doi10.1145/3558489.3559072en_US
dc.identifier.isbn978-1-4503-9860-2
dc.identifier.urihttp://hdl.handle.net/11693/111572
dc.language.isoEnglishen_US
dc.publisherAssociation for Computing Machineryen_US
dc.relation.isversionofhttps://doi.org/10.1145/3558489.3559072en_US
dc.source.titleInternational Conference on Software Engineeringen_US
dc.subjectGitHub Copiloten_US
dc.subjectCode generationen_US
dc.subjectCode completionen_US
dc.subjectAI pair programmeren_US
dc.subjectEmpirical studyen_US
dc.titleAssessing the quality of GitHub copilot’s code generationen_US
dc.typeConference Paperen_US

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Assessing_the_quality_of_GitHub_copilot’s_code_generation.pdf
Size:
1.13 MB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.69 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Department of Computer Engineering