An empirical analysis of issue templates on GitHub

Abstract

Many open-source software projects use GitHub Issues for issue tracking. Unlike other issue trackers, the initial versions of GitHub Issues were highly flexible and had no standard way of using it. Its unstructured nature may have made it prone to incomplete issue reports that may negatively affect software development and maintenance productivity. To potentially address these problems, GitHub introduced issue templates in 2016. This thesis aims to reflect the current status of issue template usage by mining open-source projects. Also, we analyze how the templates have evolved since their introduction in 2016 and further investigate the impact of issue templates on several issue tracking metrics, such as time to resolution, the number of reopens, and the number of comments. We evaluated 350 templates and their previous versions from 100 large-scale and popular open-source projects. We also analyzed 1,916,057 issues to understand their conformance to templates and the impact of issue templates. Lastly, we conducted a survey with open-source software maintainers to understand their opinions about issue templates. We found that issue templates are almost always used (99 out of 100 projects). The historical analysis suggests that issue forms, which are more structured issue templates, started to gain popularity over vanilla issue templates. We also observed that issues created when the project has an issue template are statistically resolved faster (p-value 0.00, effect size 0.59) and have less number of comments. Similarly, when issue forms are used, time to resolution, the number of reopenings and the length of discussion significantly decrease. According to the survey, 85% of project maintainers agree with the benefits and they believe issue templates construct a balanced midpoint between the flexibility of vanilla issues and the strictness of other issue tracking systems such as Jira.