Cleaning ground truth data in software task assignment

Simple item page
buir.contributor.authorTüzün, Eray
buir.contributor.authorMoran, Cansu
buir.contributor.orcidTüzün, Eray|0000-0002-5550-7816
buir.contributor.orcidMoran, Cansu|0000-0003-2101-1449
dc.citation.epage106956- 14en_US
dc.citation.spage106956- 1en_US
dc.citation.volumeNumber149en_US
dc.contributor.authorTecimer, K. A.
dc.contributor.authorTüzün, Eray
dc.contributor.authorMoran, Cansu
dc.contributor.authorErdogmus, H.
dc.date.accessioned2023-02-17T10:49:51Z
dc.date.available2023-02-17T10:49:51Z
dc.date.issued2022-05-25
dc.departmentDepartment of Computer Engineeringen_US
dc.description.abstractContext: In the context of collaborative software development, there are many application areas of task assignment such as assigning a developer to fix a bug, or assigning a code reviewer to a pull request. Most task assignment techniques in the literature build and evaluate their models based on datasets collected from real projects. The techniques invariably presume that these datasets reliably represent the “ground truth”. In a project dataset used to build an automated task assignment system, the recommended assignee for the task is usually assumed to be the best assignee for that task. However, in practice, the task assignee may not be the best possible task assignee, or even a sufficiently qualified one. Objective: We aim to clean up the ground truth by removing the samples that are potentially problematic or suspect with the assumption that removing such samples would reduce any systematic labeling bias in the dataset and lead to performance improvements. Method: We devised a debiasing method to detect potentially problematic samples in task assignment datasets. We then evaluated the method’s impact on the performance of seven task assignment techniques by comparing the Mean Reciprocal Rank (MRR) scores before and after debiasing. We used two different task assignment applications for this purpose: Code Reviewer Recommendation (CRR) and Bug Assignment (BA). Results: In the CRR application, we achieved an average MRR improvement of 18.17% for the three learning-based techniques tested on two datasets. No significant improvements were observed for the two optimization-based techniques tested on the same datasets. In the BA application, we achieved a similar average MRR improvement of 18.40% for the two learning-based techniques tested on four different datasets. Conclusion: Debiasing the ground truth data by removing suspect samples can help improve the performance of learning-based techniques in software task assignment applications.en_US
dc.description.provenanceSubmitted by Ezgi Uğurlu (ezgi.ugurlu@bilkent.edu.tr) on 2023-02-17T10:49:51Z No. of bitstreams: 1 Cleaning_ground_truth_data_in_software_task_assignment.pdf: 1926016 bytes, checksum: 9ab239180de7b8d17e6dcfa89767dfce (MD5)en
dc.description.provenanceMade available in DSpace on 2023-02-17T10:49:51Z (GMT). No. of bitstreams: 1 Cleaning_ground_truth_data_in_software_task_assignment.pdf: 1926016 bytes, checksum: 9ab239180de7b8d17e6dcfa89767dfce (MD5) Previous issue date: 2022-05-25en
dc.embargo.release2024-05-25
dc.identifier.doi10.1016/j.infsof.2022.106956en_US
dc.identifier.eissn1873-6025
dc.identifier.issn0950-5849
dc.identifier.urihttp://hdl.handle.net/11693/111501
dc.language.isoEnglishen_US
dc.publisherElsevier BVen_US
dc.relation.isversionofhttps://doi.org/10.1016/j.infsof.2022.106956en_US
dc.source.titleInformation and Software Technologyen_US
dc.subjectTask assignmenten_US
dc.subjectCode reviewer recommendationen_US
dc.subjectBug assignmenten_US
dc.subjectGround truthen_US
dc.subjectLabeling bias eliminationen_US
dc.subjectSystematic labeling biasen_US
dc.subjectData cleaningen_US
dc.titleCleaning ground truth data in software task assignmenten_US
dc.typeArticleen_US

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Cleaning_ground_truth_data_in_software_task_assignment.pdf
Size:
1.84 MB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.69 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Department of Computer Engineering