Program Comprehension Challenges Detection for Pull Requests with Machine Learning

Abstract

The importance of program comprehension in software development is such that software engineers spend more time, 35% to 70% of their time, understanding code than writing it. With pull requests (PRs) becoming the default development collaboration model, code review is becoming more common. There is a paucity of studies focusing on the challenges concerning program comprehension. This study aims to provide an insight into those challenges during code reviews, and to, specifically, detect comprehension challenges with the application of machine learning (ML) to analyze a large number of GitHub PR review comments more efficiently, and investigate the common causes of their occurrences to serve as suggestions as to the perils to avoid during code reviews. In this context, comprehension challenges are focused on instances when a code reviewer seeks information on the program behavior, the intention of a code change, who has experience with the code, or why the code is implemented a certain way. To evaluate how accurately ML can be used to detect comprehension challenges noted in review comments, an ML classifier was developed with the combination of a linear support vector machine with stochastic gradient descent learning and natural language processing of dialogue act classification, stop words removal, lemmatization, and term frequency-inverse document frequency. To investigate the causes of comprehension challenges, a sample of 748 review comments were analyzed through statistical hypothesis tests, and a content analysis was conducted to compare 384 review comments without comprehension challenges and 384 samples with comprehension challenges. The performance results of the ML classifier showed a 74.3% precision and a 66.7% recall. The quantitative and qualitative analysis showed that the lines-of-code-changed metric made no significant difference to program comprehensibility, and review comments with comprehension challenges were associated more frequently with discussions on bottom-up knowledge. These results suggest that ML can help to detect comprehension challenges and issues and provide analytics to software development teams to support them to prioritize areas of improvement. Furthermore, missing top-down knowledge regarding programming plans is the fundamental reason for many comprehension challenges.