TL;DR
This paper critically examines four common methods for collecting empirical data on software refactoring: mining commit logs, analyzing code histories, observing programmers directly, and logging refactoring tool usage. The authors highlight that each method has inherent biases and assumptions, which can significantly influence the conclusions drawn about refactoring practices. Understanding these limitations is crucial for researchers to choose the most appropriate method for their specific research questions.
What Problem Does This Paper Address?
To build better refactoring tools and understand software development practices, researchers need reliable empirical data from real-world refactoring sessions. However, the literature offers various methods for collecting this data, and it's unclear how the choice of method impacts the quality and generalizability of the findings. This paper addresses this gap by comparing four prominent data-gathering approaches, detailing their underlying assumptions, benefits, and drawbacks. The goal is to help future researchers select methods that align with their research objectives and to understand how methodological choices can shape our understanding of refactoring.
Methods
The paper compares four distinct research methods for gathering refactoring data:
- Mining the Commit Log: This method involves searching version control system commit messages for keywords like "refactor" or related terms. It's appealing due to the widespread availability of commit logs from open-source projects.
- Analyzing Code Histories: This approach examines sequences of source code versions to infer refactorings. Researchers might manually compare versions or automate this process using software tools that detect changes indicative of refactorings (e.g., MERGE CLASSES, SPLIT CLASS).
- Observing Programmers: This method involves directly watching developers work on software projects. It can be done through controlled experiments, ethnography, or indirectly via surveys and post-mortems. This allows for targeted observation of specific refactoring tasks.
- Logging Refactoring Tool Use: This technique leverages development environments that automatically record programmer activity, specifically focusing on the usage of refactoring tools. This method can be highly accurate for tool-assisted refactorings.
The paper's novelty lies not in introducing new data collection techniques, but in a systematic comparison and analysis of the assumptions and biases inherent in these existing methods.
Datasets
Not discussed in the paper. The paper focuses on analyzing existing methods and proposing experiments rather than evaluating them on specific datasets.
Key Insights and Ideas
- Methodological Bias is Pervasive: Each data collection method introduces its own set of biases, influencing the types of refactorings detected and the conclusions that can be drawn. For instance, commit logs might overrepresent large, explicitly tagged refactorings, while tool logs miss manual refactorings.
- Assumptions Matter: The validity of conclusions heavily relies on assumptions about programmer behavior (e.g., ability to recall, willingness to tag commits) and the nature of refactoring itself (e.g., frequency, granularity).
- No Single "Best" Method: The paper argues that the "best" method depends entirely on the research question. For example, if studying the use of a specific refactoring tool, tool usage logs are appropriate, even though they miss manual refactorings.
- Orthogonal Limitations: To mitigate bias, researchers should choose methods whose limitations are orthogonal to their research question or, ideally, combine multiple methods with complementary limitations to strengthen their findings.
Key Results
The paper doesn't present empirical results from experiments yet, as it's laying the groundwork for future research. Instead, it provides:
- Detailed analysis of assumptions and disadvantages for each of the four methods.
- A framework for understanding trade-offs: Table 1 summarizes how methods differ in identifying refactoring events and the level of detail captured, linking them to "context" (explaining why) and "fidelity" (capturing what).
- Nine specific hypotheses about refactoring practices and the reliability of different data collection methods, along with proposed experimental designs to test them (detailed in Table 2). For example, one hypothesis is that commits labeled "refactor" do not indicate significantly more refactoring instances than those without the label.
Limitations and Future Work
The paper acknowledges that the research community has not yet validated the assumptions underlying these data collection methods. The primary limitation is the lack of empirical validation for these assumptions.
Future work, as outlined by the authors, involves conducting the experiments proposed in Table 2 to test the hypotheses. These experiments aim to empirically validate the assumptions and quantify the biases associated with each method. The ultimate goal is to provide researchers with a clearer understanding of how to choose and use refactoring data collection methods effectively, leading to more robust and reliable research in the field.
Summary updated