Software systems researchers want to make human-centered claims, but don’t have the proper tools to do so. That’s how we ended up with the ubiquitous lines-of-code comparison found in evaluation sections everywhere. In this post, I explore: what can we reasonably expect researchers to do that’s higher quality than LOC but less effort than a full-blown user study?
Drawing on numerous examples from papers published in PLDI 2019, I argue that scatter plots can be easier to understand than bar charts.
Drawing on numerous examples from papers published in PLDI 2019, I argue that normalised data should usually be plotted on a logarithmic scale.
To avoid an empirical replication crisis in programming languages research, PL researchers should employ the best scientific practices for empirical evaluation. A SIGPLAN empirical evaluation committee has assembled a checklist to help.
