Journal · 2009
Evaluating the Quality of Open Source Software
Diomidis Spinellis, Georgios Gousios, Vassilios Karakoidas, Panagiotis Louridas, Paul J. Adams, Ioannis Samoladas, Ioannis Stamelos
What open source changed about software quality research: the code, the version history, the bug database and the mailing lists are all public, so quality can be evaluated in the open — and comparable projects can be measured against each other.
- Published in
- Electronic Notes in Theoretical Computer Science (ENTCS), Vol. 233, pp. 5--28, 2009
- Citations
- 127 on Google Scholar, read 5 September 2026 — 1 of 30 by count
- Cite as
- SGK+09
The most-cited work — a quarter of all citations to date.
The idea
Quality research used to be either proprietary and unpublished, or done from outside with narrow black-box techniques. Open source removes both constraints: the product and the process that produced it are both visible. And because many open source projects compete in the same domain, the quality of comparable systems can be contrasted — something that was never possible with closed software. Combining historical snapshots with events like bug discoveries and fixes takes it further, from measurement to cause and effect.
Where it came from
The origin story is in the paper: an idle server, a script computing the maintainability index over a directory tree, and ten years of FreeBSD snapshots run through it. That plot — the index against lines of code, kernel and user programs — was enough to show the value of process data and to start the grant application that became SQO-OSS, a two-year, 200 person-month European research effort.
Contributions
- Motivating examples, tools and techniques for evaluating open source quality — and by extension proprietary quality too.
- The structure of Alitheia, the SQO-OSS platform: a data collection system, a computation component (the “cruncher”) and a web presentation layer.
- What distinguishes it from the project dashboards of the day — it integrates metrics from product and process sources (code, version control actions, mailing list posts, bug database entries), is not tied to one programming language, and is itself open source.
Where it sits
The most-cited work here by a distance — around a quarter of all citations to date. The platform was introduced two years earlier in the PCI 2007 paper.
Written from the paper itself — the PDF linked above, which this site hosts.