A couple of years ago, it occurred to me it might be possible to compare open source ERPs, ranking them not by total number of downloads (a number which is easily skewed and/or misinterpreted) but instead by measuring how responsive we are to our respective communities. To get a reliable unit for measuring responsiveness, I decided to focus on bug reports. I liked bug reports because a) they are easily quantifiable and b) the information about them is readily available on open source ERP community websites.
There were three basic data points I was looking at:
- Total bugs reported (all time)
- Total bugs resolved/closed (all time)
- Total open bugs (current)
Once I'd collected this information, I did some basic division (total resolved / total reported) to arrive at what I started calling the responsiveness ratio. To my mind, this ratio is like a batting average in baseball. The better you're responding to incoming bug reports, the higher your average will be.
Here at xTuple, we've been "batting" for almost 12 years now. During that time, responding to our community has been a priority for us. I've always sensed intuitively that we are doing well at that. But until I started measuring our responsiveness, I didn't realize how well we were actually performing.
Like the other open source ERP I looked at for this study, the xTuple bug list is publicly available. That's part of the beauty of open source: its transparency. None of us is hiding this information. Anyone is free to inspect it, analyze it, and reach their own conclusions.
My initial findings in 2011 both surprised and impressed me. xTuple's responsiveness ratio turned out to be .957. To put that in some perspective, that meant that since the beginning of time (2002), xTuple developers had resolved and/or closed almost 96% of the bug reports they received. That .957 batting average was enough to make xTuple the clear leader in the league of open source ERPs.
Fast forward to October 2013. My statistics were two years old, and I decided to dust them off and see where things stand now. Was xTuple continuing to perform at the previously high level? How did we compare to other projects? The following table shows the current results for xTuple:
Open Source ERP Responsiveness Rate || Click on infographic above to enlarge
| As of Oct. 18, 2013 | |
| Bugs reported | 10,942 |
| Bugs resolved/closed | 10,320 |
| Open bugs | 622 |
| Responsiveness ratio | .943 |
Looking at the stats, it was clear xTuple was continuing its tradition of responsiveness. The company's ratio was down only .014 points over the two-year period — not bad, considering xTuple has been expanding its product line, including the addition of an entirely new Mobile Web client.
I turned next to OpenERP. Two years ago I looked for but had trouble locating information related to OpenERP bug reports. But this year I found the information easily on their site:
| As of Oct. 23, 2013 | |
| Bugs reported | 20,227 |
| Bugs resolved/closed | 15,281 |
| Open bugs | 4,946 |
| Responsiveness ratio | .755 |
By this measure, xTuple is almost 20% more responsive to its community than OpenERP. While it's true OpenERP's bug volume is nearly double that of xTuple's, the number of bugs isn't the key data point. The goal is to measure how responsive we are to the bugs being reported to us — and there's clearly a gap between the projects here.
Next I took a look at Openbravo. I found their responsiveness to be better than OpenERP's — but still trailing xTuple.
| As of Oct. 18, 2013 | |
| Bugs reported | 20,573 |
| Bugs resolved/closed | 19,015 |
| Open bugs | 1,558 |
| Responsiveness ratio | .924 |
Openbravo's responsiveness ratio two years ago was .878, so that's a meaningful improvement. Good for them!
The last project I looked at was ADempiere, the only of these four without a primary company coordinating the project. Their figures do not seem very robust, but I think that can be explained by their recent migration to a new bug tracking system. Legacy bug reports appear not to have survived the move. Two years ago I pegged ADempiere's responsiveness ratio at .856. As of now, they are significantly behind that previous high mark.
| As of Oct. 18, 2013 | |
| Bugs reported | 357 |
| Bugs resolved/closed | 217 |
| Open bugs | 140 |
| Responsiveness ratio | .608 |
Clearly, responsiveness to incoming bug reports is not the only measure by which open source ERPs should be judged — but I do believe the responsiveness ratio gives us one very valid metric to consider. I'm proud of the job we're doing here at xTuple, as we strive to be responsive to our community. Along those lines, if you have thoughts or comments on this methodology, please feel free to share in the comments below. Thanks!