A recently published UCLA study of eco-certified wine quality has generated a bit of discussion this week, with both positive and negative reactions. Jamie Goode describes the paper and some of its flaws well here, while Blake Gray’s article focuses on score inflation and, to me, feels much less rational. Indeed, the opening line of Blake’s article seems to purposefully conflate the notions of statistical mean and a wine being “average”, and I have to ask myself, to what purporse?
The study purports to show that eco-certified wines obtain higher scores in three influential wine publications (Wine Advocate, Wine Spectator, Wine Enthusiast). It’s a really jumbled up paper; there are results described in the methods, the authors enter into discussion when describing their results, tables are poorly described, important results are squirrelled away in an appendix, and so on. Apologies to the authors for expressing this opinion (which is based on first-hand experience writing and reviewing scientific papers, by the way), but The Journal of Wine Economics really needs to go back and see who reviewed this paper prior to publication. And then consider whether or not they were doing their job properly. It’s an interesting paper, but it needs further editorial review and a bit of a rewrite.
This makes it difficult to read; all the same, I spent an hour or two trying to pick it apart this morning. The paper has been reported as producing a 4-point rise in scores of eco-certified wine. As Jamie pointed out, this isn’t true because of the score conversion the authors undertook in order to line up the three publications, which had different score distributions. The true result is actually buried in an appendix, which is that the regression method suggested being eco-certified gave wines a 0.46-point advantage. Just less than half a point, in other words, for all that hard, organic, biodynamic work.
That isn’t the end of it though. This half-point result was arrived at through a statistical method known as regression, in which the authors attempted to develop a model which explained the scores of the wines. Regression (of any sort) is a statistical method which should be viewed with a very wary eye. While being eco-certified conferred an advantage when analysed in this manner so did other factors, while other factors had a negative effect on score, some of which can be interpreted in really interesting ways but which I don’t want to digress on here.
These negative factors may be very important. Why do I say that? Here’s why; despite the way in which the results have been presented by the authors, and by the press who have seized upon the eco-favourable result – eco-certified wines actually scored lower in the three publications. Eco-certified wines scored 47.8% (on the author’s scaled system) whereas conventional wines scored 50%. And this didn’t appear to be statistically significant, (or at least the authors didn’t state one way or the other), and to me it seems this is the most reliable aspect of the paper. But writing “eco-certified wine scores no different to conventional wine scores” isn’t much of a headline, is it?