Extra Virgin Olive Oil Judging – A little more than just tasting
admin | August 24, 2010A couple of weeks ago I conducted a workshop for the judges from the Royal Perth Extra Virgin Olive Oil Show. It was the 4th such workshop conducted annuallly at the request of the Western Australian Olive Council. The judging workshop program evolved out of the belief that whilst show judges are experienced EVOO tasters – the act of show judging requires a variety of skills which are required to bring it all together –
- Sensory – of course.
- Linguistic- for writing useful comments for exhibitors, and
- Interpersonal – as judges work in small teams of 3 or 4.
Every year I concentrate on building a particular skill required by show judges. Themes have included – Measuring judge consistency, writing effective comments for the benefit of exhibitors, and understanding and dealing with diversity of EVOO styles and characters.
Judge Consistency
One of the most important requirements of a good show judge is the ability score consistently. Agreeing with yourself is a pretty good quality to have. One would hope that a judge that gives an oil a gold medal score when tasted on one occasion would give it another gold medal score if they were asked to score it on another occasion. While it would be nice to think that this could be done by all show judges all the time, I have to say that believing so is a little naïve. A couple of years ago I measured the consistency of over 500 wine show judges (and potential wine show judges) and found that a surprising number found scoring consistently was a tricky business (Gawel and Godden, 2008). In my opinion, EVOO judging is more difficult than wine judging (I’ve done quite a bit of both) as the difference between an average quality and a high quality EVOO is much smaller than between the average and high quality wine. So it’s no surprise that consistency scoring EVOO’s for quality isn’t as easy as many people might think.
So with the aim of measuring the consistency of the olive oil judges I presented them with 12 EVOO’s which they scored for quality. Later on that day, they were presented with 15 oils, 12 of which were the same as those scored in the morning – but in a different order of course. The judges were unaware that oils were being presented for a second time as I led them to believe that we were conducting a different exercise (I can be a complete bastard!). I then correlated the scores given on the first occasion with those given on the second to the same oils. The results of the exercise allowed the judges to benchmark themselves against their peers.
Appreciating Style Differences
Every year I spend a lot of time at these workshops presenting diverse sets of EVOO’s – from different varieties and blends – early and later harvest – from different regions – mild through to very robust . All this is done to impress upon judges that it is bad form to penalise an oil because it either has or doesn’t have, a particularly flavour characteristic. The world of EVOO is wonderfully diverse – different flavour profiles and different levels of bitterness and pungency are necessary. Why? – because despite what some producers might tell you, no single style of EVOO can hope to be perfectly suited for use in every single culinary dish. While some styles might have a limited application compared with other styles, they all have their place ‘out there’ in one way or other. But there are high and low quality examples of each style, and recognising this is the point of the judging exercise.
So here were the results of the 2009 Perth Show which took place two weeks after the workshop. I’ve given the average score for oils within polyphenols ranges. The higher the polyphenols level, the more bitter and/or peppery the oil (generally speaking), so the polyphenol level is a pretty good indicator of style. I’ve also turned it around and given the average polyphenols level for each medal ranges, best of class and best of show.
Polyphenol
Range |
Average
Score |
<200 | 13.7 Bronze |
200-299 | 13.8 Bronze |
300-399 | 13.4 Bronze |
>400 | 13.6 Bronze |
Medal | Average polyphenols
(mg/kg) |
No Medal | 350 |
Bronze | 320 |
Silver | 290 |
Gold | 430 |
Best of Class* | 410 |
Best of Show | 310 |
*average of the highest gold medal scoring oil in each class
Lastly, if you’re like me and like seeing things in pictures, here is the plot of awarded score against polyphenol content.
The correlation between polyphenols level and score was a meagre 0.005 which was as close the magical random, ‘no relationship whatsoever’ value of 0 as you would hope to get. In short on all the measures, the amount of bitterness/pepperyness had no bearing whatsoever on the scores awarded to the oils. This is the way it should be. Good oils should be awarded based on quality not style. Fluke perhaps? – well I just spent some time on the flight back from the 2010 Royal Perth Show and calculated that the correlation between score and polyphenols level again showed that there was no relationship between score and polyphenols level (aka style) – the correlation was 0.001
Writing Descriptions
Unlike EVOO shows elsewhere, the majority of Australian shows provide commentary on each submitted oil via the published show results booklet. The comments are written by the head judges of each panel, and are based on their opinion and those of the other two judges. However, the comments can get a bit jargonny at times, and while they clearly mean something to the judges who wrote them, they may not mean much at all to the exhibitor. On the other side of the coin, the comments can be so generic that they mean little at all.
So in one workshop I presented a series of comments written by judges in another show (The Australian National), and simultaneously presented the oils to which they referred. The judges attending the workshop were asked to match the comments to the oils after tasting them. We then discussed at length exactly what aspects of the comment made it easy to identify the oil, and what aspects of the comment was confusing or resulted in an incorrect match of description to oil. The logic here was, well if you can identify the oil from its description then the description must be informative. After picking apart the value of the provided descriptions, the judges rewrote the comments in a way which they felt made for the briefest yet most informative comment. While I’d admit that the art of writing brief but useful commentary on oils is a work in progress, it was a good start.
Reference
Gawel, R. and Godden, P.W. (2008) Evaluation of the consistency of wine quality assessments from expert wine tasters. Australian Journal of Grape and Wine Research, 14, 1-9.
Hi Richard,
One question I have had for a while about olive oil tasting competitions is the nature of the medal awards. as a naïf, one might think that an oil that had been awarded a gold, silver, or bronze medal at a competition would have been either the best, second-best, or third-best (respectively) in the show as a whole or at least in some category. But instead, I see that there are routinely multiple silver and bronze medal winners, and occasionally even more than one gold, within a category. On what basis is a medal category awarded, and why do the competitions use such a potentially-misleading nomenclature?
Well, at least your comment feature is getting used 😉 .
Hi Michael
The gold, silver, bronze awards were never meant to mean best, second best, and third best. Gold refers to oils that are “excellent”, silver to “very good” and ” bronze” to above average. Well that is what Australian show judges are instructed to interpret things, and I’m sure something similar goes on elsewhere. When you think of it the first past the post system isn’t as fair as it seems. For example you could have a weak class with three pretty average oils being the best of a pretty crappy lot. Using the first past the post system, the judges would be obliged to give a gold to not such a great oil. The class next to it could have three absolute crackers in in there, but the third outstanding oil would be relegated to a bronze. So the gold in the first class might not even be as good as the bronze in the second. So giving lots of golds to very strong classes, and no golds to weak classes isn’t such a bad idea. Incidentally the difference in quality between classes in shows can be substantial.
Incidentally, a lot of shows give best of class awards, and some give best, second and third best of class. However most have a minimum standard for these. Usually a gold medal is a requirement for the best of class being awarded, however in some regional shows with smaller numbers of entries, occassionally a silver medal is the minimum quality level for best of class. So for those who are interested in the best oil can look for the best of class.
RG