The Value of Wine Competitions Addressed
at AAWE Conference

June 28, 2010
by Roger Lansing

The value of wine competitions for wine producers may be in the advertising and promotion received by medal winners when comparing the cost of entry fees in relation to more traditional advertising and marketing costs. A competition medal could particularly benefit smaller or lesser known brands that are less likely to receive wine critic ratings, may not have a distribution and sales network, and may not have the means to pay for traditional media advertising. This was one point that came out of a panel discussion, "The Value of Wine Competitions," held during the 4th Annual Conference of the American Association of Wine Economists (AAWE) at UC Davis this past weekend. The panel featured one wine competition representative, two critics of wine awards who have authored studies published in the AAWE Journal of Wine Economics, and a respected wine journalist. The discussion actually focused more on controversial findings in studies of wine competitions that have highlighted wine judge inconsistencies in awarding medals.

Panel speakers included: G.M. "Pooch" Pucilowski, California State Fair Wine Competition manager and chief wine judge; "Fearless Critic" Robin Goldstein, author of "The Wine Trials;" Robert Hodgson, professor emeritus from Humboldt State University, winemaker and owner of Fieldbrook Winery and a former State Fair competition judge; and journalist and author George Taber, the Time magazine writer who attended the 1976 "Judgment of Paris," and more recently wrote a book with the same title.

The session moderator was Orley Ashenfelter, professor of economics at Princeton University, and a co-editor of the Journal of Wine Economics (JWE). He observed, "We know that the prices of high-end wines are highly correlated with scores, especially ratings by Robert Parker, and several published papers have examined this." He indicated that top awards in wine competitions can also increase demand for wines, and allow their producers to increase bottle prices over time.

Goldstein has written guide books to inexpensive wines under $15 and has studied the relationship between wine price and taster preferences. He gained notoriety for creating a fictitious restaurant and wine list and submitting it to Wine Spectator magazine for its Award of Excellence program. When the phony restaurant actually received an award, Wine Spectator's evaluation methods came into question., Ashenfelter said of Goldstein, "If there's an enemies list at Wine Spectator, he's got to be number one on it."

Goldstein said one way to look at wine competitions is as a form of advertising. The producer pays an entry fee to participate, it helps promote the product when a medal is received, and he noted, "It may be a more effective means of advertising than traditional means, as it gives the product a perception of quality based on impartial judging." He also observed, "The results of wine competitions are unpredictable enough, that if you enter enough of them, you're likely to wine an award eventually."

Goldstein cited recent studies regarding what he calls "the placebo effect" for consumers who are given a set of wines with different prices. When told which samples are higher priced, these wines give the consumer more pleasure based on analysis of their brain activity during sampling. He said the same placebo effect can happen for consumers when tasting wines with high ratings and wines winning major awards at wine competitions.

Goldstein co-authored a JWE article from a study based on 6,000 blind tastings, "Do More Expensive Wines Taste Better? Evidence From a Large Sample of Blind Tastings." The study concluded that when individuals are unaware of wine price, they do not derive more enjoyment from more expensive wines. Goldstein said, ideally, consumers should learn their own preferences over time and choose their own wines without being inluenced by prices, ratings, and awards. As consumers become more aware of studies about the placebo effect, and skeptical due to inconsistencies by wine judges and critics, more consumers may move toward making their own decisions in wine purchases.

However, it's also human nature to want a connection with a product based on other background.factors. Goldstein explained: "We're a civilization of storytellers. We all just want a good story, and 100-point wines and double gold medal wines are all great stories. Without a story, all we're doing is getting drunk. With a story, we're still getting drunk, but we're also turning something physical into something more spiritual."

Analyzing and Improving Judge Consistency
Hodgson became skeptical about wine competitions when he entered his own wines in multiple competitions. The same wine could win a gold medal in one competition, but win nothing in another competiton. One of his published JWE articles, "An Analysis of the Concordance Among 13. U.S, Wine Competitions," found that of 2,440 wines entered in more than three competitions, 47% received gold medals in one of the competitions. But 84% of these same wines received no award in another competition. The study concluded, "Winning a gold medal is greatly influenced by chance alone."

Hodgson served as a judge in the California State Fair competition, and is now on the competition's Wine Advisory Task Force working with Pucilowski to try to improve judging quality and consistency.
With Pucilowski's assistance, Hodgson has been evaluating the competition judges since 2005 with trials that place three samples from the same wine bottle in one flight of judged wines to see if the judges ranked each sample consistently. Hodgson, who taught statistics at Humboldt State University, said, "Fewer than 10% of judges could judge the three wines and maintain consistency in their scores." He added, "Some of the same wines received ratings that ranged from no award to gold." When the study, "An Examination of Judge Reliabiity at a Major U.S. Wine Competition," was published in the JWE, it received significant media attention and created a stir among wine judges and within the wine industry.

Pucilowski, who has managed the State Fair competition 25 years and often serves as a judge in other competitions, openly admits that his competiition and all wine judging events are highly subjective. To his credit, he is constantly looking at ways to improve the competition and to help judges improve their abilities. He said the competition is the only one that requires judges to pass a qualification exam. He asks all judges beforehand to list their favorite varietals they prefer to judge, and to list the varietals they do not want to judge. Over the years, he's learned that judges can be influenced when given too much information, such as bottle prices and the wine region of origin. Now he only reveals the wine varietal/category and vintage. Judges are also required to arrive the day before judging begins to attend a wine education event. "I just want my judges to be better," Pucilowski said. He recently began managing another wine competition judged by consumers. Based on wine sensory studies done at UC Davis, it is recommended that wine tasters wait at least 2 minutes between tastes to clear the pallet before evaluating the next wine. Pucilowski said, "I used this method with the consumer tasting, in fact, I had a stopwatch and made them wait three minutes between tastes. This is not done in commercial wine competitions, but I'm considering it for the future at the State Fair competition."

Wine judging will remain an inexact science. Hodgson has analyzed his data from the State Fair experiments from mulitple perspectives, including the judge's gender, their experience, credentials, and other factors, but he said, "I've looked for years for patterns between judges and the results, but have not seen anything stand out." Looking to the future, Hodgson said the competition task force is looking at revising the scoring system to be more equitable, "given that not all judges are created equal."

Wine Judging with Historical Significance
George Taber was the only journalist who attended the now famous (and infamous) 1976 "Judgment of Paris" tasting event held by wine merchant Steven Spurrier, because, as he said "I didn't have anything more important to do." No other wine writers invited to the event attended, because none believed it would result in any news. It was a foregone conclusion the French wines, evaluated by top French judges, would be rated higher than the upstart American wines. Even when some of the American wines beat out French wines with long-standing reputations and much higher prices, Taber's article in Time magazine was only a four paragraph item in a section of other briefs.

Taber gave a brief history of the event that some wine writers, now in retrospect, have called the most important wine event of the 20th Century. This judging also highlighted wine judge inconsistencies, as some of the French judges incorrectly believed some of the American wines were French wines. He pointed out that the judging's top-rated Chateau Montelena Chardonnay and Stag's Leap Cabernet Sauvignon were only the second vintages from these wineries, and all the American wines were from young wineries, or wineries that had just restarted after Prohibition. Describing the value and importance of that judging event, he said, "It showed how the California wine industry came back from the depths of Prohibition."