soupnancy

Critics and The Star System

The Challenge of Quantifying Quality

by Nancy Hinton, April 16, 2007

When scoring wines, rating restaurants, or even ranking lovers, the question is whether a standard barometer for sensory pleasure makes sense, and if so, how do we fairly accomplish this? With a number, a letter, or words, and according to what rules? How effective is this anyway? And do we need it?

I got thinking about all this for a number of converging reasons, from the whole Jeffrey Chodorow affair last month (when a restaurateur with a zero star review took out a rebuttal ad in the NY Times) to the Parker effect in the wine world around me, as well as the ongoing arguments among friends over newspaper restaurant ratings.

Words or numbers?

Thanks to Wine Spectator and the Parker Phenomenon, number scores for wine are now common. This is a very American construct by the way, in opposition to the entrenched European style long on romanticism and short on numbers.

In Montreal, we see this difference in approach in our city’s restaurant reviews, with La Presse, which uses a descriptive, critical blurb and no grade, whereas in the Gazette, like in New York Times, we see a bold star rating followed by a supporting critique; the emphasis on measuring performance just above delivering qualitative information.

The New York Times restaurant review, the mother of all ratings of this kind in NA in terms of clout, is considered by many as the ultimate reference in NYC and regularly causes much uproar. The French have the Michelin guide, which is a historic three star system, but a different creature altogether, only judging the cream of the crop. A mere star is an honour, with 70 or so two-stars and only twenty six three-star restaurants.. On this side of the Atlantic , one star would be a lacklustre grade. Here, we’re more generous with our stars, but then take them away to determine the score. The guides with authority in Quebec besides local newspapers, are the CAA, the Voir, the Guide Debeur, all with slightly different criteria and scales of their own. The Zagat consumer based guide, hasn’t made significant inroads here, mostly because it’s in English only.

No matter the format of the rating system, in our evaluation of the arts, selecting a number or a making a global statement is an especially difficult, controversial process. Think about it, even words don’t mean the same thing to everyone. Unlike in math or a bicycle race, there aren’t many absolutes in the subjective world of taste and a good time out.

Grey matter

Being a diplomatic, ‘nothing is black and white’ kind of girl, absolute scores like restaurant stars have always made me uneasy. Nevertheless, I do secretly kind of like them. I consume year-end reviews and top-ten lists with glee, I eagerly flip to the dining review in the Saturday paper, all the while feeling a little trashy deep down.

Why do I like the stars, even though I don’t really believe in them? I am drawn to stars and ratings probably because like most of my generation, I grew up on them. Mine was an era of percentage scores, contests with definite winners and losers, and gold stars that I sought to get stamped on my work. I am indeed competitive by nature. But more importantly, I believe in honest opinions, I honour truth and value quality. I accept that some are better than others at a given task, and I like to see those that manage to rise above mediocrity get pats on the pack. I think laziness and poor work should be nailed as such. There is also that natural inclination of mine towards order and classification that surely has roots in my scientific background. Although life has taught me otherwise, the desire to quantify reality is deeply ingrained in me. And like most people, although I know I should hold back from being judgemental, I can’t help it.

Apples and oranges

The thing is, as Nathalie Maclean states in ‘White, red, and read all over’ (a great breezy wine book), “An emotional response can’t be quantified mathematically”. She includes an amusing Adam Gopnik quote in the New Yorker about a man and his harem that makes the point..

‘A man who makes love to fifty some women and then publishes a list in which each one gets a numerical grade, would not be called a lady’s man; he would be called a cad..’

Scaling restaurants is problematic because like women, wines and restaurants are unique; there are personality quirks, and a non-tangible, fleeting, and sometimes magical element to the relationship or experience. Just like you can’t compare apples and oranges, you can’t accurately compare a no-fuss bistro serving tried and true classics with a formal, innovative place; they’re just different.. How do you compare a new Asian restaurant that has beautiful food with chintzy décor and a poor wine list with an ‘haut de gamme’ French restaurant with history, ultra professional service, Riedel glasses and acceptable, by the book food? How do you justly gage a tapas joint or a wine bar against a BYOB? You can compare them on price point or on service, on décor or on authenticity, but overall, it is impossible to do so without nuance; the stars cannot stand alone. Even a few qualifying paragraphs hardly suffice. Restaurant critics try to deal with this dilemma by judging a restaurant according to its raison d’être, what it is trying to be. Talk about obscure guidelines, regardless of how noble the idea is. Add to that the fact the restaurant product, and to a lesser extent any wine, is in flux, constantly evolving, any one experience a singular, unique snapshot in a reel of thousands.

So given all this, how much value should these ratings be given, and who do you trust? The democratization of criticism and art today means that anyone can put out a music video (or a blog); everyone’s opinion matters, anyone can claim to be an expert.. Overall, I think this is a good thing. Dialogue and multiple views offer perspective. But like with the internet, it also means wading through a bunch of crap on a daily basis on any given subject to uncover any truth.

Short cuts

The major problem is we don’t have time for it. (For that reason, I’m sure most haven’t made it this far down my post..) Our attention spans are shorter; we’re in a rush, we’re multi-taskers and skim readers. We want the reader’s digest version of everything; there is no time for details and real complexity. Hence, we rely on such tools as top-ten lists, stars and ‘so and so’’s pick’s to tell us what to consume. We need oversimplification in our fast paced lives.

It allows us to have the overwhelming excess of information around us to be boxed and filed away for easy retrieval. We also like to feel like we know more about all the things of which we know very little, so that we can feel like we’re really living, or at least have some interesting dinner conversation. Mainly, we appreciate convenient short cuts to the good stuff because they save us time. Shouldn’t we be able to trust the experts anyway? Whoever they are..

Five stars says who?

I enjoy having access to lists, ratings and expert opinions, being keenly aware of their limitations. I know who I like and who I don’t (I still read them). But that’s the key, context. Who. We should know that a review is just one person’s opinion, one slice of a story. We should pick our guides and pay attention to who wrote an article or who backed a certain study or produced a show, etc., so we know how to take it. Like when doing research, you try to consult many sources, and check credentials, before accepting anything as currency. Wikipedia offers a quick fix, a few hints, not a basis for a thesis. Most critics, like artisans, have their personal agendas and prejudices.

More than ever, in this age of sound-bites, people make unchecked statements all the time, and it seems acceptable. I get so annoyed with quotes that such and such a place is a 5 star hotel or restaurant, for example. Five stars says who? What are their criteria, what does it mean? If there is no source, it means nothing to me. I also cringe at newspaper headlines making a big claim, citing one vague ‘scientific’ study or some anecdotal evidence, knowing that many readers will take it as law without finishing the article, and decide that butter is bad or that MSG is a plague.

My conclusion is that amidst this sea of opinions, we need the critics with credentials more than ever. I like to read, and so personally prefer an article with substance full of grey, over a black and white star rating. But the star ratings can be fun, like icing on a cake. We must all just take it for what it is…not the holy grail, but some info, a possible lead, or merely some entertainment..

When it comes to entertainment, Beware

Before getting so caught up in criticism as sport though, we mustn’t forget about the real effects it has in real people’s lives. A restaurant is someone’s business, their livelihood, years of blood, sweat and tears, and home to a family of employees and regular customers… A wine is the same thing, usually decades of hard work, investment, patience and love, with troops of earnest faces behind. A bad review necessarily hurts all of these people.

On the flip side, any artist, producer or chef has to answer for what he/she puts out there. They must be thick-skinned, able to take some criticism, and accept that ‘you can’t please everyone’. To survive, they must stay focused on their art, their product, their customers, and not what too many outsiders think. Criticism is a part of doing business and making art.

Too much influence, the uniformity of taste

Besides some potentially hard feelings, the danger of any critic or guide gaining too much influence is that it can start to alter the art, the kind of wine being made or the food being cooked. A trend towards uniformity in taste is never a good thing because not only does it mean a loss of diversity on the landscape of taste, it often results in a ‘dumbing down’ to the lowest common denominator and a lower quality product overall. Big business lobbies and marketing strategies are surely guiltier of this, but a variety of independent critics can balance this effect, in a sense protecting us, all the while challenging the purveyors of our pleasure to perform their best.

The critic, our friend

Critics often claim that they are working for the average diner or customer, that they are a defender of the public, guiding the innocent to sure hits, and away from bad meals and rip-offs. But, really, the only establishments I don’t mind seeing criticized are the frauds, those out to make a dime by fooling people, those who deal in poor quality and charge far too much. I’m all for the outing of a hoax, but cheer far louder at the celebration of an underdog. And in fact, nothing does either like a no star or a full star rating. As long as there is something to back it up, a real person you can know and trust.

Sometimes though, we must agree to disagree, and take it all with a grain of salt.