Ebert and Zeldman on Rating Systems
September 15, 2008
by Nate
Ebert continues a string of fascinating blog entries, this time musing on his demonstrably justified reputation for giving favorable reviews and on the idea of rating systems in general.
“I have quoted countless times a sentence by the critic Robert Warshow (1917-1955), who wrote: ‘A man goes to the movies. The critic must be honest enough to admit that he is that man.’ If my admiration for a movie is inspired by populism, politics, personal experience, generic conventions or even lust, I must say so.”
The most interesting sections of the piece focus on Ebert’s straightforward and compelling description of his own love for movies. I found, however, that his musings on rating systems reminded me of a post by standards-guru Jeffrey Zeldman many months ago. Ebert seems to bemoan his lack of a truly neutral rating, while Zeldman calls for the removal of all maybe or middle options. Why?
“As data, ‘maybe’ is as useless as a three-star rating in a five-star system—and as hypnotically compelling to users. ‘Maybe’ is a button that begs to be pushed.
Maybe is a magnet for neuroses. It salves guilt complexes and incites passive-aggressive avoidance behaviors.”
I am drawn toward the binary for reasons similar to those Zeldman names and those Ebert mentions: I want an ultimate judgement: should I go see it or not? Does it ultimately fight entropy or aid it? Such a diagnosis is, of course, sometimes tremendously difficult, but that’s really what lends savor to the whole endeavor. I don’t believe in the realm of morally neutral! Movies do not actually, cannot actually exist that are neither good nor bad.
This is a giddy thing to claim: it threatens to demand universality, bringing us to the Leibnizian doctrine that there is, in fact, a better and worse sock to put on first in the morning, at least for you, right now. (Don’t worry, though: the sock you picked was the best one to pick, even if it was bad.)
Leaving aside for a moment whether we Leibnizians are crazy (and, more importantly, wrong), let’s make it simply a question of friendship. Wouldn’t you always prefer a friend that’s debating that giddy place in the middle where a thing must fall one way or the other, as that tennis ball in Match Point to one who, shrugging, tells you the movie was “alright”? I know I would: and I’m glad Ebert doesn’t have a neutral rating.


Comments
On September 15 at 5'34 PM
, dwightk wrote:
I’d be for binary over most un-ideal systems.
But in my iTunes playlists, the situation is ideal ;)
I know the precice definition of 1, 2, 3, 4, and 5 star rankings. I can’t communicate them clearly to anyone else, but I don’t need to.
It would be a shame to lose the granularity there.
On September 15 at 11'00 PM
, hb wrote:
I’ll try to leave aside the Leibnizians’ rightness, but I do have to wonder how they account for people watching bad movies: I mean, that was the movie they were supposed to watch in the best of all possible worlds, no? So even bad movies can’t be condemned on that scale of universal particularity Leibniz wants to be able to describe.
I definitely sympathize with the desire for a binary rating, and I certainly operate by one. As I think I’ve said around here, a middling movie, one that’s simply entertain, doesn’t make my thumbs up (that is, “I will watch it”) list. No offense intended to those in whose company I watched Sweeney Todd but I found that to be such a work. (Was there anything to that movie but the thrill that people were being chopped up into meat pies? I mean besides the sappy loves story, that is!)
But as I definitely have said around here, those ultimate judgments are hard to make, and sometimes my judgmental impulse gets in the way, telling me that a work is middling and therefore a thumbs down, when other data (such as repeat viewings, the opinions of people whom I respect) show me that I was just unequal to the task of understanding the work on first blush. Such complications don’t mean we should claim the object is a 3, of course, but just that we were wrong.
As an aside, did you mean anything more particular by the reference to the tennis ball in Match Point than that a question is undecided? It’s a nice image of a poised question, where the answer could go either way: it reminds you of a time when you really, really want to know the answer. Of course, that movie uses that desire in a particular way, namely as a device for making the narrator’s epistemological claim. It seems like the order/chaos distinction might be in play here, too, but I wasn’t sure I saw the connection.
On September 16 at 3'22 PM
, Nate wrote:
Your first paragraph poses—if I understand you—the most basic (which, here, is the opposite of insignificant) question about the Leibnizian system: how are we to understand the presence of apparent bad things in the context of something we claim to be, in a final sense, good? The claim I mean to make is that people are supposed to watch bad movies for the same reason children are “supposed” to be murdered. The badness of it is essential to the goodness of the complete universe.
On Sweeney Todd: yeah, I wouldn’t watch it again. The circumstances and company made it enjoyable time, but I thought the story was almost entirely uninteresting. It’s possible that a more careful reading of Sondheim’s lyrics could change that perception; even so, I’m not sure I believe that Tim Burton’s directing did anything to aid a possible point to the whole thing.
As to the Match Point analogy, I meant to describe a question that not only can go either way, but must. The outcome can have no spectrum, but either total victory or total loss. I think there’s something powerful (and potentially frightening) about viewing art this way: what kind of a terrible calculus goes on when funneling thousands of hours of human lives into a concentrated, two-hour experience in the mind of one judge, who then proclaims either good or bad?
(It may be unholy of me: certainly, I wonder sometimes if I’m savoring the flavor of the tree of knowledge of good and evil, glorying in the opportunity to gaze at what God did and decide for myself whether I shall call it good. What might the whirlwind have to say to me?)
On September 16 at 11'47 PM
, Robbie wrote:
Zeldman has a strangely limited view of data. However frustrating “maybe” is for his preferred applications of data, it’s as much a datum as anything, and sometimes, depending on the question, it’s also true.
The better-devised of surveys and personality tests and so on are usually of the sort that ask for your response to a series of statements from a scale of 1 to 5, with one end being extreme agreement and the other being extreme disagreement. (And these can be both insightful and potentially useful, as in aggregates of children’s responses to statements like, “I like books,” or “There are a lot of books in my house,” correlated to socioeconomic class and school performance, for instance.)
But your reflections do shed some light on the source of my continual surprise at the content and tone of many of the movie reviews here.
To your “question of friendship,” I suppose I must admit I would prefer the “debating that giddy place in the middle where a thing must fall one way or the other” to the shrugging “alright.” But how absurd that we should be so binari-ly limited!
I’m not sure that we should grant the net in your metaphor any sort of real (or metaphysical?) existence at all, but even supposing that we do, surely there is a great and important difference between, on the one hand, bouncing off the net and barely falling to side or the other, and on the other hand, going off somewhere else entirely.
I’d say most of the movies I see (not most of the movies produced) cluster around the net, to one side or the other, with outliers in both directions. But surely the fullest and truest view of all — the god-like view, if you’ll indulge me — is one that accounts for the entire infinity of unique points on the whole of the court. And then, isn’t there something finally arbitrary (even if still meaningful) about the net?
On September 17 at 11'58 AM
, Nate wrote:
I grant you that a three-star rating in a five-star system is a datum, Robbie, but I don’t see how that answers Zeldman’s claim that it’s a useless one. That doesn’t strike me as preferring certain applications so much as preferring application. Your example, too, seems to me to be exactly the kind of thing that would be improved by a reduction of five choices to four. The claim “I am utterly neutral in respect to liking or having books” is not only useless, it strikes me as unlikely to be true. Again, Zeldman’s claim makes sense to me that having the ability to excuse oneself from answering is attractive but defeating of the purpose. I think it’s important to note that the real distinction is whether there’s a middle: you can make the gradations of response as many as you wish: as long as there’s not a middle, you have better data.
To me, denial of the net seems like squeamishness about the difficulty of an ultimate understanding of the dauntingly numerous components of a film. Because there can be so many different ways good things mix with bad things, so many different ways to succeed and fail, we don’t want to say that, in the end, there is something same about all movies that, in the final result, really do fail, or really do succeed.
I’m certainly interested in the ways that some movies fail/succeed by a hair’s breadth and others by a light year. I’m also convinced, though, that the tools of storytelling, maybe particularly so in movies, are so powerful that they are tough to work into near misses. Far more frequent and easy is making something awful or making something wonderful.
On September 17 at 4'35 PM
, Robbie wrote:
”[…] as long as there’s not a middle, you have better data.”
In this case, I think you have data that may be more usable (at least for many particular applications), but also more likely to be false.
If I am, in fact, of neutral opinion about something, or if I’m ambivalent about it, or simply undecided — in any case pulled equally by both sides or by neither, and all of these are, I assure you, possibilities — being forced by a question to come down with a simple yes or no will elicit either no data from me or coerced (and on a large enough scale, possibly misleading) data. Saying that I am moderately in favor or moderately opposed to something when I am in fact neither is false, it is a lie. It may be easier for the data gatherer to do something with that lie than with the truth, but it’s no more true for that fact.
It probably won’t help to go on much longer comparing what to each of us respectively seems likely to be true of other people, but, for whatever cautious expression of disagreement is worth, it seems to me perfectly likely that many young people (probably with less likelihood as they age) would be unsure whether to agree or disagree with a statement like, “I like books,” and I believe that knowing how many of them are could be rather useful. (Incidentally, that statement was one among many that I did so pose to several hundred young people along with a 1-5 degree of agreement apparatus; an appreciable minority indicated their neutrality or uncertainty, and I’m not inclined to doubt their honesty or sincerity.)
“To me, denial of the net seems like squeamishness about the difficulty of an ultimate understanding of the dauntingly numerous components of a film.”
That’s fair enough. And to me, a preoccupation with it seems like squeamishness about uncertainty of classification, ambiguity, and the daunting complexity of unique objects.
“I’m also convinced, though, that the tools of storytelling, maybe particularly so in movies, are so powerful that they are tough to work into near misses. Far more frequent and easy is making something awful or making something wonderful.”
And this is profoundly inconsistent with my own subjective experience of both art and craft.