Measuring Quality

April 2024 ‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒‒ 7 minute read

This entry is Part 2 out of a 4 part series on quality. Here is Part 1, Part 3 and Part 4.

In a world where both tools for production and mediums of consumption have proliferated like never before, the importance of quality has become more important than ever. And yet as far as we've come, to measure quality in a truly objective way still remains an dangerous and difficult game. The reason for this is because while objectivity should be based on fact and evidence, the very facts and evidence used must first come from the collective opinion of individuals.

If Truth is as a pane of glass — transparent, but fragile — then opinions are shards of Truth fractured into a thousand pieces. The question I'm trying to ask is this: Is Truth still Truth after every piece is put back together?

Whether consciously or unconsciously, all of us measure the quality of everything around us.
Is my chair comfortable?
Is my device responding quickly?
Is the content I'm consuming interesting?

Measuring quality allows us to judge something to be better than something else. Every judgement we make is reliant on our entire lived experience. Thus, inherent within every judgement we make is a comparison. When we ask is my chair comfortable?, we're actually in fact asking is my chair comfortable as compared to every chair I have ever sat on?

Gradually, as we begin to accumulate more and more lived experiences, we begin to piece together an idea of perfection. Although perfection is theoretically impossible, perfection is a useful reference point from which to measure quality... By giving perfection a rating of 10/10, every rating of quality we make can be considered according to how close it is to perfection.

Of course, because everyone has lived completely different life experiences, their judgements will also be completely different. Subsequently, every judgement we make is a purely subjective notion. A personal truth, not a universal one.

And yet, even our own individual judgements can be wildly dislocated from what our true, uninhibited judgements would be. The prime culprit of sabotage to your own judgements are cognitive biases. To give you an idea of what I'm talking about, here are a few examples...

The Halo Effect takes place when a positive impression favourably impacts your opinion of something. For example, if the previous work of a brand or artist was of a high quality, then you are more likely to perceive their other works as a high quality as well, even if they are objectively bad. This effect can work both ways as well; negative impressions can equally impact an opinion as well. In both scenarios judgements are based on completely unrelated factors. The only way to deal with the halo effect is to seperate the art from the artist.

The Authority Bias describes how we are influenced by people of perceived authority; if they rate something highly, they we too are more likely to rate it highly also. Perceived authorities can range from friends and family to collegues and people we admire... For a textbook example, consider influencers: their entire job is to influence the opinions and lifestyle of their followers. And yes, there's a reason they're called followers²Sheeple would be slightly more accurate, but also slightly more demeaning; because they follow. Dealing with the authority bias comes down to ignoring the opinions of others and simply judging a work for yourself.

The Bandwagon Effect describes how people tend to do what everyone else is doing, hence the 'bandwagon'. In terms of the wisdom of crowds, as soon as individuals begin to influence each other in this was, the wisdom of crowds suddenly becomes herd mentality. More than simply threatening objectivity, the bandwagon effect degrades even the sanctity of subjectivity when your own opinion is determined purely by what everybody else is doing. Again, dealing with the bandwagon effect requires you to ignore the opinions of others and slow down your decision making to simply judge a work for yourself.

Of course, its impossible to ensure or even expect that you are able to simply ignore every possible bias influencing your decisions. However, it is possible to overcome them; by bringing what would usually be an unconscious herestic to the fore-front of your mind, you can then despense with it much greater ease.

In terms of measuring quality, dealing with cognitive biases also has to do with how deeply you consider quality. Consider measuring the quality of a movie. If quality is measured according to purpose, and the purpose of a movie is to engage its audience, in order to measure the extent to which this purpose was achieved, you could initially ask how much were you engaged by the movie, and give a rating out of 10. Although this does give an indication of quality, it's only based on your impression of the movie, which though useful, is a fairly superficial measure.

The deeper you delve ‒ considering not just your impression of the movie but also the plot, the characters, the acting, the score, the script ‒ the more accurate your own judgement will be.

When judging forms as complex and diverse such as music, writing and film, opinions inevitably splinter in a million different directions. Measuring quality according to criteria allows you to approach a far more reliable judgement.

The reason I've focused firstly on more subjective, individual judgments is because they are what makes the basis of more objective, collective judgements.

For many people the most intuitive universal measure of quality is captured by an average of the populus. This measure is encapsulated by aggregators such as Google reviews, Amazon, IMDb and Goodreads, and to a lesser extent the top charts within programs like the app store, Spotify and Youtube¹And yet we don't have any widespread, central aggregators for video games, music, board games, software or art!. Interestingly, this measuring of quality according to the average of humanity is expressed within an idea dubbed the wisdom of crowds, which was first pioneered by James Surpwiechi in his groundbreaking novel of the same name.

The classic example revolves around a crowd guessing the weight of an object, with the average response of the crowd typically being the most accurate answer, a result demonstrated time and time again.

However, the wisdom of crowds isn't limited to guessing games. In fact, it's actually one of the underlying mechanics of the internet. The reason the websites you search are near the top of a search engine is because the more people linking to a page, the higher it rates.

Though not explicit, this in itself gives a reasonable measure of the quality of the website. Theoretically, if a website has quality, it will recieve more traffic and linkings, and the more linkings it recieves the high it should appear on a search engine. Although there are some obvious flaws with this particular method of measuring quality, right now this is primary method we have for aggregating the quality of websites.

Having considered our perception of quality from a few different angles, our original question remains: Is there such a thing as objective quality? First, its worth considering that objectivity and subjectivity are not two static positions; rather, they are two ends of a spectrum. An individual opinion lies closer to the subjective end of the spectrum, while the average opinion of a crowd lies closer to the objective end of the spectrum.

Unfortunately, even if you are able to somehow achieve true objectivity, there still exists one unsumountable pitfall; a phenomenon I'm going to label the objectivity paradox. It goes as follows: Say a judgement is made, and its premise is proven objectively. To prove that this proof was judged objectively, this proof must also be proven objectively. However, to prove that this proof was judged objectively, the proof of the proof must also be proven objectively. However, to prove that this proof was judged objectively, the proof of the proof of the proof must also be proven objectively... And on and on it goes. This paradox is a classic example of a strange loop; no matter how far you progress, it is fundamentally impossible to resolve.

Ultimately, true objectivity is impossible to achieve; with such complexity in the breadth and depth of everything we interact with, there is no way to either theoretically prove or unanimous agree upon anything. However, what we can achieve is inter-subjectivity ‒ not a personal truth or a universal truth, but a local truth.

In terms of quality, a local truth is what we might more commonly call 'taste'. The idea of whether there is such a thing as good and bad taste is key to the objectivity of quality, and is explored in Paul Graham's essay good taste.

We don't realize how far we'd have to go if we discarded the concept of good taste, because we don't even debate the most obvious cases. But it doesn't just mean we can't say which of two famous painters is better. It means we can't say that any painter is better than a randomly chosen eight year old. — Paul Graham, Good Taste

If there were no such thing as good taste, no art piece would be better than another. And yet as obvious as this is, it still remains impossible to objectively prove that the Mona Lisa is better than a random eight year old's artwork. Instead, we rely on the inter-subjective conclusion of experts, artists and professions. So, while there is undeniably such a thing as good taste, there is no such thing as perfect taste. This is the difference between inter-subjectivity and true objectivity.

Of course, all this writing is simply a subjective opinion of whether quality can in fact be measured objectively. So where does that leave us?

The idea that there is anything stable or permanent disappears.

The floor drops away.

And you are left there, playing with nonsense. — David McCandless, Postmodernism

← →