At the 2010 Global Dressage Forum German sport psychologist Inga Wolframm blew the audience away with her presentation on judging bias. She explained that the amount of information a dressage judge has to process is too much and exceeds the human information-processing capacity. As her presentation was so thought-provoking Eurodressage invited Wolframm to write an article on her research for this website.
Natural Bias, the Hidden Controversy in Judging Sports
Performance judgement is an inherent part of sports, be that at grassroots or elite level. In fact, almost one-third of all sports registered with the International Olympic Committee use judging to partially or entirely assess performance. Even popular team sports such as soccer, rugby, or hockey, to name but a few, depend on referees to enforce the rules of the game. Especially in moments of controversy, when split-second decisions need to be taken that could make or break a match, the “human” element of refereeing comes to the fore: Did that ball really cross the line? Was it really a foul? Yet while these types of team sports need referees to bring order to the game, ultimate performance is nevertheless assessed in objective terms: the number of points or goals scored.
Performance appraisal in aesthetics sports on the other hand, such as in gymnastics, ice-skating, vaulting, or, indeed, dressage, is almost entirely dependent on judges looking at, evaluating and categorising an athlete’s skill or talent. There can be no doubt that judges try to remain fair at all times during a competition. We can – and should - assume judges want to do their job well. But it is very likely that most judges also try to avoid the negative consequences associated with making a mistake. In the sport of dressage giving a score that is seemingly “unfair” or “wrong” has, in the past, led to outcries from the public, as well as from riders, trainers and other stakeholders in the sport. It comes as no surprise then that wanting to “get it right” is a prime motivator!
The process of judging
Yet considerable evidence exists within sport psychological research that “getting it right” isn’t as easy as it sounds. Social cognition research is a branch of psychological science investigating how people construct their own subjective reality, i.e. what type of previous knowledge and thinking processes they employ to perceive things the way they do. This is of interest not only from a sporting perspective but also to determine how people actually come to make judgements and decisions relating to their daily lives, for example when faced with a dangerous situation while driving a car, or when deciding what to buy for dinner.
Whenever a judgement is made, the brain will run through several steps of processing. The first step involves the perception of the situation, or, in the case of sporting performance, a particular movement. Following on, that movement is given meaning, i.e. it is categorized as good or bad within the context of the sports setting. Evidently, this process of categorization relies heavily on prior knowledge: what has this person (the judge) learned about classifying performance, what type of other performances have been considered as good or bad. In a last step the thus classified movement is considered in combination with relevant memories and other circumstantial information, such as where the competition is held, which team the athlete belongs to etc.
It becomes apparent that errors in judgement can be due to a number of smaller errors in any one (or all of) the three different steps of decision making. Judges could assign an “unfair” score on the basis that they could not see the movement very well, or that they simply missed an aspect of it. Erroneous scoring could also result from an inconsistent categorization, i.e. judges could evaluate a performance as good or bad based on at what point in the competition they get to see the athlete, or whether other people have judged that athlete to be good or bad. Lastly, during the integration of the information judges may yet come to a less than optimal decision, based, perhaps, on the fact that the athlete comes from a particular country, or that there exists a type of “unwritten codex” amongst judges on how to evaluate that type of athlete.
“Shortcuts” in judging
Judging sports have also been found to present an additional problem relating to the human capacity to process complex information. When assessing performances, judges are required to deliver their verdict under the increasing time and social pressure to “get it right”. Furthermore, in aesthetic sports, such as ice-skating, gymnastics and indeed dressage, different movements are extremely complex, consisting of a number of technical and artistic elements that all need to be considered at once. However, research has shown that the processing of such complex information simply exceeds human capabilities. In order to be still be able to provide relevant scores within the given timeframe, judges fall back on schemas, which essentially represent a person’s knowledge of how people (athletes) generally behave (perform) in certain situations (competitions). These schemas or “short-cuts” are based on a number of different information sources, such as the athlete’s reputation, their previous performances, which team they belong to etc., and help judges come up with judgement decisions that, in their mind, approximate actual performances.
Short-cuts may be influenced by or based on different types of preconceptions, or “biases”, which have been examined in other aesthetic sports. To date sport psychology research has identified a number of different biases, which have all been evidenced in the sport of dressage at one point or another. The “patriotism bias”, as the name suggests, indicates that judges would favour athletes from their own country, while the “reputation bias”, implies that judges, much like the rest of the sporting community, are influenced by an athlete’s reputation. The “rank order bias” refers to the tendency to expect a good or bad performance based on at which stage of the competition the performance takes place. An athlete competing early on has been shown to receive lower marks, irrespective of his or her actual performance, than someone competing later in the competition. Psychological research suggests that these effects occur because comparisons between athletes are made in one direction only: performances can only ever be compared to the preceding one, not to the one that will follow. Because of this, judges (and also people in general) will overemphasize any novel positive feature in any new performance. As elite athletes have more unique positive than negative features, scores will continually increase as the competition progresses. Furthermore, at the beginning of a show judges may be reluctant to give extreme scores, seeing that better combinations may follow. In fact, rank order effects have been found in events ranging from the Nebraska State Highschool Gymnastics Meet as early as 1973 to Eurovision Song Contests in more recent years. Such findings provide further evidence that this is not merely an issue prevalent in sports, but a “human affliction” that applies to most people.
Again, the “memory-influenced” bias will presumably also sound familiar. Research has shown that our memories influence perceptual judgements: we judge events of situations based on what we have experienced in the past. This effect is relatively robust over time, meaning that it does not really matter how long ago the experience took place. If the event was salient enough for us to remember, it will influence future decision-making. In a competitive sports environment the memory-influenced bias has several implications. Judges are likely to be influenced by performances they witnessed at previous competitions. The more performances of the same athlete they see, the greater the influence of their memories is likely to be. What is more, warm-up routines, or in the case of dressage, trotting around the ring prior to the actual test, will also impact on a judge’s – short-term – memory and thus will be likely to influence subsequent judgement.
Another type of bias commonly acknowledged in sport psychological research relates to the conformity effect. Especially when judges are able to see the scores given by their judging peers, either during or at the end of a competition, they are likely to adapt their own scoring to “fall in line”. It has been theorized that this is due to either normative or informative influences. Normative influencing refers to the general tendency that people, including judges, are intrinsically motivated to be part of a group, rather than be singled out. If judges appear to be out of sync with other judges, they will aim to adapt their scores accordingly. This phenomenon is essentially nothing else than the judging equivalent of peer pressure. Informative influencing on the other hand primarily occurs due to uncertainty. People generally wish to make the right kind of judgement. If, however, they do not know how to evaluate a certain situations correctly they tend to follow the lead of others, resulting, again, in judging conformity. Next steps
In summary, it seems apparent that especially in highly complex aesthetic sports the demands on judges’ information processing capacities far outweigh what can reasonably be expected, especially taken into account time and social pressures. Ultimately this means that judging tasks should be made less difficult and complex. Judges need to be able to rely on systematic judgements based on objective and transparent evaluation criteria. To that end very clear guidelines should be provided highlighting, for example, which type of error deserves which kind of penalty. However, the very first, and necessary, step needs to be to make judges and the wider sporting community aware of this “human element” in the judging process in order to work together at finding appropriate solutions.
by Inga Wolframm MSc; Sport psychologist VSPN/BASES/BISp for Eurodressage.com
Don't forget to follow Eurodressage on Facebook and Twitter
Related Links
The Judging System in the Firing Line at the 2010 Global Dressage Forum
Making Progress at the 2010 Global Dressage Forum