The afternoon session at the 2018 FEI General Assembly in Manama (BHR) on 19 November 2018 kicked off with a session on the Dressage Judging Working Group and its 19 recommendation to improve dressage judging.
the Chair of the FEI Dressage Committee Frank Kemperman and Bettina De Rham, FEI Director Dressage, Para Dressage, Vaulting and Reining presenting an update on the implementation of the working group’s 19 recommendations which will drive the future of the sport.
The Dressage Judging Working Group (DJWG) were tasked by the FEI Bureau to provide a general evaluation and review of the current dressage judging system and make recommendations for improvement; identify relevant factors and protocols practised in other subjectively judged sports, applicable to the dressage judging system, and make appropriate recommendations; identify and make recommendations for improvement of the current FEI dressage judging system; and evaluate and make recommendations for improvement for the measures introduced after the FEI 2009 Dressage Task force including: the introduction of 7 judges for major events, the introduction of a Judges Supervisory Panel, the introduction of half and the judge’s education system.
The Dressage Judging Working Group is composed by Frank Kemperman, Bettina de Rham, Maribel Alonso, Richard Davison, Kyra Kyrklund and David Stickland.
The DJWG have identified five fundamental principles which they consider should underpin a dressage judging system. All systems must satisfy these tests. All future proposals for improvement must also satisfy these five tests:
1. Be transparent
2. Be easy to understand by all stakeholders
3. Be fair (applied consistently & objectively to all competitors)
4. Be easy to use by all levels of judges (expert & non-expert) on a global scale
5. Be easy to review and adjust (content of CoP and judging protocols).
Review and Analysis of Dressage Judging
In reviewing the current judging systems used by the FEI, aside from drawing on their own collective experiences and those from other external sources, the DJWG invited Katrina Wüst (KW) to deliver a review of certain aspects of judging. This review was based upon a presentation KW had made at the Global Dressage Forum Stakeholder meeting in Aachen July 2018 (in collaboration with Hans-Christian Matthiesen) and also at the International Dressage Trainer Seminar in October 2018. One of the main aspects KW focussed upon was the potential influence of bias and she lobbied for a clearer understanding of these issues between all parties directly affected by them, namely riders, trainers, organizers and the judges themselves.
Possible biases in judging
KW was of the opinion that judges are inevitably subject to possible biases in their work, judges need to be aware of them and attempt to limit their influence, but also judging systems should be configured to minimise the influence of such biases.
Example of such influences are:
• Order bias: A draw according to the World Ranking List raises expectations that the most successful (or better?) riders come at the end of the class. Judges have to realize that they might be biased by this starting order and remain fully aware of where the problems lie.
• Conformity bias: should not occur, however there are pressures, both real and im- aginary, that can influence a judge to conform to what they think their colleagues may do.
• Memory influenced bias: should not happen, however, it is impossible for human beings not to memorize certain things, be it horses, riders, presentations or any- thing else. The judge has to try to 'switch off' his memory and to judge as impar- tially as possible.
• Reputation bias: Closely connected with conformity bias and starting order bias - the judge wants to stay 'in line' with their colleagues and gives good marks to famous riders who he thinks might deserve them.
• Patriotism bias: Is not acceptable, it has nothing to do with psychological phenom- ena that are common to all of us, but is a simple nationalistic attitude.
• Anchoring effect: is a cognitive bias for an individual to rely too heavily on an initial piece of information offered (known as the 'anchor') when making decisions.
In terms of regulating standards of judging, KW believed the judge’s education system, together with the re-validation system for existing judges, should be more robust and effective.
The DJWG took note of this summary and acknowledged the need for the FEI to minimise the effects of any ‘human’ factors which could lead to an inappropriate slanting of marks.
Comparisons to other disciplines & sport
The DJWG compared the judging systems and processes used in other equestrian disciplines and also in some other non-equestrian sports. To this end the aim was to identify:
• Common processes between dressage and other sports/disciplines
• Common difficulties shared between dressage and other sports/disciplines
• Lessons to be learned how other sports/disciplines implement and address change
• Lessons to be learned how other sports/disciplines review their judging systems and procedures
Recommendations Made by the DJWG
1. The DJWG recommends that all future standard judging directives and references (e.g. Code of Points etc) be formatted to ensure that they are relevant to the specific viewing position of each judge. The DJWG consider that judging directives and references should not be ‘silent’ on what is able to be observed/not observed from the various judging positions, as this leaves the priority of observations to the discretion of individual judges. Such discretion could lead to inconsistent judgements and a lack of fairness to athletes.
Khalid Al Khalifa
3. The DJWG recommends to the FEI that, where appropriate, competition levels are redefined and their objectives more clearly identified.
4. The DJWG recommends to the FEI that the focus on ranking should be reduced, especially for the purposes of evaluating judging performance. Instead the DJWG recommends that evaluation is assessed by scrutinising the appropriateness of each mark awarded for each movement.
5. The DJWG recommends to the FEI that the way results and marks are presented be amended to simplify comparison of results for judges officiating from similar viewing positions.
6. The DJWG recommended that technology within each judge’s hut be established to facilitate the communication of observations regarding a limited number of breaches of regulation. These breaches should be strictly limited to prohibited equipment, the appearance of blood etc. Each judge should have the facility to alert the President of the Ground Jury that action needs be taken during the test.
7. The DJWG recommends the FEI set up a sub-group to provide advice on development, innovation and use of modern technology in order to assist judges and improve accuracy and therefore fairness to athletes. The DJWG recommends that new and existing technology is reviewed to, among other things, assess the impact upon cognitive load for judges.
8. The DJWG recommends that the FEI publishes guidelines for judges on how to optimise their cognitive function and what factors can affect it both positively and negatively.
9. The DJWG recommends that the FEI provides evidence-based guidelines to Organising Committees regarding the duration of breaks and other appropriate requirements for judges to support and enhance optimal cognitive function
10. THE DJWG recommends to the FEI that they proceed with the draft Code of Points and, subject to testing and review, roll it out for use in all relevant classes. The DJWG recommend that the proposed Code of Points is supported, where possible, by appropriate visual imagery.
11.The DJWG recommends that the Code of Points should be formally reviewed by a panel of dressage experts every four years and that this review becomes a regulatory obligation and be budgeted for accordingly.
12.The DJWG recommends to the FEI that the practice of using 7 judges at Championships and Games is maintained.
13. The DJWG recommends to the FEI that score correction models that are based upon a percentage threshold should not be used as they fail the DJWG’s test of equal treatment and fairness to all athletes within the same class.
14. The DJWG are of the opinion that if a score correction model is to be employed then the model based upon Hi/Lo drop per movement provides the highest level of fairness as it applies to all athletes within the same class. However, the DJWG recommends that the FEI wait until the impact of the proposed Code of Points is evaluated before assessing the need for score correction models.
15. THE DJWG recommends to the FEI that summary marks awarded post-completion of the test be discontinued. Their reasoning for this is that, as this relies upon a longer-term memory retrieval when compared to awarding a mark at the conclusion of each individual movement, there is a lower degree of probability of precision when selecting the appropriate mark and leaves too much to the discretion of each individual judge. When considering the length of many FEI classes the impact of the cumulative cognitive load reduces the probability of accurate mark selection and increases the probability of inconsistency thus increasing the likelihood of unfairness to some athletes.
16. The DJWG recommends to the FEI that the Education Working Group be tasked to embed the proposed Code of Points into the education system along with those relevant recommendations from the FEI Officials Working Group report.
17. The DJWG recommends to the FEI that guidelines should be provided to judges to make them aware of factors that can impact upon their cognitive performance when judging, such as attention and concentration. The DJWG recommends that, in the interest of fairness to the Athlete, the FEI seeks further advice on the impact of medical conditions and medications upon judging decisions.
18.THE DJWG recommends to the FEI that they review the utility of the Judge’s Dashboard in 2019 and how it can be used to support the education of judges .
19. THE DJWG recommends to the FEI that a Judging Advisory Group be established as an evolution of the current Judging Supervisory Panel and incorporating also the role currently held by the FEI Judge General
The motivation for each recommendation can be read in the full report here.
Related Links
Dressage Judging Working Group Makes Recommendations for 2018 Rule Amendments
Further Steps Taken Towards a "Code of Points" Dressage Judging System
FEI Announces Working Group on Judging and Extended Score Analyses
IDTC, IDOC and Judge General React to Judges Working Group Findings Ahead of 2017 FEI Sports Forum
Dressage Judges Working Group to Present Initial Findings at 2017 FEI Sports Forum