I had the privilege to be a member of the FEI Dressage Judges Working Group that published their full report in November 2018. A year earlier than that as a result of our interim report the FEI General Assembly voted to reduce the number of Collective marks from 4 to 1 as of 1 January 2018.
We based this recommendation on the conclusions of cognitive load analysis performed in collaboration with the Nottingham Trent University. The actual text of the recommendation is given below:
RECOMMENDATION 15: The DJWG recommends to the FEI that summary marks awarded post-completion of the test be discontinued. Their reasoning for this is that, as this relies upon a longer-term memory retrieval when compared to awarding a mark at the conclusion of each individual movement, there is a lower degree of probability of precision when selecting the appropriate mark and leaves too much to the discretion of each individual judge. When considering the length of many FEI classes the impact of the cumulative cognitive load reduces the probability of accurate mark selection and increases the probability of inconsistency thus increasing the likelihood of unfairness to some athletes.
The FEI accepted the recommendation of the DJWG to remove three out of the four standard ‘Collective’ marks at the 2017 FEI General Assembly.
Recommendations Based on Science
Every one of our recommendations was based on analysis of the relevant scientific data and of course on detailed discussions with experts inside and outside the working group on the very particular case of Dressage judging.
As with all change of course not everyone was happy, and indeed it is perfectly legitimate to return and question earlier conclusions if there are new data or new analyses that may be relevant. Some of our highly experienced riders and experts - in particular on the board of the IDRC - have been advocating for the reintroduction of some of the Collective marks dropped following the GA of 2017. But, since the original decision was based on evidence, any changes should also be based either on new evidence or on re-analysis of the earlier evidence. To the best of my knowledge no new analysis or new evidence has been introduced.
Strive for the Most Objectivity in Judging
We can however look at what effect the 2017 rule change has had on dressage results. The change was motivated by the scientific observation that while Technical movements are judged at the time of execution, with both improved education and feedback as well as the potential adoption of a code of points that has been under deep study since 2018, which can lead to the best objectivity, Collective marks, however, summarize 6-7 minutes of riding and lack immediacy and accountability.
It may be informative to analyze results from before and after the change to see the effect of Collective marks on final results, in particular on final rankings which are perhaps more reliable than actual scores as they are the conclusions of the actual jury composition for the event. To this end I have analyzed Grand Prix results from 2017, using the old 4-component Collective marks, and compared them with results from 2018-2021 using the new 1-component Collective marks. For all these events I have calculated a final ranking based solely on the Technical marks and a final ranking including also the Collective marks in use in that period. (This second ranking is of course the actual ranking in the FEI published results tables).
The results are summarized below in two tables. In the first I take account of only the top 3 ranking places (the podium results), while in the second I take account of up to 10 ranking places. I have not included all ranking places for larger events as typically the points separation between lower ranked riders is much smaller and the number of rank changes will be even higher.
As is clearly shown by the data the reduction from 4 to 1 Collective mark has the effect of halving the number of final rank changes introduced by these Collective marks. This means that in half as many cases as before 2018 will a rider that performed higher in the ensemble of Technical notes have their ranked place taken over by a rider that performed a lower-marked Technical test but profited from higher Collective marks. This is true for all riders, and for riders only in the top 10 places and for riders only in the top 3 places.
Vague Definition of General Impression
Why does that matter? If the scores are very close something has to separate them: well, the big difference is that a Technical score can be autopsied, it can’t be changed, but it can be separated into its different components and it can be justified – or found wanting. But a collective mark formed from the “General Impression” is unassailable, it is vague enough that it cannot be pinned down precisely – it is unaccountable to review.
Yes, certain riders and certain horses clearly paint a better picture, and they should be rewarded for that in every affected Technical movement, but not double counted to receive another boost at the end! Collective marks are less transparent, less precise and thus less objective and fair.
This is also summarized in the graphic below:
So far so good, the reduction in the number and weight of the Collective marks has reduced by a factor of two the number of times a rider with a higher Technical score has their ranking place reduced by a rider with a lower Technical, but higher Collective, score. In my opinion that is giving more weight to the better objectively measured component of a final score.
The Real Influence of the Collective Mark
Even so, and even today with just one collective mark, some 3% of winning places are still being determined by the Collective marks over the Technical marks.
The most recent example I found was in Doha just two months ago where Patrik Kittel (Touchdown) went ahead of Dinja Van Liere (Hermes) who had been slightly ahead on Technical marks, but who fell slightly behind on Collective marks, resulting in a 1500 euro reduction in her prize money and increase in Patrik’s. So, this is not an abstract topic, it has real world implications.
In summary, there can be no objection to new analyses or new research results, but when decisions have been based on scientific results and facts, they should only be reviewed in the light of new results and facts – not solely on feelings and impressions.
- by David Stickland
24 April 2022
Related Links
Attempt to Reintroduce Collective Marks Rejected by FEI
FEI General Assembly Approves DJWG's Proposal to Remove Collectives from Score Sheets
Dressage Judging Working Group Makes Recommendations for 2018 Rule Amendments
Half Way There