Evaluation: what decision-making tools are at the heart of sporting policies?

Texte

Definitions

It is not easy to define what exactly is meant by the term evaluation. Indeed, it is used to designate a multitude of operations ranging, for example, from quality control of an administrative service or a product to the evaluation of personnel or the impact study of a new law. In the limited framework of this article, we are left with the evaluation of projects, plans, programmes, or policies in the field of sport. It seems that there is no real consensus amongst researchers on the nature of evaluation, its field, the theoretical instruments to be used, the indicators to be constructed and the types of calculations to be made. This is why we start with two commonly accepted general definitions:

• Evaluation is the set of analyses, methods and behaviours that allow us to understand why and how objectives are achieved or not, by the means that are assigned to them.

• Evaluation is also a reflection that must explain how priorities are defined and chosen, how means and financing are determined and selected, how constraints (regulatory, temporal, financial) impose certain choices, how political ambitions must be translated into different standards, how, once these choices have been made and financing defined, actions are implemented, in time and space, with operators and operations.

On reading such a definition, which underpins an entire scientific research programme, it is clear how difficult the exercise is. To try to build an operational approach, it is necessary to reduce such complexity by starting from the recognition of a real life cycle of a project, a programme or a policy based on three phases: development, execution and results. At each of these three stages, it is necessary to carry out a specific evaluation, commonly referred to in the case of public policy evaluation as a strategic evaluation, an operational evaluation and an outcome evaluation. In the field of sporting project evaluation, we will confine ourselves to strategic evaluation (ex-ante) and outcome evaluation (ex-post), the latter being conceivable in the short and long term.

• Strategic evaluation: it aims to analyse the project's stakes, make the objectives and means explicit, to reveal the underlying strategy. It is a question of assessing the relevance of the actions undertaken by the decision-makers. Here we find analyses in terms of opportunity cost: would my investment not have been more beneficial in another project? For example: should a football World Cup be held in South Africa in 2010 or in Qatar in 2022? Should we have given up on the construction of the French Rugby Federation's mega stadium? The question is what was the quality of the evaluation that underpinned such decision-making? Strategic evaluation poses many methodological problems that have often led to a real distortion of the exercise. For example, to assess the relevance of hosting a major sporting event, an economic impact study has been substituted for a social profitability study, which is not legitimate.

• Results evaluation: it aims to measure the impact of the projects or programmes on employment, income, wealth, well-being and the economic development of the territory concerned. Such a calculation can be made in the short or long term. For example, it is customary to consider a 17-year life cycle for the Olympic Games: a 7-year pre-Olympic period between the date of the award of the Games and the year of the Games; the year in which the Games are held and the legacy after the Games stretching over a 10-year period. For Paris 2024, CDES, in its ex-ante study, retained the three phases 2017-2023; 2024, 2025-2034. These three phases will of course be reassessed after the end of the Games. To carry out such an impact calculation, there are many methods available, all of which pose difficulties that are, more or less, easy to solve.

Methodological issues

Both types of evaluation (ex-ante and ex-post) pose specific methodological problems. For the ex-post evaluation, we refer the reader to the article on economic impact.

For the strategic evaluation, it seems to us that it is the most decisive but also the most delicate moment of the overall evaluation of a project. It is a matter of determining whether the actions undertaken are relevant to the issues, objectives and priorities discussed during the development of the project. It is often said (by decision-makers) that the evaluation must be neither a judgement nor a court of law questioning the competence of the actors or the legitimacy of the choices made. Whatever the reasons for such an attitude (which risks considerably restricting the field of evaluations), it seems to us that the question that cannot be ignored is: whether the actions taken are the right ones?

To answer such a question on the relevance of the actions, it is necessary to first reflect on the stakes of the project or programme evaluated, its objectives and priorities. It is at this level that the theoretical underpinnings of the proposed actions can be appreciated, which, beyond the sporting dimension, are centred around employment, economic development and economic and social cohesion. It is therefore important to assess whether the action programme is pertinent to the objectives, i.e., whether it is genuinely based on an analysis of the match between these actions and the objectives to be achieved.

Many controversies have arisen around this evaluation of the project's relevance. In particular, it is a question of knowing who should pronounce on such relevance, and consequently, two types of evaluation can be envisaged: an external evaluation carried out by experts; an internal evaluation carried out by decision-makers. Here again, these two types of evaluation can be considered complementary, the first giving a rather technical vision while the second inevitably integrates institutional and political dimensions. Whatever the formula adopted, it is impossible to avoid the fact that the actors involved in the strategic evaluation will have to make value judgements. The question then arises as to the type of norms against which such assessments are to be made, and controversy may arise in this regard:

• Can we consider that there are theoretical models that have governed the decision-making? It would then be sufficient to compare theory and reality to validate or not the decisions. There are many examples to illustrate the theoretical origin of certain actions: the role of transport infrastructures in economic development, the basic nature of certain activities (tourism, sport, export industries, etc.), the importance of the spatial proximity of economic activities, etc.

• Conversely, should we consider that the proposed actions result from the decision-makers’ beliefs, which thus constitute a model? What is tested would not be the direct result of a theoretical model but rather the consequence of the decision-makers' adherence to a certain number of beliefs that are more or less scientifically validated but which have seduced them.

The first case would be an expert assessment and the second would be a decision-maker's assessment. It is not for us to determine which is the best way to proceed, but it is necessary to be aware that the nature of the evaluation will be completely different:

• Assessing the legitimacy of actions in the light of a theory comes down to judging the usefulness of said theory in the development of a policy, which, scientifically speaking, may shock the most positivist. The positive/normative quarrel can be found in economics. Nevertheless, this approach can be very practical in the event of an unfavourable evaluation as it is the theory that will be called into question, not the decision-maker!

• Assessing the relevance of a programme to the beliefs of decision-makers is more consistent with policy evaluation but more dangerous for decision-makers who may be challenged through their proposals. Nevertheless, this procedure can be an instrument of progress by encouraging the abandonment of outdated beliefs and the adoption of a new value system more in line with current reality.

These two conceptions of evaluation should be carried out together, as they are not completely independent of each other. There is no pure form of evaluation: a theoretical model always contains largely explicit values, whereas beliefs always have a more or less distant link with a theoretical model. Yet, whatever the mode of evaluation, there is always an unresolved question about the step from evaluation to decision-making. It does not seem legitimate to us to go directly from one to the other without providing for a phase of negotiation between all the parties concerned by the project, especially the citizens.

From evaluation to negotiation

Citizens’ conference

One way of improving the social acceptability of hosting mega-sporting events would be to get all the actors around the negotiating table. A public debate should be envisaged, insofar as the organisation of such events is not only a sporting matter but involves real social choices. Experience shows that if properly trained and informed, a panel of citizens can give opinions that are just as relevant as those of patent experts. These citizens' conferences are consultative mechanisms set up within small groups of people solicited by the public authorities to give a reasoned opinion on controversial projects or issues.

Citizens' conferences were born out of the observation that scientific expertise was being instrumentalised to serve private interests, following a number of scandals (contaminated blood, mad cow disease, asbestos, etc.). Moreover, to inform the decision of elected officials, the usual procedures have proved ineffective: referendums, surveys, public debates, and public enquiries. Conferences have been set up since the 1970s to compensate for these various shortcomings, mainly in Northern European countries, but since then have gradually extended to other regions.

To be successful, a citizens' conference must meet a certain number of conditions and, in this case, it becomes, alongside experts, elected officials and the associative movement, the fourth partner traditionally neglected in the elaboration of public choices. Thanks to the participation of ordinary citizens, participatory democracy has been set up to compensate for the laxity of representative democracy. The hosting of mega-sporting events could then be the subject of citizens' conferences on the most controversial points, such as the measurement of the intangible legacy, the social utility or the opportunity cost of these organised events.

Olympic games

The main criticism levelled at the hosting of these mega-sporting events now concerns their excessiveness. A citizens' conference could be tasked with analysing the scope of this criticism around four questions:

• What are the consequences of organising such an event from an economic, social, ecological and geopolitical perspective? The citizens will be informed by the best experts on the global impact of the event with a convincing justification of the results, a discussion on the controversies of the methods and the difficulties of measuring the externalities.

• For whose benefit and to whose detriment is the event being hosted? This implies a reflection on the people affected, on the rights that are often flouted and on the most disadvantaged populations during the organisation of mega-sporting events.

• What are the nature and consequences of the decision-making process? The citizens' conference will have to pronounce on the relatively democratic character of the decision-making process, the risks of corruption, etc.

• What is the overall project for hosting the event? The success of a mega-sporting event depends above all on its inclusion in a regional project. Territorial foresight exercises are necessary, for example, with the creation of scenarios. It is widely acknowledged that too many long-term decisions have been taken far too lightly.