Educational Evaluation Guide PDF

1. The ingredients of educational 13 evaluation 1.1 What is Educational Evaluation? Definitions Some possible definitions of educational evaluation are: “Educational evaluation is the systemat...

1. The ingredients of educational 13 evaluation 1.1 What is Educational Evaluation? Definitions Some possible definitions of educational evaluation are: “Educational evaluation is the systematic investigation, observation and interpretation of information.” 23 “Educational evaluation is a method (procedure) and to prove if the expectations and aims of an educational process reflect reality (results of the process).” 24 “Educational evaluation is the process of obtaining information and using it to come to some conclusions which will be used to take decisions.” Without wishing to present only one definition or to summarise all existing ones, we can conclude that different processes of educational evaluation have some common components. The following definition embraces them. “Educational evaluation is a systematic and ongoing process which includes: R  esearching and collecting information, from different sources, about the learning process, the content, the methods, the context, the outcomes of an educational activity The organisation and analysis of that information The establishment of certain criteria (evaluation criteria) T  he discernment and judgement of the analysed information (according to the set evaluation criteria and at the light of the educational objectives). D  rawing conclusions and recommendations which allow the re-orientation and eventual improvement of the educational activity”25. It is necessary to distinguish educational evaluation from the process of collecting and obtaining certain kinds of information. The collection of information is something punctual, and it is done in the evaluation process at certain moments. But educational evaluation is an ongoing process. It implies judgement (good, bad, acceptable, ok, advantageous, disadvantageous, of high quality, of low quality, etc.) Educational evaluation implies measurement. But, educational evaluation goes beyond the mere measurement: it provides explanations and conclusions. 23 Tenbrink, T. & Cooper, J. M. (2003). Educator’s Guide. Page 64. 24 Nydia Elola, Lilia V. Toranzos 2000 Evaluación educativa: una aproximación conceptual. Page 2. 25 Giovanni Iafrancesco, 2001. Hacia el mejoramiento de los procesos evaluativos en relación con el aprendizaje. T-Kit on Educational Page 6. Evaluation in Youth Work www.training-youth.net  SHOULD NOT BE 14 Educational evaluation should not be a justification of shortcomings. As a result of the evaluation we might find out why certain shortcomings appeared, but the actors involved should not conceive and initiate the evaluation process looking for “excuses” or with a defensive attitude. Educational evaluation should not be a money making strategy. As we will see later, one operational objective of educational evaluation is to reflect whether the educational activity met the expectations of partners and funding institutions. Those institutions often use evaluation as an instrument of control and sometimes even as a criteria to decide which projects and organisations to support. This has financial implications for those concerned. But, the starting point, the nature and primary aim of educational evaluation, is not to make money. If we “financially” misuse evaluation and fall into a lack of honesty and transparency, we might in the best case gain some short-term financial benefit. Experience, however, shows us that apart from weakening the legitimacy of educational evaluation, financially motivated evaluation damages trust between funders and those they support. Educational evaluation should not become an exercise in power politics. In educational evaluation, different actors with different levels and areas of responsibility are involved. Often and naturally their conclusions differ. This, together with the fact that conclusions can to changes that can affect different actors in more or less positive ways, might imply that educational evaluation degenerates into a kind of power game, with different actors trying to assert their power over others using the evaluation as an instrument of their objectives. There is no magic rule for resisting this. But, actors who hold positions of responsibility should be those most open to criticism and generous in assuming respon- sibility, even if the results of the evaluation are not to their liking. These attitudes are an important contribution to making sure that the aims and nature of the evaluation are properly respected. Educational evaluation should not be a public relations exercise. Of course, the outcomes of educational evaluation and its achievements should be shared witwh concerned target groups, organisations and institutions. But, this should be an integral consequence of the process rather than an objective of the evaluation in itself. Conceiving and planning educational evaluation as a public relations exercise will damage the process and in long term becomes counter productive co-operation and partnership with others. T-Kit on Educational Evaluation in Youth Work www.training-youth.net  15 1.2 Why evaluate? The aims and nature of educational evaluation 1 To learn The first and most basic aim of educational evaluation is to learn. The aim is the learning of all actors involved: their access to additional knowledge and to a new learning opportunity. The educational purpose is the inherent characteristic of what makes educational evaluation different from other kinds of evaluation. While evaluating, the actors involved learn to understand, to give a value and to draw conclusions on their own learning experiences. Through educational evaluation we learn from experience. The changes and actions resulting from educational evaluation become critical action and reflective praxis26. All the actors involved in educational evaluation learn to express their knowledge: knowl- edge not of “topics” but of the relevance of their educational experience to their own lives. A certain educational activity might seem to be very good from a lot of points of view but in reality it might be disconnected with the life of participants, and vice versa. This relevance, and connection between youth work and the lives of young people, is probably the most important “knowledge” in youth work. Very often it is learned during the evaluation process. Participants also learn while sharing and confronting their judgements with those of their colleagues. During the evaluation process, different interpretations, meanings and inter- relations are raised and debated. Very often actors involved in educational evaluation ask themselves: What does it mean? How should I interpret this or that result? What are the implications? The doubt implied by the diversity of answers that one might get to these questions can be considered as a matter or competence. The different actors need to have a certain “tolerance of ambiguity”, the ability to live with several different pos- sible outcomes, not all of which fit easily together or with one’s personal, professional or organisational values, to be able to accept that the outcomes of the evaluation may not be to their liking. Where this competence is present, educational evaluation can be a motor of curiosity, a source of learning and an impulse to continue learning. When evaluation and learning take place at the same time, the actors involved create, discern, imagine, analyse, contrast, elaborate answers, formulate questions, come up with doubts, search for other sources. In other words, they truly evaluate. In our opinion, and without wishing to ignore other purposes or diminish their importance, educational evaluation should first and foremost be put at the service of learning. T-Kit on Educational 26 Praxis: the translation of an idea into practice. Source Word Reference Dictionary http://www.wordreference.com. Evaluation in Youth Work www.training-youth.net  16 2 To motivate The evaluation process should lead to improvements and change. Change, improvement, evolution and further development are factors of motivation for all the actors involved in the educational process. That is the reason why a constructively carried out educational evaluation contributes to maintaining a challenge and to fostering motivation within a project. An evaluation whose results or process de-motivates becomes limited and incomplete because it cannot maintain the participation of all actors. Some participants might have negative and discouraging perceptions of evaluation. This can be a result of the fact that at times in formal education, evaluation (or more precisely put student assessment) is used as a mechanism to “select” or “exclude”. However, achieving the objective of motivating while evaluating does not only depend on “recognising the achievements as well as shortcomings” of what is being evaluated. It also depends a lot on the attitude adopted by those involved, the atmosphere in which the evaluation takes place and on the imagination of the actors about what will happen after the evaluation results have been made public. 3 To participate Educational evaluation is an opportunity both to promote the values of participation and to practice it. Obvious as it may seem, all the actors involved in the educational process should therefore also be involved in its evaluation. This participative dimension goes beyond the “democratic legitimacy” of changes to the educational process. It also has an educational dimension. It would not be coherent or consequent to aim for the pro- motion of participation in an educational activity but to evaluate the fulfilment of that aim in a non-participatory manner. This aim of promoting participation while evaluating has methodological consequences: in educational evaluation participatory methods are very important. 4 To change and improve As we have seen in its definition, change and improvement are integral to the process of educational evaluation. This idea of change is generally assumed in an “operational” way: change of tools, formats, methods, places, targets. Change as a consequence of the accelerated changes taking place in our societies and in the reality of young people. In educational evaluation the changes also happen at the personal level: change of attitudes, of values, of ways of understanding. This “personal” dimension of change is often less visible than the “operational” one. But, both are equally important: educational evaluation requires openness to changing our ways of doing things as well as our way of thinking. Resistance to evaluation is often rooted in resistance to “internal” and “external” changes that might be required of an individual or of a group as a result of the outcomes of the evaluation process. T-Kit on Educational Evaluation in Youth Work www.training-youth.net  17 1.3 What is evaluation for? The Operational Objectives of Educational Evaluation It is possible to identify some “operational objectives” or uses of educational evaluation, in addition to the general aims of educational evaluation. The following objectives are linked to the implementation and practice of educational evaluation and, to the uses of its outcomes at different moments of the process. The following list is certainly not exhaustive. Educational evaluation has a lot of potential and uses, but among others, the following: To plan better Educational evaluation can help to change things and to plan “different things”, but it can also help us to plan things better, in order to prevent negative consequences and to compensate for possible shortcomings. To take stock of achievements It is important to recognise, name and give value to the achievements of the educational process so that they do not get lost or not sufficiently used. To consolidate results Identified results can be consolidated by making them explicit at the end of the evalu- ation process. The description, sharing and further use of results are natural follow-up steps of educational evaluation. To check if we met the interests of the funding institutions When funding institutions support a certain educational project, they do it according to certain criteria: the nature of the project, its objectives, their priorities. Funding institu- tions usually ask to receive a descriptive and evaluative report at the end of the project. Even so, educational evaluation plans and criteria should not be limited to the expecta- tions of the organisations that fund the project. But, it is important to consider and include them. Usually this is not a difficult exercise: in most cases the questions they T-Kit on Educational would like to answer through evaluation would be part of our evaluation anyway. Evaluation in Youth Work www.training-youth.net  18 To reinforce co-operation with partners If partners are involved in the educational project, they will be involved in its evaluation. A constructive and participative evaluation will naturally reinforce co-operation. But, even if your partners are not directly involved, the results of the evaluation can be of interest to them. You might share new ideas for common projects, other fields of common interest and ways of co-operating, new partners and networks with them. T-Kit on Educational Evaluation in Youth Work www.training-youth.net  19 1.4 What to evaluate? Fields and “models” of evaluation - a critical view and a proposal Together with the question “Why evaluate?”, “What to evaluate?” is another key to under- standing the different options and approaches to educational evaluation available to us in our youth work. In the following sections, we will briefly present some models, which propose a certain structure of “What to evaluate?”. We will look at the models we present critically, not individually, but rather reflecting on some approaches to evaluation present in all of them. Based on this exploration, we will make a proposal: “Educational evaluation as a total experience”. Evaluation fields and “models” In the T-Kit on “Training Essentials” (Page 77) we can find the description of four differ- ent models of educational evaluation. To those we can add one favoured by the USA Department of Education as mentioned in the T-Kit on European Citizenship (Page 57). Each of these models focuses on different fields, when it comes to “What to evaluate?” The following table describes briefly all those models and their associated fields of eval- uation. The Kirkpatrick model: Four fields: Reaction: personal reflection from participants, i.e. on satisfaction, effect and util- ity of the training programme Learning: Growth of knowledge, learning achievements B  ehaviour: Changes in behaviour, transfer of competencies into concrete actions/ situations Results: long term lasting transfer, also in organisational and institutional terms. An example of educational evaluation carried out following this model is described in the Chapter of this T-Kit entitled “Evaluation Practices”. See “Overall evaluation of a Pilot Long Term Training Course” (Pages 94-96) for more in-depth information. T-Kit on Educational Evaluation in Youth Work www.training-youth.net  20 CIPP model: Four fields: Context evaluation – Are the chosen goals the right ones for this activity? Input evaluation – Is the programme well planned? Are there enough resources to implement the activity? P  rocess evaluation – How was the flow of the activity? What feedback was received from the participants? Product (outcome) evaluation – Were the objectives reached? The Brinkerhoff model: Six fields: Goal setting – What are the needs? Are these needs real? P  rogramme design – What is required to meet these needs? Is this design going to meet the needs? Programme implementation – How do we evaluate the programme in practice? Immediate outcomes – Did the participants learn? What did they learn? Intermediate or usage outcomes – Are the participants implementing their learning? Impacts and worth – Did it make a worthwhile difference to the participants’ organisations and their personal development? Systems approach (Bushnell): Four fields: Input – What goes into the training effort? (Trainee qualifications, trainer competence, resources, etc). P  rocess – How adequate are the planning, design, development and implemen- tation of the activity? O  utput – What are the participants’ reactions? Have they gained knowledge or skills?, did they reflect on their behaviour? Did their attitudes change? Outcomes – What are the effects on the participants’ organisations?” 27 United States Department of Education: Three fields: Outcome – The immediate, direct consequences of an educational activity on its participants. Impact – Long-term outcomes of a programme as well as unanticipated effects. Process – Focuses on the procedures, methods and their implementation.”28 27 “T-Kit No. 6 – Training Essentials”. Council of Europe and European Commission (October 2002) Page 77. Original Sources J.J Jackson, Training and Evaluation and R.L. Simone and D.M. Harris, Human Resource Development. T-Kit on Educational 28 “T-Kit No. 7 - Under construction...Citizenship, Youth and Europe”. Council of Europe and European Evaluation in Youth Work Commission (May 2003) Page 57. www.training-youth.net  A critical view on evaluation fields and “models” 21 As we have seen, each of those models put the emphasis on different evaluation fields. Instead of analysing in detail, one by one, the potentialities and limitations of each model, we would like to critically reflect on the different main focuses that an evaluation can take horizontally: by objectives, by competencies, by achievements. This horizontal perspective will provide us with elements to critically understand the previously described models or others and (what is more important) will inspire our own approach to educa- tional evaluation. Evaluation by objectives This is the first and the most natural orientation of an educational evaluation: to evaluate a project according to its objectives. The purpose of the objectives in a project is to guide the educational process and to give an orientation to its evaluation. The objectives should be clear, relevant, be organised in a hierarchy of priorities, be adapted to the needs and profile of participants. They should also be organised in a time frame for their achievement. Evaluation by objectives is associated with “keeping track of those objectives” and re-orienting the evaluated educational process if it is not going in the right direction. The advantages and disadvantages of this kind of evaluation are: Advantages Disadvantages T  here is a clear direction in the eval- A  n evaluation that focuses exclusive- uation process: looking at the fulfil- ly on the set objectives cannot alone ment of the objectives guarantee the fulfilment of the desired learning objectives T  he objectives of an educational activity are the common reference for T  here is the risk of paying little the different elements and actors. For attention to the process and other this reason, evaluating by objectives aspects of the activity facilitates the interaction between S  ticking only to an evaluation of the them objectives limits the creativity and originality of the educational process and does not reflect the diversity or multidimensionality of relevant infor- mation that can come out of an edu- cational activity “Woolf (1999) states that evaluation can only be effective if SMART objectives are set for a project. SMART objectives are those that are Specific, Measurable, Achievable, Realistic, and achievable within a Timescale. This approach can be very appealing for its neatness and simplicity, and indeed evaluation by SMART objectives does have its place and value. However, we feel that an evaluation process defined purely around measurable objectives cannot always do justice to the richness of out-comes, and the long-term impact of educational processes. Is SMART so smart, after all?” 29 29 “T-Kit No. 7 – Under construction...Citizenship, Youth and Europe”. Council of Europe and European T-Kit on Educational Commission (May 2003) Page 57. Evaluation in Youth Work www.training-youth.net  22 Evaluation by competencies This approach evaluates the competencies (knowledge, skills, attitudes, abilities and values) gained, developed or achieved during the educational activity. In non-formal education, competence is understood as “knowing how to do something in a certain context”. Consequently, evaluation by competencies analyses the pertinence, relevance and value of the educational process within its social context. This kind of evaluation tries to answer the question: In the end, what is the value of this educational activity in our social context? For example: evaluation by competencies could look at the value of an activity in which participants develop their intercultural competencies in the context of our multicultural societies. The advantages and disadvantages of this kind of evaluation are: Advantages Disadvantages E  valuation by competencies makes It is not always straight-forward to more explicit the link between non- evaluate competencies in non-formal formal education and its social con- education and to be able to relate text. And this relation / interaction their value to a wider context can be a strong source of learning For example: a certain participant for participants develops in an educational activity the competence of “team work”. T  he combination of individualised Although “team work” is a very qualitative and quantitative informa- important competence in a wider tion that can be gathered in this kind social context, the extent to which of evaluation can be very useful for (s)he be able to put it into practice supporting the personal develop- outside of the educational activity is ment of participants. For example, difficult to asses individual information can be the basis for designing personal devel- T  he concept of competence is asso- opment plans and tools for self- ciated with a large number of catego- evaluation / assessment ries and indicators, which can complicate the organisation and T  he interpretations, arguments and implementation of the evaluation. For propositions explored in this kind of example, the team work competence evaluation can become a catalyst for is associated with: communication, the further development of partici- planning, management, negotiation, pants’ competencies mediation among others. If the context (particularly the finan- cial context) has too much influence in the identification and definition of the competencies to be developed in educational activities, there is a risk that non-formal education will become subordinated to the needs of the market. In recent years, an important focus of evaluation in the field of non-for- mal education has been on the ways in which the “employability” 15 of par- ticipants can be improved through non-formal education. 15  mployability can be defined as “the capability to move self-sufficiently within the labour market to realise E T-Kit on Educational potential through sustainable employment” Source: The Wiki Encyclopaedia. Evaluation in Youth Work http://en.wikipedia.org/wiki/Employability www.training-youth.net Evaluation by achievements 23 Achievement, as such, is an incomplete concept: Achievement of what? In educational evaluation the notion of achievement is related to the “achievement of objectives” or the “achievement of a certain level of competence”. As such, one would be forgiven for thinking that evaluation by achievements is just another way of evaluating the fulfilment of the objectives or evaluating the competence developed through any given activity. But, evaluating by achievements does provide some new perspectives to reflect on. Associated with the notion of achievement is the concept of an “indicator”. “An indicator is a signal, indication, characteristic, data of perceivable information that after being confronted and interpreted according to what was expected, can provide us with evidence of the level or evolution of a certain aspect of the educational process.”30 For example, if one of the objectives of an activity is to promote participation, an indi- cator of the achievement of that objective could be the number of people taking the floor in plenary to express their opinion. If another objective is to encourage team-work, an indicator for the achievement of this objective could be the number and composition of teams voluntarily set up by participants. If another objective is to promote co-operation among participants, an indicator could be the number and characteristics of common initiatives and projects. The advantages and disadvantages of this kind of evaluation are: Advantages Disadvantages A  chievements and indicators of T  here is little conceptual clarity achievement are more concrete com- around the notion of achievement. pared with the ambiguity of evaluating Achievement of what? For what? the fulfilment of the objectives and the Under which circumstances? development of competence in a more Achievements are always related to general way other aspects of evaluation E  valuation by achievements and using A  narrow and limited use of indica- indicators can generate further ques- tors can be counter-productive and tions and debates. confusing. If in the middle of an activity the train- If the only indicator to evaluate the er says “Come on, so far just one achievement of participation is talk- quarter of the participants have so ing in plenary then the notion of par- far taken the floor to speak in ple- ticipation might become exclusively nary”, then, participants might start equated with speaking in plenary to reflect why, or even to discuss if, (e.g. participation=talking in plenary). taking the floor in plenary is an indi- If the expected level of achievement cator of “participation”. and/or the indicators are set by an S  ome indicators could be common external person or institution, they to different educational activities cre- can become instrumental objectives ating some comparability between of the activity. evaluation results. If trainers know that the funder, the organiser or the external evaluator will “take note of the number of people who talk in plenary”, then “talking in plenary” (and not “participating”) might become an instrumental and misleading objective of the activity. T-Kit on Educational 30 La Evaluación como Experiencia, Cerda, G. Hugo. Coop. Editorial Magisterio, Bogotá, 2000 Evaluation in Youth Work www.training-youth.net  24 Evaluation by performance Performance is a concept that is commonly found in the private sector, and is often it related to the assessment of human resources and is often used as a criterion in staff selection. Evaluation by performance focuses on the appreciation of how individuals behave “pro- fessionally”, taking into account their potential for further development. For example, evaluation by performance of a youth leader would analyse how (s)he organises an activity, interacts with participants, facilitates a group discussion or develops an exercise. In non-formal education, evaluation by performance is commonly found in the context of “training for trainers” or in long courses in which participants are expected to implement workshops or projects autonomously as part of the learning experience. Evaluation by performance is strongly linked to the further training and lifelong learning needs of expe- rienced actors in non-formal education. The advantages and disadvantages of this kind of evaluation are: Advantages Disadvantages T  his kind of evaluation is good for T  his kind of evaluation is only rel- assessing human potential and evant for a limited group of actors putting it at the service of a wider in non-formal education, those who group (other participants, colleagues already have a certain level of expe- - trainers, youth leaders and youth rience in non-formal education. workers). E  valuation by performance implies T  his kind of evaluation can be very the management of a large number helpful to set up strategies for update of variables: individuals performing and further training. in different circumstances, doing dif- ferent things, in different contexts. It can be used to promote the recog- This personalised evaluation requires nition of individual professionalism time, specialised staff and a lot of and of non-formal education, in gen- resources in order to be consistent. eral, in other contexts. If the evaluation environment is not safe or professional enough it can imply lack of transparency and pro- voke mistrust between those engaged in the evaluation and those who per- formance is being evaluated. In this kind of evaluation personal styles and preferences can play an important role. If it is not carried out with the necessary distance from social roles, there is the risk of the evaluation suffering from subjectivity. A good illustration of these advantages, disadvantages, potentialities and limitations is one kind of evaluation, often used in training for trainers in non-formal education and youth work. This usually takes the form of participant-trainers31 having to plan and imple- ment a workshop in teams. After it the participant-trainers and the trainers’ team evalu- ate the whole process. The evaluation includes a critical analysis of their performance as trainers in the preparation and running of their workshops. This evaluation by perfor- mance leads to a reflection on the competencies required for being a trainer and how to improve them. T-Kit on Educational 31 Participant-trainer refers to the double role of participants in training for trainers. They are trainers and Evaluation in Youth Work in the frame of the course, at the same time, participants. www.training-youth.net Evaluation by outcomes 25 The concept of outcome refers to different “consequences” of the educational process. In terms of educational evaluation, “outcomes” are mostly associated with fulfilling the objectives, learning achievements, organisational implications and the impact of the activity for a wider social context. Evaluation by outcomes gathers all these aspects. We have already described the advan- tages and disadvantages of focusing the evaluation on each of them and we shall not repeat those arguments now. In principle, considering all of them under the umbrella notion of “outcome” allows for a greater richness of evaluation results and for a wider perspective on the educational process evaluated. But there are risks associated with educational evaluation that uses this, in principle, integrative perspective. Due to the influence of the commercial sector, there is a growing tendency to evaluate educational programmes (and not only their management) by com- paring incomes and outcomes using the purist logic of production. This perspective might be a valid one when it comes to the management of non-formal educational processes or to ensuring transparency in the use of public resources. But, its application without mediation to the educational processes that constitute non-formal education affects its most basic character. Using a simplistic income-outcome evaluation, non-formal education can be reduced to a “social technology” or an instrument for “social intervention” functioning like a machine: you put something in and you automatically get something predetermined out. Incomes Non-formal Outcomes education!? One of the clearest visualisations of this danger was shown in the video “The Wall”32. To the music of “Another brick in the wall”33, anonymous pupils enter a factory, forming an enormous row on a conveyor belt. They fall into a grinder whose product is minced- meat. This symbolises the homogenisation of individuals. The lyrics of the song are “We don’t need no education, We don’t need no thought control…” This is, of course, a caricature of formal education, but the “machine logic” can also be found in non- formal education. The reductive misuse of “evaluation by outcomes” can contribute to the reinforcement of this logic. “Perhaps due to the pressures upon project organisers to prove the value of their work to external bodies (particularly those from which they seek funding) there is a tendency for disproportionate attention to be afforded to outcome evaluation… …Also, it is likely that unpredicted outcomes, and certainly unpredicted processes, will arise over the course of a project. The fact that some outcomes cannot easily be evaluated, and certainly cannot be quantified, should not diminish their importance. We consider it imperative that educational aims are not reduced to those that can be measured, for the sake of being able to prove what has been achieved. An evaluation based only on preconceived notions of outcomes, thus, is unlikely to do full justice to any project.” 34 32 Pink Floyd The Wall (1982) Directed by Alan Parker. 33 “Another brick in the wall”. Rock opera/concept album “The Wall”. Pink Floyd 1979. 34 “T-Kit No. 7 – Under construction...Citizenship, Youth and Europe”. Council of Europe and European T-Kit on Educational Commission (May 2003) Pages 57-58 Evaluation in Youth Work www.training-youth.net  26 Evaluation by process Historically, this approach emerged a critical alternative to evaluation by objectives and by achievements. Evaluation by process tries to overcome their limitations and it is based on qualitative evaluation35. In general terms, in evaluation by process, the starting point is the needs of the differ- ent actors. From those needs expected achievements are formulated. The objectives are fixed to “conduct” the educational process starting from the needs and moving towards the expected achievements. In this case, the “results” would be the sum of most of the elements of the educational process. Evaluation by process is not just a matter of checking “how things are going”. This is very common and at times intentioned simplification. Evaluation by process analyses the relationship between the needs of participants, the expected outcomes, the objectives and the results of a youth project. For example, evaluation by process would mean to analyse the relation, connection, coherence and correspondence between: the need of having leisure activities the expected outcome of creating a cultural association the objective of promoting co-operation and participation during leisure time the result of having sport competitions and a small cyber centre The advantages and disadvantages of this kind of evaluation are: Advantages Disadvantages E  valuation by process has a particu- T  here is little agreement and lar educational value because it does conceptual clarity around the notion not just look at the different stages of “process”. of an educational activity: it goes It is difficult to “capture” the complexity through them. and dynamism of the educational S  ince this kind of evaluation takes process. has its starting point the needs of In reality it is difficult to make the different actors, it covers par- compatible the “theory” and the ticularly well the functions of diag- “praxis” of this kind of evaluation nosis, orientation and motivation. due to its complexity It can help participants, trainers and B  eing basically qualitative, there is organisers to clarify their own chal- a bigger risk of the evaluation lenges and to get to know the most becoming too partial important obstacles in the educa- tional process. Consequently, it helps to develop a more solid and consis- tent educational process. 35 Qualitative evaluation looks at “qualities”: characteristics of the educational experience. It also looks at their meaning for different actors. It raises questions like How? Why?, rather than how many or how much? Quantitative evaluation aims to measure and count. It focuses on the “quantity” of the educational expe- rience raising the questions like How many? How much? How often?... T-Kit on Educational  More complete definitions of qualitative and quantitative evaluation can be found in pages 33-34 in the Evaluation in Youth Work section called “Typologies of evaluation”. www.training-youth.net The difficulty of evaluation by process is very obvious in training courses. The following 27 questions commonly arise: H  ow can the diverse needs of participants be related meaningfully with the objec- tives of the course and with the programme at the beginning of the course? H  ow can the needs be checked in the middle of the course in light of the objec- tives and considering the development of the group? H  ow can the needs be referred back to at the end of the course to see the level of their achievement? On many occasions evaluation by process is unfortunately reduced to picking out some elements of the above here and there without embracing the full complexity of the dif- ferent processes that participants experience during a course. Our proposal: Educational Evaluation as a Total Experience Given the complexity of the educational process the first and most spontaneous answer to the question “What to evaluate?” would be EVERYTHING. In this line, the editorial team of this T-Kit understands educational evaluation as a total experience. This is our proposal and our invitation. Educational evaluation as a total experience: is characterised by its global vision, its integrity and by the articulation of different approaches, methods, theory and praxis combines quantitative and qualitative approaches to evaluation looks at the whole educational process (integrally and from different points of view) prioritises the most valuable information, without falling into “doing everything” The methodological proposal of this approach to educational evaluation is the combina- tion of diverse spaces, actors, times, methods, sources and techniques of evaluation. By doing so, the “conceptual” (learning, motivating…) and the “instrumental” (judging, changing, deciding…) visions of evaluation become complementary rather than contra- dictory. The ethical and educational foundations of this kind of evaluation are: Dialectic36 unity Dialectic unity describes when an idea, a conclusion, an explanation or a thesis and its opposite are not exclusive: both are part of a whole. This happens very often. For example, for some participants a session was too long and for others too short. Educational evaluation as a total experience would not pretend to discern who is right or wrong about that, but to understand, value, put in a certain perspective an eventually challenge both conclusions. Unity in diversity The presence of diversity in educational activities is a fact. Very often it is one of the most important sources of learning for the group. For the purpose of analysing them, evaluation often puts the emphasis on separating and breaking down back- grounds, opinions, experiences, conclusions, contexts, contents, organisational frameworks and methods. However, and necessarily, this diversity should be respect- ed and all of the aspects of the educational process have to be considered. This does not contradict the idea of considering all of them part of a whole or of a unity, the educational process itself. 36 Broadly defined, a dialectic is an exchange of propositions (theses) and counter-propositions (antitheses) resulting in a synthesis of the opposing assertions, or at least a qualitative transformation in the direction T-Kit on Educational of the dialogue. Source: Wikipedia 2006. http://en.wikipedia.org Evaluation in Youth Work www.training-youth.net  28 Educational evaluation as a total experience aims not just to look at diversity and to cover the different dimensions of the educational process, it tries to analyse the connections, mutual influences, coherence and correspondence between them. It raises questions like: How does the organisational framework influence the con- tents? Does the profile of participants correspond to the activity format? Can any trends be observed in the different experiences participants went through? This approach constitutes a significant challenge. In non-formal education, the approach is clearly still under development. It is difficult to translate this approach into practice because it requires effort to deal with complexity. And in practice, given limited time and resources, it is necessary to prioritise among all the possible evaluation fields, among different ways of collecting and processing information. Our invitation to you is to discover and develop educational evaluation as a total expe- rience. A total experience as it is integral to the non-formal educational process, which the evaluation should serve. All the previous “models” of evaluation and any other you may use or know of evaluat- ing have their place and their role. We believe it is good to know them, to look at them critically, and to create a basis for the development of our own evaluation approaches and strategies. “We propose that evaluation, as an element of practice, should be a responsive process, implemented in a manner suited to the particular project, and according to the particular skill, taste and understanding of the actors involved. Evaluation must also be founded in ethical practice and should reflect the aims being sought in its implementation. Perhaps even more important than the need for evaluation to address immediate and long-term outcomes, is the need for ongoing evaluation of processes and outcomes to inform responsive educational practice. The fact that unanticipated results might be significant and that outcomes are complex and not always predictable or measurable, does not save us from the need to set goals and think about what we want to achieve from the earliest stages of planning. Equally, the fact that step by step processes cannot guarantee success does not remove the need to consider what approach you might adopt. In particular it will be important to work out your values and ideas – the ethos or philosophy of your approach, if you like.” 37 Ingredients, recipes, previous experiences, tips… are, for a chef, just the basis for elabo- rating, time and again, a new and suitable menu. We believe the same can be said for effective educational evaluation. T-Kit on Educational 37 “T-Kit No. 7 - Under construction...Citizenship, Youth and Europe”. Council of Europe and European Evaluation in Youth Work Commission (May 2003) Page 58 www.training-youth.net  29 1.5 Who and for whom? The actors of educational evaluation Having reviewed the possible definitions, aims, operational objectives and fields, it is not difficult to identify the actors of educational evaluation. The questions “who?” and “for whom?” are deliberately formulated together because everybody involved in the educational process should participate in its evaluation and should be informed about its results, as follows: The participants: as learners and the target group of the activity T  he facilitators, leaders or team members: as responsible persons for animating the activity The organisers and partners: as promoters of the activity The funders: as supporters of the activity T  he decision-makers: as those “responsible” for considering the results of the evaluation in further decision making processes. These different actors have different responsibilities and tasks in the evaluation. They should participate at different levels and in different phases of the evaluation. Nevertheless, it is important to involve everybody. The evaluation should be democratic, transparent and simultaneously accessible for all actors concerned. Sometimes evaluation is conducted by an “external evaluator”. But, this does not play against the idea of involving all the actors. Moreover, one of the tasks of the external evaluator is to involve the different actors and facilitate their interaction during the eval- uation process. A common assumption is that the external evaluator will be in a position to do that with more easer and with fewer compromises than so-called “insiders”. T-Kit on Educational Evaluation in Youth Work www.training-youth.net  30 1.6 When to evaluate? The timing of educational evaluation Evaluation is an ongoing and continuous process! However, “when” we evaluate is also an important decision to make in the planning of the evaluation process. It is important because it will give us an idea about how things are going at different and distinct points and will allow us to use the information collected to make necessary changes to improve during the educational process. Three main types of evaluation can be identified according to when they take place. These are initial, mid-term and final evaluation. Imagine you are cooking! Before you start cooking, you place all the different things you need for cooking on the work surface and you check if you have everything you need, and more importantly, if you have all the right ingredients. It may not be a good idea to notice half way through cooking that you do not have the corn, or even worse, that you bought fish instead of chicken!!! After you start cooking, you occasionally open the lid on the pot to check the colour and the aroma, to make sure things are going according to plan. Of course, you also occasionally taste the dish you are cooking to check if you have added the right amount of salt or if the pasta is cooked to perfection. And finally, when you have proudly served the meal you prepared to your friends or family, you naturally ask them if they like it. You have probably got the idea already: The above is simply a sequence of initial, mid- term and final evaluations. Evaluating the process and outcomes at different stages gives you more control over the big picture, and ensures that you do not end up with under salted and under cooked chicken. Without even noticing it, you have used initial, mid- term and final evaluations all the way through. Initial evaluation is the evaluation done at the very beginning. Before we start our proj- ect, whether it is a youth exchange or a training course, it is a good idea to take some time out to check our aims, objectives and methodologies. Some questions you might want to ask yourself at this stage could be: Are the objectives of our project in line with our overall aim? Do the methods chosen serve the fulfilment of this aim? Does the programme cover everything we want to address? Is it realistic? D  oes our team have the necessary expertise and capacity to run this programme or do we need to call in some support? T-Kit on Educational Your turn! Take five minutes to write down a few questions you may want to ask yourself Evaluation in Youth Work during your initial evaluation and before you start the project. www.training-youth.net Information about the participants is another important part of initial evaluation, and is 31 commonly called “profile information”. It can be understood as “an initial snapshot of participants”. This information is vital for checking how much effect your programme will have on the lives of young people. In this information there are data, which do not change, for example the age of the participants. This type of information is useful for checking, who on our programme is most likely to benefit. Profile information is crucial for designing your programme in a way that fits the needs of participants. The mid-term evaluation is our chance to check how things are going. As the name implies, it takes place during your project. It may be an “ongoing evaluation”, such as daily reviews at the end of each day, or a whole set of methods that take place at the mid-point of the programme. Mid-term evaluation may prove to be extremely useful for project teams, as it gives them the possibility of identifying potential problems or shortcomings as the programme pro- gresses. Team members may decide to make necessary changes in the programme, to tackle these challenges, or meet additional needs that the participants may raise. Once we complete our programme, it is time for the final evaluation. Using a variety of methods, the participants and the team evaluate the whole project in light of the out- comes (i.e. fulfilment of objectives, learning achievements, organisational implications and impact in a wider social context) with a certain distance and perspective. T-Kit on Educational Evaluation in Youth Work www.training-youth.net  32 1.7 Typologies of Evaluation We have seen in the previous section different types of evaluation according to their timing (initial, mid-term, and final). In this section, we will introduce other types of evaluation, this time categorised according to their actors (personal, interpersonal, group), to their functionality (formative, summative) and to their nature (quantitative, qualitative). Personal/interpersonal/group Personal evaluation is the kind of evaluation in which each individual or actor involved in the educational process makes their own judgements and draws their own conclusions about the experience they have had. Interpersonal evaluation happens when more than one individual actor involved in the educational process shares and discusses their judgements and conclusions. Often this kind of evaluation takes place in a small group setting. While individual judgments may change as a result interpersonal evaluation, arriving to a consensus is not the aim. The purpose is simply to share and discuss those individual evaluations. Group evaluation has an added dimension. It is not simply an interpersonal evaluation in which the number of actors involved is bigger. Since the group as such is part of the context and very often an important source of learning in non-formal education, group evaluation specifically at aspects and dimensions of the learning process that can be observed and judged from a group point of view, including for example the atmosphere, the co-operation among participants, the contribution of the group to the learning and the group process. It should be noted however that this is not its exclusive function. Formative/summative “When the cook tastes the soup while cooking, that’s formative; when the guests taste the already finished soup, that’s summative.” 38 Formative evaluation accompanies the learning process and can contribute to it. It consists of continuous appreciation, ongoing analysis and drawing conclusions. Summative evaluation looks at to the overall and final outcomes (e.g. the fulfilment of the objectives, learning achievements, organisational implications and impact in a wider social context). In other words, it consists of the verification of the expected results and drawing conclusions at the end of the process. T-Kit on Educational 38 Robert Stakes quoted in “Evaluation and Education: A Quarter Century”. Chicago: University of Chicago Evaluation in Youth Work Press, 1991: p. 169. Robert Stakes is professor of education and director of CIRCE at the University of Illinois. www.training-youth.net  Quantitative/qualitative 33 Quantitative evaluation focuses on the “quantity” of the experience. It aims to count or measure different phenomena (literally). The major questions that quantitative evalua- tion raises include “How many?”, “How much?” and “How often?”. For instance: How many young people participated in the youth exchange? How many countries were represented? How often did they stay in touch with each other after the exchange? Qualitative evaluation, on the other hand, relates to the quality of the programme and of the experience. So, qualitative evaluation looks at the meaning of the experience for different actors. This can be on an individual level or on a group level. For example, the experiences of a given youth exchange for Arda from Turkey is considered along side the experience of a group of youth workers involved in a complex training in the city where Arda lives. The questions that qualitative evaluation usually raises are “How” and “Why”. For instance: W  hy did the participants of the contact-making seminar not create any follow up projects? H  ow were the methods of work presented in the training course used by the partici- pants once they went home? Before we get into further detail of quantitative and qualitative evaluation, it is important to note that solid and professional educational evaluation usually involves a combination of both quantitative and qualitative evaluation methods. These provide different types of information that enables you to have a larger and sharper image of the performance of your project. Exercise: Quantitative or Qualitative? Read the following questions and try to find out if they are quantitative or qualita- tive. Afterwards, discuss why. 1. How many young people have participated in the activity? 2. In 2005, how many volunteers participated in voluntary work camps organised by your organisation? 3. We had 200 volunteers from Istanbul in our project! Why do we have much fewer volunteers from the Eastern cities in Turkey? 4. We have had 200 volunteers in our project! How many of them went to Eastern European countries? 5. Why are students and especially university students more likely to apply to become volunteers? 6. How did our campaign on volunteering help to raise awareness? 7. Where is the best place to recruit volunteers? 8. How often do volunteers participate in voluntary work camps after their first experience? 9. Why do we work with volunteers exclusively? 10. Why do we have more female volunteers than male ones? Answers: 1 : Quantitative – 2: Quantitative – 3: Qualitative – 4: Quantitative – 5: Qualitative – 6: Both – 7: Both – 8: Quantitative – 9: Qualitative – 10: Qualitative. T-Kit on Educational Evaluation in Youth Work www.training-youth.net  34 The differences between quantitative and qualitative evaluation are not limited to the type of questions. There are also other major differences: Quantitative and Qualitative Evaluation: Basic differences:39 QUANTITATIVE QUALITATIVE 1. Measurement 1. Explanation 2. Conclusions are based on 2. Conclusions from observations the analysis of data are dependent on interpretation 3. Replication of the process 3. The process is difficult to replicate is possible 4. Structured 4. Unstructured The first difference is that quantitative evaluation measures phenomena that appear in our programme. Quantitative statements contain measurements such as “55% of par- ticipants were from Western European countries”. Qualitative evaluation, on the other hand, seeks to explain why the majority of the participants were from Western European countries. One explanation might be that that participants from outside the European Union did not have enough time to take apply for and receive their visa. The second difference arises from issues of transparency. In quantitative evaluation, how we reached the result on the number of participants that took part is quite clear – we counted the number of participants. However, when using qualitative methods, we move from observations to conclusions using interpretation something, which is considered more subjective than objective. The third distinction relates to replication. Quantitative methods can be replicated exactly, so that each actor will have the same evaluation experience. On the other hand, qualitative data depends much more on the context. Even something as simple as the time of day the evaluation takes place can have an impact on the results. Finally, the fourth distinction: Qualitative evaluation is said to be unstructured. The focus of the evaluation is on the importance of the different actors in the process having the opportunity to express their opinions on the activity. However, quantitative evaluation is far more dependent on a set structure, with set information sought from each respondent having been decided in advance by the evaluators. Another level of difference between quantitative and qualitative evaluation is therefore the methods we use, although, as with the exercise we did identifying qualitative and quantitative evaluation questions, the line between the two can be blurred. The following table shows different methods of qualitative and quantitative evaluations: QUALITATIVE QUANTITATIVE UNSTRUCTURED SEMI STRUCTURED STRUCTURED Observation of participants Structured interviews Surveys Letter to oneself Observation with guidelines Questionnaires Focus groups Spot checks Checklist observations Diaries Film/Video In-depth interview T-Kit on Educational 39 Kneale, Dylan (2004), Un-published presentation on Evaluation, Youth Express Network seminar on Evaluation in Youth Work Evaluation, Avanos: Turkey. www.training-youth.net  35 1.8 Educational evaluation in a broader framework Yeast and a pizza in Italy! If as a foreigner in Italy you would like to prepare a pizza, you might run into a few problems. As soon as you arrive in the “supermercato” to buy yeast to prepare the dough you could easily take the wrong one and end up with a pizza with a sweet cake-like base. That is because in Italy, there are several kinds of yeast. But, as a foreigner you probably would not know that. Anyway, baking a pizza could bring you into deep discussions about different perceptions of pizza with your Italian friends. That’s because the question of pizza is not only about understanding what “yeast” is, it’s about what should be on it, about when to eat it, about regional differences, about the history of the pizza, about what is good tomato-sauce, what is the right cheese, which herbs to use… it is about the whole framework. At the time of writing, the term “evaluation” is frequently raised in discussions of non- formal education, together with terms such as “assessment”, “validation”, “accreditation”, and “qualification”. All this is part of the ongoing discussion about the recognition of non-formal education. There seems to be an agreement that people learn a lot in non- formal education. But what does learn a lot actually mean? What exactly do people learn? How can we measure that? Does it have any value on the labour market? What can the average 17-year old girl do with her three years of experience of working as a volunteer in the local youth association? Would it not be useful if her experiences were noted on paper, so that when she has an interview for a job she could show her competencies? Of course, that would be great. But, what does it prove? Did she really learn from those experiences and how good is she at the things she claims she is able to do? How can non-formal education be more precise in describing what people learn and the extent to which they learn it? One of the things that makes non-formal education different from formal education is that in non-formal education assessment of participants does not take place. In non-formal education we do not organise exams and we do not give marks or grades. We evaluate the programme, the process, the outcomes, but we do not assess the individual level of participants. We rather ask participants what they think they have learned. Still, it could help the 17 year-old girl to gain recognition as a competent person for that job if she had a precise description of what she learned and on what level. This would also mean that her youth association is accredited as an organisation that meets certain standards, allowing it to give her the paper that describes her competencies and ensuring T-Kit on Educational that the paper is recognised as having a certain value. Evaluation in Youth Work www.training-youth.net  36 A large debate exists as to whether the role of non-formal education should also be to provide certification and diplomas for the learning that takes place during its educational processes. It is a controversial issue, as actors concerned disagree about whether this is the way non-formal education should evolve. Supporters of “recognition” say that lots of young people would benefit from the certi- fication of non-formal education. This would broaden their options in their transitions to the labour market and to an autonomous life. Opponents, on the other hand say th at non-formal education would lose its specific nature and character if it would provide certification. The inherent role, social importance and civic mission of non-formal education could be threatened by the movement towards certification. It would completely change the relationships between youth leaders, youth workers and youth trainers and to their target groups. Their role as “facilitators” would be conditioned by a power logic, as they would have to assess participants in their pro- grammes. Discussions around assessment, certification and accreditation of non-formal education are provoking strong educational and ethical debates at very different levels. One of the key criticisms of the movement towards certification is that such approaches can be used as tools for selection and exclusion, risking the restriction of equal access to knowledge and having negative consequences for the nature, curriculum and practice of non-formal education. If we understand non-formal education as a democratic process of access to processes of self- development, any selective misuse of certification would be more than question- able, would raise ethical issues of the most fundamental nature and would put into question the legitimacy of non-formal education and its actors. One significant emerging “outcome” of those debates and an alternative to the so far “black and white” nature of the debate is the concept of self-assessment. The idea behind this form of assessment is to provide participants with adequate tools and support for assessing their own learning and for documenting its results. For example the portfolio: a kind of folder bringing together all materials that prove a person’s efforts in non-formal education and their learning achievements. In recent years some self-assessment tools for non-formal education have been devel- oped on the European level. The Council of Europe has developed a self-evaluation matrix, which helps you to assess your language competencies40. In the youth sector of the Council of Europe, a group of experts has developed the European Portfolio for Youth Leaders and Youth Workers41. 40 This self-evaluation matrix is part of the “European Framework of Reference for Languages: Learning, Teaching, and Assessment”. This framework is a tool for setting standards in an internationally comparable manner. It facilitates a clear definition of teaching and learning objectives and methods and provides the necessary tools for assessment of proficiency. It can be accessed at http://www.coe.int/T/E/Cultural_ Cooperation/education/Languages/Language_Policy/Common_Framework_of_Reference 41 The European Portfolio for youth leaders and youth workers is a tool which has been designed to enable T-Kit on Educational youth leaders and youth workers in Europe to assess and describe their competences, on the basis of a Evaluation in Youth Work core set of European quality standards. It can be accessed at http://www.coe.int/youthportfolio www.training-youth.net  37 The words explained “Evaluation: In English, evaluation only means to make a reasoned judgement about or to give a plausible account of something. It does not imply any specific purpose (such as grading individual performance), nor does it imply any par- ticular method of evaluation (such as a written test), and nor does its outcomes automatically suggest that something is of greater value or importance than something else (such as Council of Europe activities in comparison with SALTO activities). Assessment takes place when evaluation has a comparative dimension that involves setting individuals, activities or institutions into a ranking order of per- formance or achievement. The ranking may be set in relation to criteria that are specific to the context, process or outcomes that are being assessed (such as: who swam the river fastest, or which EVS42 agency has the highest success rate in attracting socially disadvantaged young people into the programme). Alternatively, relative performance may be assesses against an external standard (such as in the case of the PISA43 attainment tests for 15-year-olds in different countries). Certification refers to a standardised process of formally validating knowledge, know-how, skills and/or competencies acquired by an individual or represented through a learning/service provider. Certificates or diplomas are the “pieces of paper” which record the outcome of the certification process. It most frequently has the status of an official document, but this is not an absolute prerequisite. Accreditation: formally or socially recognised authorities or instances (university, department of education…) accredit courses, activities and their outcomes. This means they testify that organisations and individuals meet standards to which all have agreed to conform. They vouch for the credibility of the certificates and diplomas that are issued, and hence for the reliability and validity of the moni- toring, evaluation and assessment of the individuals and the organisations whose judgements are given the stamp of approval.” 44 “Feedback: giving feedback is to pass on to a person the effect of his/her behaviour for their use and learning. Feedback is supposed to be helpful to the person receiving it.”45 42 The European Voluntary Service (EVS) is a part of the Youth in Action programme of the European Union, that promotes the mobility and voluntary involvement of young people. More information at http://eacea.ec.europa.eu/youth/programme/index_en.htm#structure 43 The Programme for International Student Assessment (PISA) is an internationally standardised assessment that was jointly developed by participating countries and administered to15-year-olds in schools. The sur- vey was implemented in 43 countries in the first assessment in 2000, in 41 countries in the second assess- ment in 2003 and at least 58 countries will participate in the third assessment in 2006. More information at http://www.pisa.oecd.org 44 Taken from the “Terminology Cheat Sheet” (by Lynne Chisholm Pages 44-46), part of the report of ‘Bridges for Recognition 2005’. This report is accessible at http://www.salto-youth.net/bridgesReport/ 45 Definition adapted from the report of the Training Module II: Responding Effectively to Cultural Insensitivity T-Kit on Educational of the Multicultural Leadership Institute, 2004. Evaluation in Youth Work www.training-youth.net  38 1.9 Educational evaluation and quality in youth work Why talk about quality? Educational evaluation and quality are very closely related. Evaluation can be considered part of quality. And, vice versa: quality can be part of educational evaluation. What does this mean? Educational evaluation as part of quality: Educational evaluation can be considered as a tool or mechanism for achieving quality. In this case quality would be the aim and evaluation the instrument. Quality Educational Evaluation Quality as part of educational evaluation. Educational evaluation involves making a value judgement on what is evaluated. This “giving a value” would be equivalent to “attaching a certain quality” to what is evaluated. In this case quality would be a layer, a reference at the service of the evaluation process. Educational Evaluation Quality As we can already see, the intimate relationship between educational evaluation and quality is dialectical and uneasy, if one tries not to conceive one of them as a mere instrument of the other. In the debate on Educational Evaluation and Quality there are two poles: O  n the one hand, to reduce educational evaluation to a mere tool at the service of achieving quality would, we believe, imply not to respect the educational nature, aims and potentials of evaluation. Even if “improving” is one of its objectives, edu- cational evaluation is much more than a tool for improvement; O  n the other hand, to consider “quality” as just another “element” of the educa- tional evaluation process would, we believe, imply not facing the quality challenge. Indeed, nowadays, the social demand for quality is growing in a lot of fields (indus- trial products, social services, public services, etc) including non-formal education. T-Kit on Educational Funding bodies, partners, organisers, trainers and participants work for and expect Evaluation in Youth Work a growing level of quality in educational activities. www.training-youth.net With these “poles” in mind we can affirm that in the relationship between quality and 39 educational evaluation there is a plenty of potential for challenging pre-defined boundaries and, as a result, for mutual enrichment. Educational Evaluation Quality “Understandings” of quality What is quality? There are many different conceptions of quality:  ne coming from the private sector would be: “Quality is: if the client comes back O to you”, or “quality is: satisfying the needs of the customer”. Even if we do have a generous and flexible interpretation of this commercial understanding of quality (identifying customers as participants, providers as organisers…) we can conclude that this understanding is rather limited for the educational field. The educational phenomenon and the relations between its actors (participants, trainers, organisers…) go beyond this “commercial transaction logic”. This approach certainly has its place when it comes to the management of pro- grammes or to the use of public resources. But, reducing the educational fact, or experience to being evaluated, to a client-provider relationship is, in the best case, a simplification. T  he European Committee for Standardisation46 agreed on the following definition in the non-governmental and business fields, published as a European standard EN ISO 900047: “Quality is a degree to which a set of inherent characteristics fulfils requirements. The term quality can be used with