Jennifer Watson, PhD
Executive Summary
This case study emerged from work involved in a review of the implementation and use of CISV’s new educational resources, promoted as part of the organisation’s strategic plan for 2009 to 2012, (Watson, 2012) combined with interest in self-evaluation of their learning by participants in programmes of non-formal learning. Earlier evaluations of such programmes had commented on the ‘over inflation’ of self-scored intercultural competence at the commencement of a programme (Bennett, 2009; Jackson, 2009) or on the apparent regression in scores on repeated use of a questionnaire at the end of a programme compared to the beginning (Jiang Yan, 2010). This project set out to investigate the use of a purpose-designed predictive and reflective self-evaluation strategy which might mitigate these effects as well as provide a means for youth participants to record their own learning.
Programmes such as those provided by CISV claim to be ‘educational’, not in the sense of formal education (defined by Kemmis, 2007, as ‘schooling’) but delivered through purposeful activities working towards specified goals. Such non-formal education frequently employs educational methods or strategies deemed to be ‘experiential’, (Kolb, 1984). That is, the content is delivered through an activity or experience on which participants are, subsequently, encouraged to reflect in order to be able to generalise from the specific learning context to their wider life and thus, hopefully, to be able to apply their learning in future reality. Appropriate attitudes, skills and knowledge (three of the four domains in Fantini’s, (2000), model of intercultural competence) are currently promoted as areas of learning in CISV education policy and detailed in programme goals and goal indicators. This work noted the value of Fantini’s (ibid) fourth domain, awareness, as intrinsic to the ability to score one’s position on a scale of (self-reported) competence and, particularly, during the final days to reflect on where one might have been at the beginning of the programme. In Fantini’s model, awareness is seen to be central to the development of the other domains and, also, to further develop from learning in those three areas.
The case study was implemented with the 36 participants from nine countries in a Summer Camp (now Step Up) for age 14. A purpose designed Predictive and Reflective Questionnaire (PaRQ), developed from the goal indicators for the programme was completed by participants at the beginning and end of the programme. At the beginning they scored their current position and where they felt they would be by the end on a seven point scale. At the end of the programme they again scored their current position but on this occasion they also noted where they felt they had been at the beginning. In addition, at the end of the programme they were encouraged to use narrative spaces to write about their learning during the programme and about what they felt they had learned about themselves. The youth participant self-scores were compared with the leaders’ records of participant of achievement noted on the Programme Director’s Planning and Evaluation Form (PDPEF). Comparison was also made between scaled scores and narrative comments.
The results of the self scoring on the scaled questions showed a general movement towards achievement of the goal indicators and 75% of participants were recorded on the PDPEF as having achieved ten or more of the 12 goal indicators (53% achieved all indicators). In 84% of cases leaders and participants scores were in agreement. Several of those cases where they did not agree focussed on specific indicators which appeared to be related to language (English) competence, with the youth participants indicating that they felt they were able to communicate effectively while their leaders did not always share that view. The majority of these cases of disagreement were in three specific delegations so, in the absence of any direct assessment of language proficiency, it is not possible to determine whether the participants actually were less able to use English or if the leaders had different expectations of language competence. Narrative comments made by participants often suggested that they felt their ability to communicate with others and / or their proficiency in English had improved.
When self-scored attainment at the end of the programme was compared with participants’ initial predictions of their position 14% were at the position predicted, 43% were higher than the participant had predicted and 41% were lower. It was noted that 6.5% of the end of programme scores were actually lower than the participant’s initial, self-scored position. Of particular interest, 53% of scores for where participants felt they had been at the beginning were lower than the point at which they had initially placed themselves. However, even where there was an apparent discrepancy, participants’ comments in their narrative spaces generally reported positive learning. It was suggested that this ‘recalibration’ (Thurber, et al, 2007) of starting position was due to an increased understanding of what is required in order to fulfil the requirements of the indicator; or, as suggested by Kruger and Dunning (1999) ‘making the incompetent competent helps them to realise their incompetence.’
Interviews with adult leaders provided opportunity to discuss aspects of the development of participants, the educational content of the programme and use of the CISV programme monitoring tool, the PDPEF. The way in which the Summer Camp participants had worked together cooperatively from the beginning of the programme was noted, although there were some comments about not reaching the depth of post-activity discussion that might have been expected. Leaders commented on the use of the PDPEF in previous programmes in which they had been involved as well as in the current programme. It was concluded that the form had not been used for continuous monitoring of programme development, as advocated in the online guidance notes (leaders were not aware of these notes) but that it had been effective in recording the final achievement of participants and, thus, the success of the programme.
In conclusion, the use of the PaRQ provided the opportunity for participants to consider the programme goals, and their current skills in relation to these, at the beginning of the programme and to reflect on their progress on these at the end. Commenting on their learning in the narrative spaces gave the chance to note any aspects of learning that were not covered in the programme goal indicators. The potential to adapt this evaluation strategy to other programmes which have clearly articulated goals and goal indicators could be explored so as to see if the strategy is of benefit in other contexts.
References:
Bennett, M J (2009) Editorial: Defining, measuring, and facilitating intercultural learning: a conceptual introduction. Intercultural Education 20 (Supplement 1 – 2) 3 – 13
Fantini AE (2000) A Central Concern: Developing Intercultural Competence, SIT Occasional Papers, 3
Jackson, J (2009) Intercultural learning on short term sojourns. Intercultural Education 20 (Supplement 1 – 2) 59 – 72
Jiang Yan (2010) Intercultural learning in a CISV Village and its short-term and long-term impact on the participants’ intercultural communicative competence development. PhD Thesis, University of London
Kemmis, S (2007) Participatory action research and the public sphere, in Ponte, P and Smith HJ (Eds) The Quality of Practitioner Research. Rotterdam: Sense Publishers
Kolb, D (1984) Experiential Learning: Experience as the Source of Learning and Development. New Jersey: Prentice Hall
Kruger, J and Dunning, D (1999) Unskilled and unaware of it: How difficulties in recognising one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121-1134
Thurber, CA., Scanlin, MM, Scheuler, L. and Henderson, KA. (2007) Youth development outcomes of the camp experience: evidence for multidimensional growth Journal of Youth and Adolescence, 36:241-254
Watson, J (2012) CISV resources for programme improvement (executive summary and full report available to download). Accessed: 11/11/2013