Back in September Tony Taylor offered a few personal thoughts as the intro to Jon Ord’s closing address at the Transformative Youth Work and Impact Evaluation conference held in Plymouth. He concluded that the occasion was ‘’critical and contradictory, but above all uplifting’. More generally at IDYW we were especially taken with the fact that the research project behind the conference focused on collecting young people’s stories – an approach dear to our hearts. At the same time, we were challenged, given our caution about the dominance of neoliberal definitions of impact,  by the project’s advocacy of a specific methodology, ’Transformative Evaluation’. As ever the danger with research is that its findings are not sufficiently and seriously discussed, often simplified, often indeed ignored. Hence we are especially pleased to post this critical review of the book based on the research, ‘The Impact of Youth Work in Europe: A Study of Five European Countries’. The author is Howard Williamson, Professor of European Policy at the University of South Wales, whose  Finding Common Ground, a sweeping traversal of the European youth work landscape, we described as a veritable tour de force. Despite his reservations about the book, particularly its sense of history, he depicts Transformative Evaluation as ‘a nugget of gold’, expressing a deep concern that its merits may be overlooked. We hope that his thought-provoking review will open further debate and ensure that this does not come to pass.

 

The Impact of Youth Work in Europe: A Study of Five European Countries, edited by John Ord, with Marc Carletti, Susan Cooper, Christoph Dansac, Daniele Morciano, Lasse Siruala and Marti Taru.  Published by Humak University of Applied Sciences Publications, no.56.

 

Over the past 25 years, there have been a number of studies purporting to explore and examine the position, role, impact and value of youth work in Europe.  Two questions arise immediately from such a claim: what is the ‘youth work’ that is being studied, and what constitutes ‘Europe’? To the latter question, there is the narrow Europe represented by the European Union (28, soon to be 27 countries) and the wider Europe represented by the Council of Europe (47 countries and counting).  To the former question, there is the diversity of ‘youth work’ documented and discussed robustly by the 1st European Youth Work Convention in 2010 and debated ever since. The many histories of youth work throughout Europe have, further, been captured in six volumes published by the EU-Council of Europe Youth Partnership.

Most of the contributors to this study have the benefit of having been involved in writing those histories and participating in those events.  Nevertheless, the study is limited to EU Member States – the UK (in fact, just England – an important point as youth work policy and practice is very different in the four nations of the UK), Finland, Estonia, France and Italy.  This means that the very different forms of ‘youth work’ in the wider Europe, in terms of historical evolution, contemporary understanding, funding regimes, training structures and practice provision, have not really been considered, which is a significant weakness if we are concerned with a broader discussion of the impact of youth work in Europe.  I could be even more nit-picky about the book, because it would have benefited enormously from more conscientious proofreading. There are too many frustrating typographical and presentational errors, and occasionally some more serious substantive and factual omissions and mistakes – all suggesting to me that the final publication was, regrettably, composed in undue haste.

 

But let us dwell on the strengths of this study.  Despite being limited to just five essentially western European countries (though objectively a former Soviet Union country, Estonia has moved dramatically to the forefront of youth work policy and practice, and is already arguably a world leader), there are fascinating comparisons and contrasts to be made.  Finland and Estonia have thriving youth work stories to tell. Italy and France have very different traditions, as youth work (though differently named) seeks to make its mark. And the UK (England), once perhaps a benchmark for a repertoire of youth work theory and practice, has suffered from a corrosive decline in recent years.

 

The idea of youth work stories lies at the heart of this study and, notwithstanding the country (hi)stories, its great strength is the innovative methodology (one concerned with ‘transformative evaluation’) that has been pioneered by Susan Cooper.  This anchored the study and merits very serious attention in any future study of youth work anywhere else in the world.

 
First, though, the book addresses the very different histories, practices and status of ‘youth work’ in the five countries.  These are instructive summaries of often complex and invisible contexts. Only in Finland and the UK have there been long, unbroken traditions of a particular mosaic of youth work, in contrast to Italy and France where the concept and indeed even the name ‘youth work’ remains largely unrecognised, even though elements of practice with young people could readily be identified with more transnational definitions of youth work.  The profiles of youth work in the five countries are largely a repetition of material already included in Council of Europe history publications; their strength here lies in the structure of each chapter which allows for immediate contrast and comparison, in relation to themes such as the relative influence of the state as against more grass-roots or faith-based initiatives, the emergence of more targeted practice, pressures to engage in more ‘cross-sectoral’ collaboration (which brings both advantages and challenges), the balance desired or demanded between preventative and promotional activity, how much measurement and evaluation of youth work is taking place, and – one of Jon Ord’s particular concerns – the extent to which top-down bureaucratic and managerial expectations might be stifling more democratic, negotiated and participative forms of youth work practice. As Daniele Morciano observes (p.81), with regard to the ‘new spaces’ emerging for youth work in Italy:

 

‘There is therefore a danger that a fundamental shift in ethos could take place from the creation of a relational space in which the youth worker and young people co-construct meanings (sense-making) to the development of technical abilities to product specific results (production) [emphasis original]’.

 

It is only on p.100 that we finally encounter the nugget of gold that constitutes this study and its central rationale – the application of Sue Cooper’s methodology of Transformative Evaluation (TE).  I am actually very surprised that this did not come first, as it is the driving force for the project. Instead, the opening chapter seeks to set the European youth work policy context, but it is deeply flawed, misinformed, historically out of synch, confuses the work of the European Commission and the Council of Europe in the youth sector, omits critical policy landmarks (such as the Council of Europe’s Agenda 2020), and distorts the overall direction of travel which, whatever is said about attempts to connect youth work to wider policy agendas such as combating radicalisation, remains heavily – and arguably disproportionately – weighted towards a form of youth participation that privileges self-governed youth organisations.  Chapter 1 concludes that the European youth work policy agenda “has the potential to run counter to the autonomous actions of youth workers and youth work’s person-centred practice”. Yes, it is a possibility, but there is still plenty of European youth work policy and practice that is fighting against this ‘instrumentalising’ direction of travel.

 

Cooper’s methodology is relatively new, yet it has already been embraced and celebrated in various parts of the world, including a recently published report on the value of youth work in Scotland [1].  Given the inexorable pressures on youth work to demonstrate its value, measure its impact and outcomes, and clarify the social return on investment in it, we need both to resist those methodologies that feel (and usually are) alien to our grounded knowledge and values and to embrace alternative methodologies that are not only meaningful to us but are also plausible and persuasive to others.  We need to reject any ‘narrowly-formed technocratic concept’ (p.100) of evaluation and replace it with an evaluation methodology that “must be commensurate with youth work itself… only then will we genuinely be able to articulate the value of youth work to policy makers and funders” (p.100). Transformative Evaluation has a chance.

 

Put simply (forgive me, Sue), TE uses a participatory, dialogical and process-driven methodology that enables practitioners to engage with evaluation as a result of a redistribution of the power that is inherent within the evaluation process.  It elicits from the systematic collection of experiential stories told by participants the Most Significant Change (MSC) that has taken place. It draws from a number of strengths-based theoretical ideas and involves all relevant stakeholders in a four-stage process of data collection, reflection and analysis.  It is a robust and rigorous process that should reveal the most significant ‘domains’ of impact and outcome effected on young people who have participated in youth work relationships, experiences and opportunities.

 

The research itself engaged largely with ‘ordinary’ (as opposed to ‘vulnerable’) young people involved in open-access youth provision.  It is not claimed to be an objective process, but – and these are my words and my emphasis – it does endeavour to dig deep in order to illuminate what is often well-known (self-evident) to youth workers themselves, but can seem magical, mystical and even mythical to those beyond.  The TE process was followed over three cycles, to reinforce and cement the central findings from each location [2].

 

What is really impressive about this study is the broad consistency in the raw data that underpins the subsequent analysis.  Each country team adhered closely to the processual methodological requirements (generating an overall average of around 150 significant change stories, from usually three youth work settings – 715 stories in all), although there was considerable divergence in the contexts in which the research was carried out.  Though the project lead proclaims (p.112) that all research sites were ‘open-access’, elsewhere it is conceded that most were predominantly open access, often also accommodating more targeted practice.  The youth work covered also accommodated a variety of thematic practice, from alternative educational provision to art and culture.  And there was also considerable diversity in the age ranges of the young people engaged in these different projects, centres and programmes.

 

That diversity is, simultaneously, a strength and a weakness.  The weakness is that, inevitably, different kinds of youth work engaging different kinds of young people are likely to generate different kinds of outcomes, whatever the evaluation methodology invoked.  The strength is that, though in different hierarchies of priority, there is impressive, even dramatic, commonality across this variety of youth work and across the five partner countries.  None shared their analysis with the others until it had been concluded, yet even before reading the final chapter, my notes point to the striking cross-over in the paramount thematic clusters drawn from young people’s accounts of their youth work engagement and experience.  It is no surprise to a seasoned practitioner that the principal stories recounted relate to new and positive experiences, a greater sense of inclusion and well-being, the value enshrined in relationships with and trust of youth workers, the development of friendship through associative space, a general improvement in social responsibility and communication skills, and an all-round strengthening of resilience with a concomitant decrease in risk.

 

It is, indeed, these very attributes that are presented in the final chapter.  Jon Ord, as the project co-ordinator and lead editor of the publication, has the invidious task of trying to draw the lessons from the study together.  He does an excellent job, first seeking to provide some important ‘impressions’ of youth work’s diversity (not least its place in the space between leisure and learning, and its provision for different segments of a broad age range of young people) but then moves on to advance his ‘impressions’ of youth work’s similarity.  And, rather as I have listed what I detected as the primary themes arising from the research, Ord does this more systematically from the data, maintaining that there are five overarching themes: relating to others, sense of self, creating places and spaces for young people, social inclusion, experiential learning.

 

Ord is absolutely right about this, but we are almost hoisted by our own petard.  These ‘outcomes’, which have always been at the heart of what many youth workers say they do (while others tell them to ‘prove it’), need elaboration, assertion and calibrated explanation to a wider public.  Youth work helps to build and cement relationships, strengthening young people’s identity and connections. Youth work builds confidence and strengthens young people’s well-being.  Youth work provides spaces for association and self-discovery.  Youth work contributes to young people’s civic and economic engagement. Youth work offers opportunities and experiences that widens and deepens knowledge and skills.  Who would not wish any of that for their children and young people?

 

Ord is too modest in the caveats he offers regarding this research.  We need to proclaim loudly that it does draw important conclusions, anchored within a credible methodology.  Dare I say it, but it may prove to be a valuable, complementary evidence base to the emergent neuro-scientific evidence about youth development, which also points to the value of youth work!  Conversely, though, there is a risk of it being dismissed as the self-indulgent ramblings of those already inside the youth work bubble, who have found a methodology to suit their case and cause.  I disagree, but I wish more care had been taken with the technical presentation of the book, which will provide doubters with an Achilles’ heel. The order of chapters is inconsistent and even the order of some conclusions (for example, what is the order of the findings from Italy, that of p.179 or Table 11.1?) produces confusion.  Is the book articulating a case, or evidencing a result? The different formats adopted in the chapters presenting country analyses and discussion do not help. There is a risk of being accused of presenting old wine in new bottles.

 

Some of the more technical aspects of this critique could so easily have been rectified.  Many of the more professional concerns could not. Throughout the book, Ord and others make the critical point that young people’s youth work experiences and their stories that flow from them are “complex, multi-layered and not limited to only one feature” (p.173).  As Ord himself concludes, correctly, “Youth work is multi-layered, and the different aspects interlink to create a holistic educational experience” (p.227). Despite my criticisms, all credit to him and his colleagues for seeking to unravel that complexity in response to the burning pressure of our time – to demonstrate the value of youth work – before others try to do it for us in less empathetic and probably more unpalatable ways.

1. Fyfe, I. et al (2018), The Impact of Community-based Universal Youth Work in Scotland, a study commissioned by the Scottish Youth Work Research Steering Group, Edinburgh: Youthlink Scotland
2. Cooper herself reflects, in an Appendix to the book, on the wider positive implications of using TE in this study.

 

Howard Williamson

University of South Wales

 

As we reflect on Howard’s concern that we find ways to fend off the ‘unpalatable’, we are spurred to continue this discussion by sharpening our thoughts on the possibilities of Transformative Evaluation. We will endeavour to post these in the near future.