Making up the Numbers – the elephant in the room

In this piece, Tania de St Croix continues our ongoing and necessary debate about the ramifications of the impact/outcomes/ measurement agenda upon a process-led open youth work.

fiddling

Making up the numbers?

 

Do youth workers ‘make up’ numbers in order to demonstrate measureable outcomes? In a recent article, I argued that open access youth work is disadvantaged by an increasing policy emphasis on measureable impact. (The article is available open access here: Youth work, performativity and the new youth impact agenda: getting paid for numbers?). In a thoughtful post on this site, Tony Taylor responded:

 

“My one reservation is that Tania does not pursue what I think is a debilitating consequence of datafication, namely fabrication. Getting paid for by numbers leads to numbers being made up. This tendency is systemic. From my conversations, there is no reason to believe youth work is exempt from this malady.  Perhaps I exaggerate and it would appear that this issue did not emerge explicitly within Tania’s research. Or perchance it remains suppressed.”

 

I agree that fabrication and gaming are an intrinsic aspect of the datafication of public and voluntary services. The concept of fabrication is explored by Stephen Ball in his 2003 article, ‘The Teacher’s Soul and the Terrors of Performativity’:

 

“Fabrications are versions of an organisation (or person) which does not exist – they are not ‘outside the truth’ but neither do they render simply true or direct accounts – they are produced purposefully in order to be accountable. Truthfulness is not the point – the point is their effectiveness… their transformational and disciplinary impact” (emphasis added).

 

In the research for my book, Grassroots Youth Work: Policy, Passion and Resistance in Practice, I interviewed part-time and volunteer youth workers, and reflected on my own experiences as a practitioner. I discussed several instances of what could be seen as fabrication (see chapter 4):

 

  • Youth workers awarded young people ‘easy’ certificates and accreditations for things they could do anyway, irrespective of their youth work participation, and counted these as ‘outcomes’. (Many of these certificates were condescending at best; AQA unit 83522 ‘Making tea or coffee’ was the most striking and oft-repeated example.)
  • Young people were ‘incentivised’ with a trip, pizza, cash, and expensive motorbike competence courses, in return for attending, filling out paperwork, or completing a course.
  • Youth workers exaggerated the nature of their project’s achievements in multi-agency meetings.
  • Managers recorded the results of ‘easy to evidence’ projects, sometimes even creating these projects for that purpose, to enable less ‘countable’ projects to happen.

 

When workers shared such tactics – many of which I have used myself – it was often with a palpable sense of embarrassment, sometimes even shame. Blatant ‘making up’ of numbers was not discussed; perhaps it was hidden, but in most cases it was probably just unnecessary. After all, the best lies are usually those that are closest to the truth. It seemed to me that workers were pressured to ‘get their numbers up’, and they were probably expected to use gaming practices, but these were left deliberately opaque (thus it was often grassroots workers left to take most of the risk):

 

“Part of the ‘game’ is knowing which fabrications are desirable and which are unacceptable. Workers are kept guessing: how far should the truth be pushed and made to bend? Should we prepare a special session when the inspectors are due? Should we add a young person’s name to the attendance list if they only popped in for a moment? Should we share our doubts and false starts when we attend a neighbourhood meeting, or focus only on our achievements? Knowing which compromises are acceptable and which are straying too far from the truth requires a deep and habitual familiarity with systems of judgement. These games are complicated; cheating is frowned upon, but providing wholly honest versions will not make the grade.” (Grassroots Youth Work, p.91).

 

Fabrication is a useful concept, precisely because it shines a light on the murky area between truth and lies; it also makes me think about the (sometimes overlapping) impulses of conformity and resistance that are often characteristic of a commitment to youth work. Workers feel compelled to engage in inauthentic practices they do not believe in, yet to some extent this ‘gaming’ of the system is also a form of rebellion that buys space for ‘real’, ‘meaningful’, and less measurable forms of practice.

 

Like Tony, I am interested in what kinds of fabrications will become systemic under the influence of newer impact mechanisms such as ‘pre and post tests’, comparison groups, and Randomised Control Trials. Working these days in a research institution, my sense is that such methods – especially when carried out without the academic rigour of ethical approval processes and peer review – are highly vulnerable to distortion. A number of tactics come to mind, including but not restricted to:

  • focusing practice – or at least its evaluation – on the ‘most engaged’ and ‘most amenable’ young people;
  • measuring a large number of indicators, in the hope that some will ‘prove’ significant;
  • exaggerating the importance of small effect sizes;
  • burying negative evidence; and,
  • presenting data in inaccessible or incomplete ways.

 

The feelings invoked by numbers and ‘scientific’ data in a field are important here. Without clear and transparent communication, numbers can act to obscure, legitimise, and exclude. As a consequence of the neoliberal fashion for measuring and monetising everything, we can be almost certain of a continuing increase in the emphasis on targetted ‘projects’ and ‘interventions’ at the expense of open youth work. If I was a youth work manager now, I might well feel compelled to ensure that we had some easily ‘measurable’ projects, with clear and achievable ‘outcomes’. Of course, I would do my best to make space for grassroots youth work too; but this means that open youth work will continue to exist only where passionate individuals fight to make space for it, rather than being available to young people by right. Surely this is not acceptable.

Youth work, performativity and the new youth impact agenda: getting paid for numbers? – Tania de St Croix

 

DeStCroixT-Cropped-146x159

Tania de St Croix

 

The ‘impactful’ youth organisation relies on self-improving youth workers and self-improving young people – ideal entrepreneurial, neoliberal subjects.

Continuing the debate on the youth impact agenda Tania de St Croix, a member of the IDYW steering group, has published a provocative, yet nuanced, incisive and widely-researched article, Youth work, performativity and the new youth impact agenda: getting paid for numbers? in the Journal of Education Policy. I’d be tempted to say it is robust and rigorous, if that tired phrase had not been done to death and lost all meaning. Its appearance is timely, coming only a few days before the Centre for Youth Impact’s gathering in London, Shaping the future of impact measurement. Her shot across the bows of the ‘impacteers’ is that their fixation threatens to marginalise further open access, process-centred youth work. A distinctive strength of her analysis is that it is grounded in her research project’s face-to-face engagement with part-time youth workers and volunteers, often a silent and silenced constituency.

Abstract

A growing policy emphasis on measurement and outcomes has led to cultures of performativity, which are transforming what educators do and how they feel about themselves in relation to their work. While most analysis of performativity in education has focused on schools, this article investigates parallel developments in youth work. Youth work is a practice of informal education, in which young people learn and develop through activities, conversation and association. Its evaluation and monitoring have changed over the past two decades, as funding has become tied to targets and measureable outcomes. This article focuses on the English context, where government and third sector organisations are promoting a ‘youth impact agenda’, encouraging organisations to predefine and measure their outcomes. Drawing on data from interviews and focus groups with youth workers, the article argues that the current emphasis on impact risks further marginalising youth work at a time when this practice is already suffering from extensive spending cuts. The article concludes that we need to re-think the purposes and processes of evaluation and accountability – in youth work and beyond – in ways that genuinely value the perspectives of young people and grassroots practitioners.

A brief excerpt to whet the appetite:

Open youth work is particularly unsuited to ‘measurement’ because of its open-ended nature and its basis in peer group learning and informal education. Rather than outcomes being defined in advance, they emerge in negotiation with young people, and the focus is likely to shift and develop in relation to the specific individuals and groups attending, their needs and interests, and the changing social and political context in which they take place. The everyday activities of open youth work can even appear chaotic or purposeless to an outsider: perhaps a rowdy game of cards is in progress in a corner; another group is gathered around chatting and laughing; some people are painting a mural; others appear to be in deep and serious conversation by the kettle. These ‘everyday’ situations are supplemented with more structured elements introduced in negotiation with young people (perhaps an outdoor activities residential or making a film); ‘projects’ that are easier to report on. What is more difficult to describe, let alone measure, is the long-term relationship-based engagement that is at the core of the work, and without which specific projects would be less likely to happen; there is a significant focus in open youth work on process, on what happens ‘between the cracks’ and over time. It is this emphasis on and celebration of the informal and the open-ended that brings youth work into conflict with cultures of managerial accountability and performativity.

My one reservation is that Tania does not pursue what I think is a debilitating consequence of datafication, namely fabrication. Getting paid for by numbers leads to numbers being made up. This tendency is systemic. From my conversations, there is no reason to believe youth work is exempt from this malady.  Perhaps I exaggerate and it would appear that this issue did not emerge explicitly within Tania’s research. Or perchance it remains suppressed.

In a piece, Threatening Youth Work,  I put together with Marilyn Taylor the following exchange takes place.

I’m sure some people will be deeply offended by the implication that results, the need to compete are undermining the integrity of practice.

 
Without doubt, it is happening. To return to the overall argument made by Toby Lowe, his research into Outcomes-based Management reveals that wherever it is being used – in the Health Service, in Social Services, in Housing – ‘gaming’ occurs. To put it bluntly, the need to meet targets and outcomes leads managers and workers into manipulating and fabricating the data. As Toby is at pains to say this is not about maverick individuals, bad apples. ‘Gaming’, falsifying the figures, is a systemic dilemma. It is the consequence of a flawed approach to evaluating the purpose and quality of practice. As things stand youth work has invested its very soul into the Outcomes project. Whilst workers will talk off the record about malpractice, the cost of blowing the whistle would be enormous. It would be perceived as an act of treason.

New development: The paradox of outcomes—the more we measure, the less we understand – Toby Lowe

Conference of European Research Network of Open Youth Work: ‘Theory and Practice – Understanding Youth Work’ 19 – 20 January 2017 : Places Available

There are places available at this conference in Vilnius, Lithuania, where IDYW will be contributing a workshop on the insidious impact of neoliberalism on the provision and philosophy of open youth work in England.

open-yw

Conference of European Research Network of Open Youth Work: ‘Theory and Practice Understanding Youth Work’ and Launch of the International Journal of Open Youth Work 19 – 20 January 2017

Conference venue: Hotel Panorama, Vilnius

Conditions:
There is no participation fee, boarding and lodging will be covered January 19-20,
travel costs up to 100€ will be reimbursed during or right after the conference.

Application form here.

 

POYWE LOGBOOK 2 focuses on youth work and young refugees

refugees

The second issue of the LOGBOOK E-Magazine on Professional Open Youth Work is out now. The main topic of this issue is open youth work and young refugees – read perspectives from Austria, Croatia, Germany and Sweden and much more.

The situation of young refugees in Europe is critical in regards to their access to human rights, their participation and their status when they turn 18. Youth work needs support in terms of capacity or funding, coalition building and having a voice in discussions concerning young refugees.

We also present again the current situation of our field of action in some countries – this time the spotlight is on Austria, Malta and Norway.

Voices from a young person and a youth worker from Lithuania, how to become a detached youth worker in The Netherlands and how to bridge the gap between research and practice in youth work round off this issue.

Enjoy and tell us what you think!

LOGBOOK – new platform for professional open youth work

POYWE

Message from POYWE, of which IDYW is a full member, particularly through the brilliant work of Pauline Grace. More to follow. 

We proudly present our new platform for professional open youth work – LOGBOOK. It offers a video channel featuring experts about different aspects of youth work and an E-Magazine that takes up relevant trends, challenges, methods and opinions of/for the field. Soon to come there will also be webinars where we invite all of you to join the discussion. It is a product of our Strategic Mapping Partnership, who today launched the new service at our meeting in Helsinki, Have a look, listen in and enjoy!

http://logbook.poywe.org/

logbook1-CON-300x170

The first edition of LOGBOOK Magazine takes up the issue of radicalisation and extremism among young people and discusses the role professional open youth work might have in this, presents an insight in the current state of affairs in Croatia, England and The Netherlands, interviews youth workers and young people and asks experts for their opinion. Have a look and enjoy reading!

Coming Events, the Corbyn Effect and an Apology

This week I’m going to be dashing about so there won’t be many posts. One of my weaknesses is failing to keep the Events page up-to-date. Hence this post tries to catch up on what events are in the offing

Friday, September 8 International Study Visit meeting in Birmingham organised by Newman in partnership with Norway and Professional Open Youth Work Europe [POYWE]. IDYW is contributing sessions on the state of youth work in England through Bernard Davies and Tony Taylor.Other speakers include Howard Sercombe [Scotland], Sue Morgan [Northern Ireland] and Mick Conroy [Wales]

Friday, September 8 A Campaign for Youth Work, exploratory meeting in Nottingham initiated by Jason Pandya-Wood. IDYW will be represented.

Tuesday, September 22 Filling the Vacuum – Leadership and the Youth Sector in Leeds. Hope some our supporters will be there.

Thursday October 8 One Hundred Years of Youth and Community Work Education: A Celebration at YMCA George Williams College, London. IDYW will be supporting. STOP PRESS : Places still available

Friday, October 16 The National Youth Sector Debate organised by NCVYS in London. IDYW will be represented.

Tuesday, October 20 Positive for Youth, Positive for Society: The Future of Young People’s Services organised by Public Policy Exchange in Central London. Expensive £195 per person to listen to the dominant orthodoxy, but we’re hoping to get someone there.

Wednesday, November 11 Youth Work and Faith: debates, delights and dilemmas organised by Youth & Policy in Bradford. We will be supporting.

——————————————————————————————————————————————————–

Ta to Shropshire Star

Ta to Shropshire Star

Jeremy Corbyn’s victory has certainly animated many a youth worker on Facebook. We’ll try to do a post next week exploring the possible implications as a catalyst to further discussion.

———————————————————————————————————————————————————–

Finally an apology to several folk, who have sent in material, which has not yet appeared. I’ll endeavour to catch up next week. Best Wishes to all.