Making up the Numbers – the elephant in the room

In this piece, Tania de St Croix continues our ongoing and necessary debate about the ramifications of the impact/outcomes/ measurement agenda upon a process-led open youth work.

fiddling

Making up the numbers?

 

Do youth workers ‘make up’ numbers in order to demonstrate measureable outcomes? In a recent article, I argued that open access youth work is disadvantaged by an increasing policy emphasis on measureable impact. (The article is available open access here: Youth work, performativity and the new youth impact agenda: getting paid for numbers?). In a thoughtful post on this site, Tony Taylor responded:

 

“My one reservation is that Tania does not pursue what I think is a debilitating consequence of datafication, namely fabrication. Getting paid for by numbers leads to numbers being made up. This tendency is systemic. From my conversations, there is no reason to believe youth work is exempt from this malady.  Perhaps I exaggerate and it would appear that this issue did not emerge explicitly within Tania’s research. Or perchance it remains suppressed.”

 

I agree that fabrication and gaming are an intrinsic aspect of the datafication of public and voluntary services. The concept of fabrication is explored by Stephen Ball in his 2003 article, ‘The Teacher’s Soul and the Terrors of Performativity’:

 

“Fabrications are versions of an organisation (or person) which does not exist – they are not ‘outside the truth’ but neither do they render simply true or direct accounts – they are produced purposefully in order to be accountable. Truthfulness is not the point – the point is their effectiveness… their transformational and disciplinary impact” (emphasis added).

 

In the research for my book, Grassroots Youth Work: Policy, Passion and Resistance in Practice, I interviewed part-time and volunteer youth workers, and reflected on my own experiences as a practitioner. I discussed several instances of what could be seen as fabrication (see chapter 4):

 

  • Youth workers awarded young people ‘easy’ certificates and accreditations for things they could do anyway, irrespective of their youth work participation, and counted these as ‘outcomes’. (Many of these certificates were condescending at best; AQA unit 83522 ‘Making tea or coffee’ was the most striking and oft-repeated example.)
  • Young people were ‘incentivised’ with a trip, pizza, cash, and expensive motorbike competence courses, in return for attending, filling out paperwork, or completing a course.
  • Youth workers exaggerated the nature of their project’s achievements in multi-agency meetings.
  • Managers recorded the results of ‘easy to evidence’ projects, sometimes even creating these projects for that purpose, to enable less ‘countable’ projects to happen.

 

When workers shared such tactics – many of which I have used myself – it was often with a palpable sense of embarrassment, sometimes even shame. Blatant ‘making up’ of numbers was not discussed; perhaps it was hidden, but in most cases it was probably just unnecessary. After all, the best lies are usually those that are closest to the truth. It seemed to me that workers were pressured to ‘get their numbers up’, and they were probably expected to use gaming practices, but these were left deliberately opaque (thus it was often grassroots workers left to take most of the risk):

 

“Part of the ‘game’ is knowing which fabrications are desirable and which are unacceptable. Workers are kept guessing: how far should the truth be pushed and made to bend? Should we prepare a special session when the inspectors are due? Should we add a young person’s name to the attendance list if they only popped in for a moment? Should we share our doubts and false starts when we attend a neighbourhood meeting, or focus only on our achievements? Knowing which compromises are acceptable and which are straying too far from the truth requires a deep and habitual familiarity with systems of judgement. These games are complicated; cheating is frowned upon, but providing wholly honest versions will not make the grade.” (Grassroots Youth Work, p.91).

 

Fabrication is a useful concept, precisely because it shines a light on the murky area between truth and lies; it also makes me think about the (sometimes overlapping) impulses of conformity and resistance that are often characteristic of a commitment to youth work. Workers feel compelled to engage in inauthentic practices they do not believe in, yet to some extent this ‘gaming’ of the system is also a form of rebellion that buys space for ‘real’, ‘meaningful’, and less measurable forms of practice.

 

Like Tony, I am interested in what kinds of fabrications will become systemic under the influence of newer impact mechanisms such as ‘pre and post tests’, comparison groups, and Randomised Control Trials. Working these days in a research institution, my sense is that such methods – especially when carried out without the academic rigour of ethical approval processes and peer review – are highly vulnerable to distortion. A number of tactics come to mind, including but not restricted to:

  • focusing practice – or at least its evaluation – on the ‘most engaged’ and ‘most amenable’ young people;
  • measuring a large number of indicators, in the hope that some will ‘prove’ significant;
  • exaggerating the importance of small effect sizes;
  • burying negative evidence; and,
  • presenting data in inaccessible or incomplete ways.

 

The feelings invoked by numbers and ‘scientific’ data in a field are important here. Without clear and transparent communication, numbers can act to obscure, legitimise, and exclude. As a consequence of the neoliberal fashion for measuring and monetising everything, we can be almost certain of a continuing increase in the emphasis on targetted ‘projects’ and ‘interventions’ at the expense of open youth work. If I was a youth work manager now, I might well feel compelled to ensure that we had some easily ‘measurable’ projects, with clear and achievable ‘outcomes’. Of course, I would do my best to make space for grassroots youth work too; but this means that open youth work will continue to exist only where passionate individuals fight to make space for it, rather than being available to young people by right. Surely this is not acceptable.