Verdict on recent TES reporting on university access in Scotland: could try harder

The TES recently published this piece.  The opening paragraphs give a sense of its content (emphasis added):

Scotland is doing a comparatively good job of sending poorer students to university, according to new analysis.

Within the UK only Northern Ireland, London and the West Midlands perform better at minimising the impact that poverty has on a pupil’s chance of accessing higher education, according to data analyst Tom Forth.

He broke the UK up into 11 geographical areas and used data from Ucas (Universities and Colleges Admission Service), the Higher Education Funding Council for England (HEFCE) and the End Child Poverty campaign group to rank each on the strength of the link between wealth and a university education.

Tom Forth’s original piece is here. Forth has done the good thing of looking behind the headlines on a topic he’s interested in, finding data and trying to work out the answer for himself (all unpaid, in his own time): how he approached his analysis is discussed in more detail in the footnote below.  To his credit, he publishes all the data his analysis is based on.   He also recognises and admits his limitations, saying:

Lastly, I’m not an expert in the UK’s higher education systems. There’s so much more I’d like to understand but I don’t have the time or the money to do so yet. I’m especially interested by what affect different classifications of higher education vs.  further education between UK nations may have on these results.

He’s right to identify the classification of HE as being important.  Although his piece talks throughout about “university access”, and that’s how he titles his graphs, a look at his data shows he is using the Young Participation Rate, which covers participation in all forms of HE by age 20, including sub-degree HE in FE colleges.  This can checked by comparing his all-Scotland average and with the all-HE YPR for the relevant cohort (the answer is 40% in both cases: Table K here).

That matters in Scotland, of course.  Broken down by sector, the YPR for 2009 (the year Forth looks at)  shows that 26.9% of participation was in HEIs in Scotland, 12.4% in FECs in Scotland and 1.6% in rUK institutions (type unspecified). So 32% of the participation Forth examined where thetype of institution is known was not at university-level and the unknown cases are de minimis.

Other data shows that in Scotland degree level HE is concentrated in HEIs: FECs provide relatively little degree-level HE: table A here.  The complicating factor is that a minority of HN students will progress later to a degree at an HEI (I think as many as one-sixth of HN leavers may “articulate”  to a degree with full credit for the years they have already done, while a similar number may start again in first year – a costly model for those individuals: but I’m not so confident with this data, so don’t quote me).  How to count the Scottish college-to-university group in cross-UK comparisons is an additional headache.

So while the UK comparative analysis is valid if the issue at stake is participation in HE as a whole, it isn’t if it’s access to university, arguments about which in Scotland were the stimulus for the piece

Indeed, given the YPR breakdown, and what can be deduced about the much more limited role of non-university and sub-degree HE in England from HEFCE figures,  if we could run a similar analysis comparing participation and child poverty just for universities or just for degree-level HE, it looks quite likely that the headline findings would be far less in Scotland’s favour.

None of this is intended as any criticism of Forth: he’s been entirely open about the limitations of his own knowledge and understanding, and shared his analysis in a way which makes it possible to check – and remember he has done this as an amateur, in the pure sense.

Less understandable, however,  is why the TES  has not shared Forth’s own caveats with its readers or done what it took me a few minutes to do, in the light of those caveats, and check what data he was using and whether it did indeed relate to university access, as he suggested.

I’m all for independent thinkers who look at the underlying data for themselves in their own  unpaid time and share their results, and inputs, for debate (how could I not be?).  But I’m less impressed by one of the UK’s main education journals deciding to publish a report on a controversial topic based on such a piece work by a self-professed non-expert,  without first doing the sort of straightforward checks on its claims which the author of the original piece himself flagged up might be needed, and are easy to do.   It could still have run the piece (it’s an interesting combination of HE and poverty data), but explained that it only worked insofar as a comparison of all forms of HE was the topic.

Indeed, Forth was quoted in The Scotsman the week before in a piece on access (see here).  There, however, to the credit of both parties, the issue under discussion is clearly participation in higher education and Forth makes it clear that it’s adding the non-UCAS (ie non-university) HE which makes the difference.  When that’s done, Forth comments: “Loud voices in London think Scotland’s higher education system is failing its poor. Loud voices in Edinburgh insist that everything’s great. From Leeds, the exact midpoint of the two cities, my look at the data seems to show something in between.”  If all we are interested in is access to any form of HE, that’s a pretty fair comment: if anything, it may undersell Scotland’s performance on HE once HN-level study in colleges is included. The underlying question then is how to weigh HN-level HE in the overall balance.

TES readers were not so well-served.  The risk is that its piece will be gratefully grasped as a welcome counter-narrative to more challenging (and important) claims about the university access in Scotland and lead to more substantial and uncomfortable pieces of work being dismissed out of hand.  As The Scotsman’s decision to include him as the sole expert voice in its piece shows, the appetite for less gloomy alternative perspectives on the access debate means that his own careful expressions of limited expertise will get easily lost.  That’s especially likely if his work gets picked up on social media, with its tendency to confirmation bias (his original piece has been re-tweeted over 200 times).   But the TES  is in a different game from Twitter et al and ought to publish some sort of correction or clarification  underneath its piece.

Meantime, I hope Forth carries on doing what he does (and if he is up for trying to repeat the analysis by level of HE, when he has the chance and if the data permits, that would be a really interesting read).

Footnote

Finding it difficult to use UCAS data to make satisfactory comparisons, Forth turned to comparing the YPR to child poverty rates by area.  He amassed the data on this at parliamentary constituency level and then reaggregated it into the three devolved nations, plus England divided into 9 regions (making 12 units: the TES wrongly says 11- a minor point, but further suggesting limited engagement with the detail of the piece). Forth’s is a really interesting approach with potential to add to the debate and I’m not sure if it’s been tried by anyone else.

He has the data for both measures in enough detail to produce weighted averages, which I am therefore assuming he has used for reaggregating the figures to regional level.  So his regional-level comparisons ought to be meaningful (as ever, for all HE, but not university-level alone).  He doesn’t provide an all-England comparison, which is a pity, as that’s the level at which policy is determined there: on a very rough estimate, it looks as though England would lie about half-way between Scotland and Wales.  However, his regional breakdown of England is useful, as it shows how much variation there is underneath the English average.  In particular, London – the focus of so much attention and investment in its schools since the turn of the century, via the London Challenge – comes out particularly well, adding further weight to the argument that it’s school-level intervention which makes the biggest difference to widening participation. Meanwhile, the south east of England lies at the bottom of the table. I’m pretty sure that the south east would show the highest persistence of grammar/selective schooling in England, plus use of the private sector (certainly outside the area covered by the London Challenge).  I therefore suspect that Forth’s analysis, though prompted by an interest in UK comparisons and fee policy, in its current form really comes into its own in adding to the debate about the role of the school system, especially within England, and deserves further attention in that context.