Missing the bigger picture: why applications statistics need context

A case study of how looking  in isolation at one set of improving figures – applications to university from the most deprived areas in Scotland – makes it easy to miss important questions for the debate about widening access.

In particular, concentrating on the numbers applying to university from poorer backgrounds – often a focus of reporting in Scotland – doesn’t tell us how far extra applications are translating into extra actual students.  There’s little achieved in encouraging more people to apply if their academic track record means they will stand a thin chance of getting a place or if the system doesn’t have space to accommodate them, and application figures alone tell us nothing about whether the system is getting better at recognising the potential of students from disadvantaged backgrounds.  Looking at figures from only one part of the UK – whichever it is – also limits what can be learnt.

Application rates from disadvantaged areas in Scotland since 2006

Yesterday the Scottish Government drew attention to a new UCAS analysis showing that, amongst the SIMD quintiles,  the largest percentage growth in applicants to Scottish higher education since 2006 has been in the most deprived group, in which the rate of application rose by 50%.  As a result, UCAS reports that the gap in the rate of application between the least and most deprived groups has closed over the decade.  Those from the least deprived are areas now 3.5 times likelier to apply than the the most deprived, compared to 5 times more a decade ago (a 30% fall in the ratio).

These are numbers going in the right direction, but they come curiously lacking any context.   It is not a standard UCAS publication but a piece of one-off analysis and the press notice does not explain what or who has prompted it.

It has most probably been produced at someone’s request to support the  work of the Commission on Widening Access.  The existing UCAS analysis for 2014 uses POLAR  classifications as the measure of deprivation, while Scottish policy is based on  SIMD, so there’s sense in checking how the two compare. However, looking at the resulting set of figures in isolation tells only a small part of the story.

Application rates across the UK

For example, these figures cannot be compared with changes in other parts of the UK, for which only POLAR data is available.  Fortunately, the new UCAS note includes a footnote showing how the SIMD results produce similar results to those obtained using POLAR classifications: that may turn out to be one of its most useful conclusions, indeed.

The UCAS application data for the POLAR3 Q1 across the UK over the decade looks like this:

(Table 15/16 here)

For applicants from the most deprived backgrounds from Scotland,  using POLAR in fact produces a larger growth (63% rather than 50%).  The increase is higher than in Wales or Northern Ireland, but a little lower than in England.  The first question then is how well do we understand why the patterns are so distinct in different parts of the UK (access initiatives? demographics? changes in the school system?).  Some of the common assertions about student finance and fees – so often cited as critical  – look oddly irrelevant here, when the two most improved systems are the one  with the highest fees and most debt (England) and the one with free tuition but also unexceptional total living cost support and low grant (Scotland).  For all the political, media and academic interest in widening access, it still feels as though far too little opportunity is taken to investigate what is driving different effects in different parts of the UK.

Closing the gap in application rates

On closing the gap in application rates between the most and least deprived, the POLAR figures for Scotland again show a similar change to those for SIMD (a 33% rather than 30% fall in the ratio), with England and Scotland once again the parts of UK seeing the most improvement.

(As above)

Entry rates

The new UCAS note looks only at applicants, which provides a valuable measure of young people’s changing aspirations.  However, UCAS also produces data on entry rates, and this is important, because these tell us more about who actually gets to university.

Using POLAR data, the entry rate for the most deprived group has risen by 49%, lower than the 63% increase in  applications.  At first sight that suggests  that the increase in  applications is not fully translating into an increase in entrants: however, a different POLAR measure is used here – POLAR2 – which may explain the difference.  It is a missed opportunity therefore that the new UCAS report does not also provide the entry rate on SIMD terms, to see if the same effect occurs if measured that way.

(From here: table 71)

England sees the highest rise in entry rates, but as in Scotland it is lower than the increase in applicants in this group: Wales sees no difference and Northern Ireland (intriguingly?) sees a larger improvement in entry rates than applications.  So using different POLAR measures does not consistently pull the figures in any particular direction. Still, at first sight something interesting seems to be happening in the Northern Irish system – they look to have become better at converting applicants from disadvantaged backgrounds into entrants (despite having a tight cap on numbers within Northern Ireland itself).

Maybe the lower growth in applicants from this group in Northern Ireland means that those who do apply tend to be better-qualified and that increases their success rate.  Could fixing too hard on increased applicants as a measure of success risk tying up increasing numbers of young people in a process where they have a limited chance of  success, for the sake of a making the system look more successful than it is, when it comes to providing actual opportunities? Comparative data doesn’t answer such questions, but it helps them get asked.

Closing the gap in entry rates

Turning to the gap between the entry rates for the least and most deprived, this has closed in Scotland at a similar rate as it has for applications.  That still suggests that the progress in closing the gap in actual entrants  may be largely down to changes in applications.  How far have success rates for applicants from deprived backgrounds changed? How is that pool changing? Perhaps mostly fundamentally, does an applicant with a particular set of qualifications from a deprived background have any more chance of being accepted than they did 10 years ago, compared to one from a better off background?  Looking at application figures in their wider context doesn’t directly tell us the answers but, again,  helps form the questions.

(As above: table 73)

The comparison of change in application and entry rates may mean less than first appears and be due to use of POLAR3 for one and POLAR2 for the other: but the comparison brings out the importance of analysing what  has changed – or not – at each stage in the system.

Conclusion: seeing the bigger picture

Any new data here is potentially useful.  But in this case the absence of any comparative context – whether with  other parts of the process or other parts of the UK – makes this document on its own less useful.    There is progress here to be celebrated – but concentrating on Scotland-only figures, for just one stage of the UCAS process, carries the risk of failing to look at the operation of the whole process from application to  entry, as well as reducing the scope for some critical distance from policy-making and practice in Scotland.  It is not that some positive reporting isn’t justified,  just that it is not the whole story, in important ways.

Responding to the new UCAS analysis, the Cabinet Secretary said “It is important that the [Widening Access] Commission has access to a full range of evidence”. That must be right and is a good reason why this new analysis is worth putting in a broader comparative context. In particular, while rising applications from disadvantaged young people are a necessary element of widening access,  we  need to get more cautious about what these statistics alone can tell us.