Why early data from clearing should be handled with care

The first set of UCAS acceptance figures for this year are due out on Tuesday (4 August), covering the position for Scottish students immediately following highers results.

These figures often attract quite a bit of political and media attention (see for example the 14 August 2015 press notice issued by the SG reporting the 4% increase in Scots accepted onto a course), but need to be taken with a pinch of salt.  As the table below shows, anything published this week, or next, is only the initial position of a fluid process.

This week’s figures will provide information on how many people this year compared to last in absolute terms have got onto one of the courses they initially applied for  –  from the perspective of applicants, it is generally a good thing if this number rises.  But we won’t know whether a higher or lower proportion of applicants did so than last year, or how many and what percentage of applicants have got a place somewhere.  These things for a little longer to emerge.

How the position can change

Last year the year-on-year increase in Scots accepted at the very start of clearing was 4.5% compared to 2.4% by the end of the application cycle for all those who had applied before 30 June and -3.7% for all Scottish applicants in total (but see footnote).

Factors affecting the initial UCAS figures

The figures show that last year more people got one of their initial choices and fewer ended up getting a place through clearing than in 2013.  The number getting the course they initially applied for will tend to be higher when a system is expanding, more places are available and offers may be lower – but it will also increase when exam results are improving.

In same way, when a system is contracting, the number getting their initially offered place might be expected to fall – but worsening exam results would also mean fewer students meet their initial offers but more places are then filled through clearing. In that case, the percentage change from the previous year could rise  as clearing proceeded. So, for example, if concerns about the difficulty of this year’s SQA Higher maths exam were to translate into lower grades, there might be some visible impact on the numbers meeting their initial offer criteria (though the SQA has said that its marking system should adjust for a more-difficult-than-usual exam).

In other words, if the figures on Tuesday show a relatively low percentage increase, or even a fall, it will be surprising but we’ll need to wait a couple of weeks before any real conclusions can be drawn.

Comparability with 2014

There’s also an issue of comparability with 2014.  This year, a large group of teaching training courses not previously using UCAS has come into the system, which is increasing the figures: explained here.  When it published applicant figures earlier this year, UCAS therefore produced two sets for Scotland – the raw data and figures adjusted to remove the new group of students, for like-for-like comparability. Before adjustment, applicants appear to have risen by 10%; with the adjustment, the figure is 2%.

If the clearing figures are not similarly produced in two forms, they will not be much use – any press release or article drawing on the unadjusted data will be extremely misleading.

The underlying big story: responsiveness to rising demand

There’s been coverage lately of signs that the number of Scottish students accepted onto a course (particularly in Scotland, covered by free tuition) is not keeping pace with rising demand, something already discussed elsewhere on this site. Contributing factors identified have been growing numbers of EU students (who compete with Scots for fully-funded places) and declining willingness on the part of Scots to go to the more expensive south (for which the SG offers no portable fee grant). Also, while the total number of funded places is still growing, that growth remains quite tightly capped, particularly relative to England.  Last year, acceptances rose at a similar pace for students from Wales (2.7%), fell for the Northern Irish (0.7%) but rose by 4% for those from England  (table 4 here, p4 of report, p23 of pdf).

This week’s figures won’t tell us anything reliable about the year on year change.  However,  by the end of August last year the numbers had pretty much settled down, so that’s the first time it’ll be worth taking a proper look,  to understand how far the numbers are growing.  By December,  the final position will be clear (presuming the reporting problem with late applicants discussed in the footnote has been fixed).

It would be remarkable if the number of Scottish students getting a place through UCAS does not increase further, as the number of funded places is still increasing.  For the number of Scots to fall would take a very sharp increase in EU students being awarded places in Scotland or for universities to reduce the numbers they take in over and above the official number of funded places.  A bit of unfunded growth may have been something of a safety valve in Scotland in recent years – it’s hard to tell from the statistics – and at some point institutions may find their capacity for that becomes too strained.  However, it’s not likely that either of these two things will kick in with enough force to undermine the general upward trend of recent years.

Potentially more salient will be to compare how the growth in acceptances compares with the growth in applications.   The “acceptance rate” for Scots has fallen markedly over the past decade, including in the last year (as in Northern Ireland, but in contrast to Wales and England)  – see Figure 11 here (p11 of report, p30 of the pdf).  That means Scotland is doing better at encouraging more young people to aspire to university than it is at making sure there’s actually a place for them to go to.

Like-for-like applications were 2% higher this year by the end of June.  The government will probably be hoping to see the increase in acceptances by the end of this month at least matching that figure and – if the trend of falling acceptance rates over the past decade is to start to be reversed – exceeding it.

 Number of Scottish-domiciled students accepted over the period of clearing: 2013/2014 change

UCAS issued daily bulletins for the 2 weeks following A-level results.

Footnote

Frustratingly, the end of cycle figure for 2014 Scottish acceptances  contains a large uncertainty. UCAS reports that:

In 2014 there have been fewer very late acceptances than in previous cycles recorded in the UCAS data for some Scottish providers. These changes may mean that the number of applicants and acceptances to Scottish UCAS providers in 2014 recorded through UCAS could be understated by up 2,000 compared to how applicants and acceptances have been reported in recent cycles. This means that comparing 2014 applicants and acceptances for Scottish providers (or those from Scotland) to other cycles will not give an accurate measure of change. (see note on p5 of pdf here).

UCAS hasn’t provided much detail beyond that.  So it may well be that the -3.7% fall recorded in total acceptances can simply be ignored.  Or it may be that there really were fewer places available to be filled by very late applicants last year – but perhaps not a whole 2,000 fewer.  It is completely impossible to tell. It’s not clear whether the possible change in reporting practice is a one-off or likely to affect this year’s figures also.

A comparability problem therefore lies ahead for this year, too: as things stand,  if the final figure rises again, we won’t be able to tell easily how far it is real and how far a further reporting effect (certainly, come December it will be important not to  look at the 2014 to 2015 change in isolation).

In essence, we can no longer reliably compare year-on-year changes on total applicants and acceptances in Scotland. It’s hard to imagine that the government, opposition parties, institutions, think tanks and the media collectively in England would have stood for a similar large uncertainty being left hanging over the figures there, but we seem to be stuck with it here.

Even if the like-for-like number can’t be pinned down better, it would be helpful if  the UCAS end of cycle report this December could say more about what lay behind the fall (which institutions were affected, for example? did these all change their reporting practice in 2014 and if so why?) and explain in more detail why it believes a recording change may explain all or most of the drop.  It could also helpfully explain how far this year’s figures may be similarly affected.

Meantime, the only reliable comparisons which can be made are for those applicants who applied by the end of June, whose figures are unaffected, which gives most but not all of the picture.