The following is an extract from a draft report that is currently being considered through the committee structure at Edge Hill University to assess the validity of the methodology for Equality Impact Assessments. It has previously been to the Widening Participation Group on Thursday 21st January 2010 and will be discussed at the Equal Opportunities and Student Support Committee meeting on Wednesday 17th February 2010. It shows the new approach we are hoping to implement in the light of new data being available, in particular, via the University and Colleges Admissions System (UCAS) and also new methods internally of looking at data.
For the purposes of responding to the FOI request in relation to impact assessments based on Race, only data in respect of this aspect has been extracted from the draft report. However the general commentary on how the cohort samples have been identified and the issues that arise with this are included.
Edge Hill University
Equality Impact Assessment - Data Analysis (DRAFT)
Stuart MacFarlane February 2010
Introduction
This report analyses data on UCAS applications to Edge Hill in the 2008 cycle, and the retention of full-time degree students in the 2005-2008 cohort, to assess the impact of the Admissions Policy and the Teaching and Learning Strategy on a number of cohort subgroups.
A major purpose of the report is to test the methodology used, so that it can be refined in order to carry out a more focused examination of some more recent data.
The Applicant/Student Journey
The journey of a potential student into, through, and out of Edge Hill is illustrated in simplified form in Figure 1. Each of the stages and transitions is discussed in more detail below, but, in essence, the journey comprises a number of stages, which each individual either completes successfully (from Edge Hill's viewpoint), or does not.
This analysis for the Equality Impact Assessment looks at the progress on this journey of several subgroups of (potential or actual) students. The method used here is to look at the percentage of `successes' at each transition on the journey, and to check (where possible) whether there is evidence that subgroups of the population are affected equally. If there is a difference, then there may be an equality issue to address.
For each stage, the (potential) student body will be divided into subgroups according to the relevant characteristics of the individual, and according to the programme areas they are applying for, or studying.
Figure 1 - The Applicant-Student Journey
Individual Characteristic Subgroups
For each of the five variables of interest, two or three classifications are used. Numbers are too small in most cases to allow further subdivision (for example into different non-white ethnic groups).
Gender
Male vs Female
Ethnicity
White vs BME
Disability
Disability declared vs No known disability
Age (on 1 Sept before start of course, or at the end of the application cycle)
Under 21 vs 21 to 25 vs over 25
Deprivation level of area of home address
40% most deprived parts of England vs the 60% less deprived
Applicants and Students Included
For this Equality Impact Assessment, a decision was taken to concentrate on applicants via UCAS, and on full-time students on bachelor's degree programmes. Details of exactly which applicants and students are included in the analyses, and why, are given in sections 3 and 4 below.
Programme Areas
A hierarchy of subdivisions is used with results given for the whole sample, each faculty, and subdivisions within each faculty, subdivisions which differ somewhat between the sections on applications and students. Details are given in the relevant sections below.
In some cases numbers in subgroups become too small for meaningful analysis, so only the higher levels of the hierarchy are used.
Stages and Transitions
Figure 1 shows four major transitions, from enquirer to applicant, from applicant to confirmed place holder, from confirmed place holder to student, and from student to successfully completing student. Each of these is considered further in the sections below, and consideration is given to stages that would be amenable to equality impact analysis.
Enquirers
Records are kept on some 25000 enquirers each year, and around 2000 of these later apply for a place. In principle it could be possible to examine whether subgroups of enquirers are equally likely to apply, but details of the personal characteristics listed above are not currently recorded for enquirers, so such an analysis is not yet possible.
Applicants
Students arrive at Edge Hill by several different routes:
Applicants for full-time undergraduate courses normally apply through UCAS.
Applicants for PGCE go through GTTR (administered by UCAS, but the application process and data collected is different).
Applicants for postgraduate, part-time undergraduate, and some CPD programmes apply direct to Edge Hill on paper or on-line forms.
In addition, many students on CPD courses in Health and Education don't formally apply to Edge Hill at all; they are selected or nominated by their employers. No EIA assessment is possible for these potential students.
Applicants through the first three routes above are ultimately `successes' if they firmly accept an unconditional offer from the University, and `losses' if they do not. It is also possible to look separately at the various stages at which applicants are lost.
More detail of the applicant analysis, and the findings, appear in Section 3 below.
Confirmed place holders
Applicants become successful applicants when they confirm their acceptance of an unconditional offer of a place at EHU. Most such offer holders subsequently enrol at the University, but a small proportion fail to appear at enrolment. An analysis could be done of the absentees, by personal characteristics, but the numbers involved are almost certainly too small for any meaningful conclusions to be drawn, so this option will not be pursued.
Students
Once students enrol on a programme of study, they may withdraw or fail, or they may achieve their target award. Some students eventually achieve the target award, but not in the target time scale, for example taking 4 years to complete an undergraduate degree, after intercalating or repeating a year. For the purposes of this analysis, a `success' is achieving the target award in the target time; a `loss' is failing, withdrawing, being awarded a lower award, or failing to complete in the target time. The process used here is to track a cohort of students through their entire programme, looking at the most recent intake whose final results are available.
For programmes lasting more than one year, it is possible to go further, and to look at the success/loss rate at the end of each year of the programme. Students who are still on course to succeed in the target time scale are `successes', and those who are not are `losses'.
More detail of the student analysis, and the findings, appear in Section 4 below.
Applicant analysis
Applicants through the UCAS, GTTR, and direct routes are dealt with differently, but essentially all applicants pass through the process illustrated in Figure 2, which shows a six-stage decision process. Each of the stages is discussed below.
On receipt of an application, EHU makes a decision, unless the applicant withdraws first. Most decisions are made quickly before the applicant has time to withdraw, but for courses that interview or audition applicants before the decision, applicants might withdraw because they are deterred by the process, or because in the meantime they have received offers from other institutions. It would be useful to analyse whether the interview process has differential effects on the subgroups of interest, but it is not possible to tell from the Admissions data whether applicants withdraw before or after interview. This will be possible in future.
No useful analysis is possible on the current data.
EHU staff make a decision, either to reject or to offer a place, conditionally or unconditionally. Sometimes the offer is for a different course or entry date to the one applied for.
Here it should be possible to check whether all subgroups are treated equally. As above, it would be better to be able to check whether the rejections followed an interview (or a no-show at interview), but this isn't possible yet.
The applicant accepts or declines the offer.
It should be possible to check whether all subgroups respond proportionally.
For those holding conditional offers, EHU receive results and either confirm or reject the application. Generally applicants who meet the criteria will be accepted, and those who don't will be rejected, but some who miss the conditions will still be offered a place.
In principle it should be possible to analyse applicants who are offered a place after missing the criteria, but numbers are likely to be too small to produce meaningful results.
After receiving a confirmed offer of a place, applicants should respond. They may accept or decline.
It should be possible to check whether all subgroups respond proportionally.
Finally, having firmly accepted an unconditional offer, applicants either enrol, or disappear.
The numbers failing to enrol at this point are too small for a meaningful analysis of this stage.
A complication is that UCAS and GTTR do not release to EHU details of some personal characteristics of applicants, including ethnicity, until the applicant accepts an unconditional offer. The details of applicants who do not get this far in the process are normally released to EHU only in summary form. However, it is possible to purchase the required data from UCAS; ethnicity data on applicants has been purchased to enable this analysis.
UCAS applications
Choice of cohort
The following sections examine the progress of the 11448 applications (from 10025 different applicants) made to EHU in the UCAS 2008 cycle. These are applications submitted in the main UCAS scheme (not Extra or Clearing) from September 2007 until June 2008. Most of these applications were for entry in Autumn 2008, but some were for Spring 2008, or for later years.
A benefit of choosing this definition for the cohort was that totals could be checked against the UCAS management statistics provided each year to the University.
An alternative method would be to consider all the applications for entry in a particular year; this would have the benefit that it would be possible to follow the cohort through the application process and into their courses. However, it is much more difficult to isolate these applications from the database.
Analysis approach
Applications were classified by `Department' using the UCAS course code applied for, and using the set of `Departments' used in Admissions data (with a small number of changes), which don't correspond exactly to actual EHU departments.
The initial analysis looks at the proportions of various groups in each department who, having applied, receive an offer. However, about 5% of applications result in an offer for a course other than the one that was applied for. This raises the issue of how the proportion of offers is counted. Two methods were considered; either counting only offers for the course applied for, or counting any offer in response to the application. It is not clear which is more meaningful; in some ways an offer for an alternative course is like a rejection from the original course chosen, but in other situations, the alternative offer might be seen by the applicant as an improvement on the original choice. For the analysis of gender both methods were tried to see whether any differences were apparent in the results; they were similar.
Counting only offers that were for the course applied for has the effect of reducing all of the percentages; this is likely to mean that any differences between subgroups of applications should be slightly more visible. Hence this definition was chosen; the columns below headed `Offers' show the percentage of applications that received a conditional or unconditional offer from Edge Hill. The columns headed `Accepts' show the percentage of offers that resulted in a firm or insurance response from the applicant.
Note that these measures should not be interpreted as a `success rate' of any kind, nor should they be used to compare departments or faculties. The calculations below are only to compare the treatment of subgroups of the applicants.
Simpson's Paradox
Simpson's Paradox is a statistical effect that occurs occasionally when taking averages of items that differ significantly in a number of ways - the average can be misleading. This effect occurs rather strikingly in several places in the data below; this is because Edge Hill has a number of programmes that are very oversubscribed, and thus make offers to only a small fraction of applications. This includes programmes in Health, Education, Performing Arts, and Social Work. At the same time many of these programmes attract applicants who, compared to the University average, are disproportionately female, and more likely to be older.
In such circumstances, looking just at the headline figures for the applications to the University overall, can be very misleading (for example, see section 3.2 below). In fact any average could be affected, and all of the figures given in this report are averages of results for a number of courses. Hence care has been taken to separate out courses that might otherwise give misleading averages - in particular, Social Work is listed separately from the rest of the SPS department, as the profile of applications is very different, and it is also much more oversubscribed that the department's other programmes.
Gender
Age
Disability
Ethnicity
Ethnicity data on applicants is not normally provided to the HEI until an offer has been firmly accepted; the ethnicity of applicants who don't become `firm accepts' is not usually released to the HEI at all. For the purposes of this analysis, data on the ethnicity of all of the applicants in the cohort was purchased from UCAS.
Because numbers for each separate ethnic group are too small for meaningful analysis, they have been grouped into `White' and `BME'. `Unknown' includes those who refused the information, about 1% of the total.
|
Applications |
Offers |
Accepts |
||||||
|
BME |
White |
Unknown |
BME |
White |
Unknown |
BME |
White |
Unknown |
FAS |
480 |
6248 |
105 |
73% |
84% |
77% |
31% |
37% |
28% |
Bus |
92 |
752 |
30 |
84% |
94% |
73% |
21% |
31% |
18% |
Eng+His |
47 |
832 |
13 |
96% |
95% |
85% |
33% |
34% |
18% |
Law+Crim |
64 |
602 |
8 |
98% |
98% |
75% |
17% |
28% |
0% |
Media |
60 |
808 |
18 |
65% |
74% |
83% |
36% |
41% |
53% |
NGAS |
6 |
231 |
3 |
100% |
94% |
100% |
50% |
30% |
33% |
Perf Arts |
40 |
556 |
11 |
43% |
52% |
55% |
71% |
55% |
83% |
Social Wk |
58 |
311 |
3 |
3% |
12% |
0% |
0% |
65% |
|
Sport |
64 |
1502 |
11 |
88% |
92% |
91% |
32% |
41% |
10% |
SPS |
49 |
654 |
8 |
96% |
97% |
100% |
40% |
33% |
25% |
FoE |
155 |
2363 |
27 |
33% |
40% |
19% |
73% |
69% |
80% |
Educ |
1 |
48 |
2 |
100% |
35% |
0% |
100% |
71% |
|
KS 2/3 |
20 |
293 |
1 |
35% |
45% |
0% |
71% |
70% |
|
Primary |
103 |
1805 |
19 |
33% |
39% |
21% |
76% |
70% |
75% |
Secondary |
31 |
217 |
5 |
29% |
41% |
20% |
56% |
67% |
100% |
FoH |
186 |
1843 |
41 |
28% |
29% |
24% |
68% |
69% |
70% |
Health |
13 |
96 |
4 |
38% |
43% |
75% |
80% |
63% |
33% |
Midwifery |
24 |
382 |
6 |
17% |
5% |
0% |
100% |
95% |
|
Nursing |
110 |
1203 |
27 |
33% |
33% |
22% |
58% |
64% |
83% |
ODP |
39 |
162 |
4 |
21% |
49% |
25% |
88% |
90% |
100% |
Grand Total |
821 |
10454 |
173 |
56% |
64% |
55% |
40% |
44% |
35% |
The data indicates that applications from BME applicants in Education, and in the `selecting' departments in FAS (Performing Arts, Media, Social Work), appear less likely to receive an offer than their white counterparts. No difference is evident in FoH. There is no evidence of any differences in whether offers were accepted.
It is difficult to do further analysis of this data to explore the reasons for these findings, as the application forms and some details of the processes followed, have now been destroyed in accordance with Data Protection policies. However, more detailed analysis of the 2009 data will be possible.
Deprivation
Student Analysis
Figure 3 illustrates the successes and losses for full-time three year degree students (on programmes starting in September).
Figure 3 - Successes and losses during a three year degree programme
The following analysis looks at the 1718 students who began three-year full-time degree programmes in September 2005. The students are tracked for three years, and the percentages who successfully completed each year and re-enrolled (or graduated) by September 2006, 2007, and 2008 have been calculated for each Faculty and Department.
Note that this is a particularly severe criterion for `success' - for example some of those `lost' on this measure will in fact graduate later, having taken an extra year for a variety of reasons. The measure used here should not be regarded as a `#success rate' for Edge Hill students; and should not be used to compare departments, as circumstances differ. It's use here is to compare the performance of different subgroups within departments; a severe measure is used as it shows any differences more clearly.
The cohort
Gender
Age
Ethnicity
Ethnicity was assessed from the ethnicity codes stored in the student record, most of which come from UCAS application forms. Of the 1718, 21 refused to give information, and 5 are unknown (and omitted from the table). Because numbers for each separate ethnic group are too small for meaningful analysis, they have been grouped into `White' and `BME'.
Just 72 (4.2%) of the cohort were BME, so at the departmental level numbers are too small to draw conclusions. Data is presented at Faculty level.
|
Total Students |
Completed year 1 |
Completed Year 2 |
Completed Year 3 |
||||||||
|
White |
BME |
Refused |
White |
BME |
Refused |
White |
BME |
Refused |
White |
BME |
Refused |
FAS |
1082 |
55 |
17 |
74% |
67% |
41% |
62% |
62% |
29% |
52% |
49% |
24% |
FoE |
407 |
13 |
2 |
95% |
92% |
50% |
88% |
92% |
50% |
81% |
92% |
50% |
FoH |
131 |
4 |
2 |
95% |
100% |
100% |
87% |
100% |
100% |
67% |
75% |
50% |
Grand Total |
1620 |
72 |
21 |
81% |
74% |
48% |
71% |
69% |
38% |
60% |
58% |
29% |
There seems to be no reason to suspect that white and BME students from this cohort performed differently.
For information, the table below shows the percentages of white and BME students in each department in this cohort.
|
Students |
White |
BME |
White % |
BME % |
FAS |
1156 |
1082 |
55 |
94% |
5% |
Business |
76 |
65 |
6 |
86% |
8% |
Criminology |
49 |
47 |
0 |
96% |
0% |
English & Hist |
144 |
135 |
6 |
94% |
4% |
I.T. |
43 |
42 |
1 |
98% |
2% |
Law |
35 |
30 |
5 |
86% |
14% |
Media |
189 |
180 |
7 |
95% |
4% |
NGAS |
33 |
32 |
0 |
97% |
0% |
Performing Arts |
95 |
90 |
4 |
95% |
4% |
Psychology |
69 |
66 |
3 |
96% |
4% |
Social Sciences |
146 |
130 |
15 |
89% |
10% |
Sport |
277 |
265 |
8 |
96% |
3% |
FoE |
424 |
407 |
13 |
96% |
3% |
KS 2/3 |
75 |
69 |
4 |
92% |
5% |
Primary |
303 |
294 |
7 |
97% |
2% |
Secondary |
46 |
44 |
2 |
96% |
4% |
FoH |
138 |
131 |
4 |
95% |
3% |
Total |
1718 |
1620 |
72 |
94% |
4% |
Disability
Deprivation
Conclusions
A weakness of the methodology used here is that each variable has been analysed separately. It is likely that there are interactions between some variables that might provide explanations for some findings. For example, the good performance of students over 25 may be in part because many of the students in this age group are women, and women appear to perform better. However, a full analysis of all possible relevant variables and their interactions would be complicated and time-consuming.
One particular factor that might interact with other factors, and that has not been considered in the analysis, is previous educational performance, which can be considered a proxy for the `academic quality' of a student. This could be assessed using the total tariff points for enrolled students, but would be more difficult to take account of for applications, since the majority have not completed their qualifications when they apply.
Another factor that should be investigated, if possible, is the type of qualification - `traditional' or `vocational'. Detailed data is available on the qualifications of applicants and students, but it may not be complete, and it may not be straightforward to distinguish the two types.
Where a group of interest is small, it can be difficult to draw any conclusions from calculations such as those above. For example, the numbers of students in ethnic groups other than `White', and the numbers declaring disabilities, are sufficiently small that any differences in performance may not be visible. It may be more appropriate to assess the impact of policies on such groups by other methods.
In the analysis of applications, there is little sign of differences in patterns of acceptances of offers, and it may be appropriate to drop this from future investigations. Instead it would be useful to investigate, where possible, the different ways in which applications are `lost', including withdrawals, and the reasons for rejection, such as inappropriate qualifications, or poor interview performance.
With reference to the retention analyses, for most factors it appears that the first year progression data predicts what is likely to happen in subsequent years. Hence it should be possible to analyse much more recent data, such as the 2008 intake, rather than having to wait for the cohort to complete the entire programme.
An issue that remains to be settled is how courses should be allocated to `departments' in analyses such as this. For a number of reasons, different classifications were used here for the applications and retention analyses.
For degree students, an award of an ordinary degree at the end of year 3 is regarded as a success, while the award of a DipHE is regarded as a loss.
Regarded as being three years for an undergraduate degree.
14
Other applicants
Losses
Successes
Students who leave or fail to complete target award in target time
Offer holders who don't turn up
Applicants who withdraw, decline, or are rejected
Enquirers who don't apply
Successful students
Registered students
Confirmed offer holders
Applicants
Enquirers
Students graduating by September in year 3
Losses
Successes
Vanish
Those who withdraw, intercalate, achieve a lower exit award, or fail during or at the end of year 3
Those who withdraw, intercalate, or fail before start of year 3
Those who withdraw, intercalate, or fail before start of year 2
Students enrolling for year 3
Decline place
Students enrolling for year 2
Students enrolling for year 1
Enrol
Confirm final acceptance
If conditional, reject after results
If conditional offer, confirm place after results
Accept offer
Losses
Successes
Figure 2 - The Application Process
Decline offer
Withdraw
Reject
Conditional or unconditional offer
EHU decision, possibly after interview
Application