Mr Durham Burt
Deputy Information and Data Protection Manager
Governance and Executive Support
Durham University

Dear Mr Burt,

Please could you provide test marks for the 11+ tests set by The University of Durham's Centre for Evaluation and Monitoring (CEM) in the Autumns of 2014, 2015 and 2016. For each candidate I would like to request the following information
1. School or consortium. (I understand from the Schools Adjudicator that different groups of applicants are ‘locally standardised’ so the reason for requesting this information is to identify which results have been standardised as a group.)
2. For each of the sub-tests set (numeracy, verbal reasoning and non-reasoning) I would like to request
a. The raw test scores for each test before any age weighting is added
b. The candidate’s birth month (or age in months if this is how it is recorded)
c. The raw score for each test with age weighting added, (or the amount of age weighting if this is how it is recorded)
d. The final standardised scores
e. The mean and standard deviation values used to calculate the standard scores.

Please could you provide the test marks in CSV or Excel spreadsheet format.

Secondly please could you provide details of the overall income CEM have received for setting these tests in 2014/15/16.

Yours faithfully,

James Coombs

ACCESS I., Durham University

Name: James Coombs

Email:  [FOI #363147 email]

 

Dear Mr Coombs

 

Freedom of Information Request

 

I acknowledge receipt of your email received 6 October 2016 requesting
information from Durham University.

 

Please be advised that the University will respond as soon as possible and
in any event by 3 November 2016.

 

Yours sincerely

 

Julie Gowland

Information Governance Team

University Secretary’s Office

Durham University

 

show quoted sections

ACCESS I., Durham University

Dear Mr. Coombs

 

Thank you for your request for information from Durham University. Please
see our response below.

 

Please could you provide test marks for the 11+ tests set by The
University of Durham's Centre for Evaluation and Monitoring (CEM) in the
Autumns of 2014, 2015 and 2016.  For each candidate I would like to
request the following information

1.            School or consortium.  (I understand from the Schools
Adjudicator that different groups of applicants are ‘locally standardised’
so the reason for requesting this information is to identify which results
have been standardised as a group.)

2.            For each of the sub-tests set (numeracy, verbal reasoning
and non-reasoning) I would like to request

a.            The raw test scores for each test before any age weighting
is added

b.           The candidate’s birth month (or age in months if this is how
it is recorded)

c.            The raw score for each test with age weighting added, (or
the amount of age weighting if this is how it is recorded)

d.           The final standardised scores

e.     The mean and standard deviation values used to calculate the
standard scores.

 

Please could you provide the test marks in CSV or Excel spreadsheet
format. 

 

Secondly please could you provide details of the overall income CEM have
received for setting these tests in 2014/15/16. 

 

Q2(a) and (c)

 

The University confirms that it holds the information you have requested.
However, the University declines your request for release under section 43
(2) of the Freedom of Information Act (FoIA). Section 43 (2) of the FoIA
exempts information from release where its disclosure under the Act would,
or would be likely to, prejudice substantially the commercial interests of
any person or public authority. As section 43 (2) is a qualified
exemption, the University has performed a public interest test and has
concluded that release of test data would prejudice the commercial
interests of the Centre for Evaluation and Monitoring (CEM).  

 

The University recognises that release of the requested information would
allow individuals to better understand the 11+ tests. However, release of
raw scores with and without age weighting would have a prejudicial effect
upon the commercial value of the assessment for Durham University as it
could weaken the effectiveness of our testing model. The Centre works hard
to design tests that are as resistant to tutoring as possible. Release of
the information could potentially allow tutoring companies to tailor their
teaching to the test. Not only would this unfairly advantage tutored
children it would also undermine the quality of our test and prejudice the
commercial value of the assessment for the Centre.

 

Q1, 2(b)(d)(e) and Details of Income

 

It is estimated that the time taken to locate, retrieve and extract the
requested information would exceed the fees limit associated with the
Freedom of Information Act, 2000 (FoIA). Under Section 12 of the FoIA, the
University is not obliged to comply with a request for information if the
cost of compliance would exceed a £450 threshold, a figure which equates
to approximately 18 hours of work spent searching/retrieving and
extracting requested information. Under Section 16 of the FoIA, the
University is, however, required to provide you with advice and assistance
to narrow the scope of your request to bring it in line with the
appropriate limit. To that end, colleagues have estimated that they could
provide a response to Q1 and 2(b)(d)(e) for a single year rather than all
three, and could also provide a full response to the request for Details
of Income (for all years).

 

Please let me know if you would like to proceed as outlined above, and
also indicate a year for which you would like to receive information.

 

Yours sincerely

 

Sophie Philipson

Information and Data Protection Manager

Durham University

 

 

show quoted sections

Sophie Philipson, Information and Data Protection Manager

Dear Ms Philipson

Thanks for your considered response. To keep the cost of compliance within the limit I can confirm that I would like data for the most recent tests taken in Autumn 2016 for admissions in 2017 but would like to request that someone more senior at the University review its decision to redact this information to remove the raw test marks as these are essential to understanding ‘grammar school standard’.

The University’s application of Section 43 (2) seems to simply repeat the arguments put to the Information Commissioner two years ago in connection with his decision FS50566015 rather than considering this request on its own merits. Since then, further information has been released, not available to him at the time of his decision, which changes altogether this request. Most notably, your sole competitor in 11-plus testing, GL Assessment, has released precisely the same type of information you are refusing to release here; namely the raw marks, broken down by test component.

GL also claim their tests are resistant to coaching but have not sought to prevent release of raw test data on the basis that it might 'weaken the effectiveness of their testing model'. Given that your direct competitor has no qualms about releasing this information I would be very grateful if you could explain exactly how you believe a long list of anonymised raw test scores could possibly allow tutors to 'tailor their teaching to the test' as you suggest. All the more so, as I was never made privy to the arguments given to the Information Commissioner which guided his earlier decision making it very hard to challenge.

The response fails to support even a hypothetical possibility that disclosure would 'prejudice substantially' the University’s commercial interests, given that; both CEM and GL Assessment maintain their tests are resistant to coaching (ie this selling point is not unique to CEM), neither has any evidence to support this, and GL have already placed raw scores from their tests in the public domain.

Notwithstanding failure to demonstrate commercial prejudice, section 43 is qualified and the University are therefore required to objectively consider arguments both for and against disclosure. The response makes no more than a passing reference that, ‘release of the requested information would allow individuals to better understand the 11+ tests’ which belittles the real significance of this information. Access to this information is the only way the public can truly understand differences in the levels of difficulty needed to gain a place at different selective schools. It seems altogether very wrong that places at publicly funded schools should be decided through tests, the results of which are withheld from the public. This comes at a time when the government is repeatedly deploying the notion of 'grammar school standard' to provide a rationale for its plans to extend selection. It is therefore more important than ever for parents to have insight into whether or not such a ‘standard’ exists or could exist to facilitate a properly informed debate on education policy.

A full history of my FOI request and all correspondence is available on the Internet at this address: https://www.whatdotheyknow.com/request/c...

Yours faithfully,

James Coombs

ACCESS I., Durham University

Thank you for your message. I am currently out of the office and will
respond as soon as possible following my return on 21 November 2016.

Kind regards,

Ami-Jayne Browning

Information Governance Officer

 

ACCESS I., Durham University

Name: James Coombs

Email: James Coombs <[FOI #363147 email]>

 

Dear Mr Coombs,

 

Internal Review Request

 

I acknowledge receipt of your email dated 18 November 2016 requesting an
Internal Review.

 

Please be advised that the University will respond as soon as possible and
in any event by 19 December 2016.

 

Yours sincerely,

 

Deborah English

Information and Data Protection Support

t: +44 (0) 191 334 6103  e: [Durham University request email]

 

CONFIDENTIALITY NOTICE:

This message is intended solely for the addressee(s) in the first instance
and may contain confidential information. If you are not the intended
recipient, please notify the sender, delete the message from your system
immediately and do not disclose the contents to any other party.

 

 

show quoted sections

ACCESS I., Durham University

1 Attachment

Dear Mr Coombs,

Further to your email below, please find attached the outcome of your Internal Review.

Kind regards,

Ami

Ami-Jayne Browning
Information Governance Officer
Information Governance Unit
Durham University

show quoted sections

Dear ACCESS I.,

For your information here is a copy of a letter I sent to the Information Commissioner on 24 December requesting they consider whether you have correctly applied section 43(2) of the Freedom of Information Act in response to this request.

Dear Information Commissioner’s Officers,
I would like to raise a new case with you regarding the way the University of Durham’s Centre for Evaluation and Monitoring (CEM) have handled my request for information.
A full history of the correspondence can be found at the following link https://www.whatdotheyknow.com/request/c...
Summary of correspondence
6/10/16 Requested the raw marks from 11+ tests set by Durham University’s Centre for Evaluation and Monitoring (CEM)

27/10/16 Authority refused relying on s43(2)

18/11/16 Requested a review of the decision

19/12/16 Authority upheld their earlier decision

Relevant background to the request
1. There are 163 grammar schools in England which are permitted to select pupils on the basis of “11+” entrance tests. Of these 139 (85%) have converted to Academy status meaning they have full control over their intake with no independent oversight such as there was when admissions were managed by elected Local Authorities.
2. The First Tier Tribunal (EA/2015/0226) established that CEM are, “one of the two main commercial providers in the UK of 11+ testing” (§6), that they provide 40% of 11+ tests (§20) and earn over £1m (§21) annually from this. Selective schools are extremely reticent for any information about entrance tests to be released into the public domain and given the sums of money involved it is clear that they exert a great deal of commercial influence over what information CEM feels at liberty to release.
3. Each school might typically examine 1000 children whilst some local authorities test all children so the total numbers of tests taken each year may be in the order of 200,000 – 300,000. The test consists of three sub-tests, Verbal Reasoning, Maths and Non-Verbal Reasoning. The authority’s reply also uses the terms sub-section and section interchangeably to mean the same thing.
4. Grammar schools frequently group together in consortia to provide a single test for admission to any of the consortium members so, whilst CEM might provide tests for an estimated 65 schools (40% of the 163) the number of organisations they deal with is likely to be far fewer and the effort of redacting personal information to prepare information for public release probably entails just removing some columns from a spreadsheet.
5. Grammar schools prevent parents from ever seeing the actual test results instead providing what they describe as standardised scores. They also maintain that only children who are of grammar school standard may be admitted. Schools are publicly judged by the final GCSE results their students achieve but this is largely dependent upon pupil’s prior attainment upon admission, meaning if selective schools were able to somehow raise grammar school standard, public perception of them would also be correspondingly raised.
6. Both CEM and their main competitor GL Assessment explain that, when done correctly, a standardised score is calculated by comparing the individual score against ‘as wide and representative a population as possible,’ ie everyone else who took the test, but at least one admissions authority has privately admitted to the Schools Adjudicator that instead of comparing students against national standards the standardised scores (sic) they give parents only compares applicants against each other.
7. It is entirely reasonable for the public to expect that grammar school standard shall be invariant and that standardised scores relate to a wider measure than just those pupils who apply to one school in one particular year. By preventing the raw test marks from being released into the public domain grammar schools are masking any changes to these so called standards.
Relevant legislation
8. The authority maintain that providing three years’ data would also have been exempt under s12 whilst a single year’s data would not. I am sceptical of this claim but agreed to limit my request to one year’s data meaning that authority’s refusal to provide raw test marks depends entirely on section 43(2) which states, “Information is exempt information if its disclosure under this Act would, or would be likely to, prejudice the commercial interests of any person (including the public authority holding it).” This is a qualified exemption. The authority must first prove that their commercial interests would be damaged and secondly that the greater public interest lies in non disclosure.
9. Considering that the authority have simply re-iterated old arguments for refusing to release this information their responses seem to have taken a long time and the information Commissioner may also like to consider whether the authority properly complied with section 10 of the FOIA. His guidance (§25) says, “an authority which provides its response close to, or on, the final day of the 20 working day limit ought to be able to both account for, and justify, the length of time taken to comply with the request.” https://ico.org.uk/media/1165/time-for-c.... To date CEM have not released any information in relation to this request, even that which they agreed to.
Commercial Interests
10. The authority maintain that s43 is correctly engaged because releasing the test scores would ‘prejudice substantially’ their commercial interests because it would ‘allow tutoring companies to tailor their teaching to the test.’ I asked them to explain how releasing a long list of anonymous test marks could possibly allow tutors to tailor their teaching to the test as they’ve always been extremely vague on this point. Their review claims release of this information would enable tutors to, ’deduce further information about the test’, which they go on to define as
a. The number of questions in each test
b. The relative difficulty of each subtest.
11. Dealing with the first of these points, it’s hard to see how knowing the number of questions in a test could possibly be of any use to tutors and if it were, they could simply ask one of the hundreds of thousands of children who sit these tests every year.
12. As to the second of these points, CEM have already released a detailed analysis into the relative facility of each subtest. This is broken down by ethnicity, gender and FSM and so provides considerably more information than CEM claim could be gleaned from the requested information. Notwithstanding this, examinees are prevented from focussing on certain subtests because CEM require each one to be administered in a separate timed session.
13. The notion that a long list of anonymised test scores could somehow benefit tutors is so clearly without any rational basis that the real reason CEM do not want to release raw test results has to be something they have, to date, chosen not to share with either the Information Commissioner (FS50566015) the First Tier Tribunal (EA/2015/0226) or myself.
The authority’s public interest test
14. Despite the lack of any evidence to support CEM’s claim that disclosure would prejudice substantially their commercial interests I asked them to consider the overall public interest, pointing out that withholding this information prevents public understanding of the inconsistency of grammar school standard at a time when the government are considering extending selective education. The review was conducted by the University’s Chief Finance Officer who made it very clear that the only factor CEM considered of any importance is their ability to profit from these tests.

Other considerations for the public interest test.
15. In 2014 Altrincham Boys Grammar School (ABGS) replaced their non-verbal test with an English test. The Schools Adjudicator asked ABGS to provide assurances that this would not disadvantage children with special educational needs (SEN) or who spoke English as a second language (EAL). ABGS asked GL Assessment to provide a report on this, but refused to release it to the public: (https://www.whatdotheyknow.com/request/r...)
16. The report was eventually released by the Schools Adjudicator: (https://www.whatdotheyknow.com/request/r...). It concluded that specific types of subtest do prejudice some pupils but said the choice of test was the responsibility of the admission authority who as already established have a conflict of interests. The Schools Admissions Code, the legislative framework under which admissions to state funded schools must operate, specifically states (§1.8) ‘Admission authorities must ensure that their arrangements will not disadvantage unfairly, either directly or indirectly, a child from a particular social or racial group, or a child with a disability or special educational needs.’ It is in the public interest to know if grammar schools are systematically and routinely breaking the law by using certain types of subtest, particularly at a time when the government are planning to expand this type of system.
17. In 2013 the fully selective county of Buckinghamshire changed from GL Assessment to CEM as their test provider. A report published by Comprehensive Futures (http://comprehensivefuture.org.uk/wp-con...) found, "The gap between the 11+ pass rate for Bucks and non-Bucks children has grown significantly since the introduction of the new exam.” Since 11+ tests claim to measure innate ability, either Bucks children’s ability is mysteriously on the decline or grammar school standard is being surreptitiously raised. A policy of routine open and transparent disclosure of this objective quantitative information, as recommended by the Information Commissioner, would disclose which of these possible explanations is driving this change and would be very much in the public interest, but the schools who commission these tests and exert commercial influence over the test providers are clearly reluctant for such information to be made public.
18. There is also public interest in the public expenditure which depends on the outcome of these tests. The annual budget for maintaining 163 state schools each having 5 forms of 150 pupils at the basic minimum annual funding of £3,000 per pupil is £366,750,000.

Yours sincerely,

James Coombs

ACCESS I., Durham University

1 Attachment

Dear Mr Coombs

 

Further to our recent correspondence with the Information Commissioner’s
Office (ICO), I am writing to inform you that we have overturned our
decision to refuse part 2b) of your request:

 

2. For each of the sub-tests set (numeracy, verbal reasoning and
non-reasoning) I would like to request

a. The raw test scores for each test before any age weighting is added

b. The candidate’s birth month (or age in months if this is how it is
recorded)

c. The raw score for each test with age weighting added, (or the amount of
age weighting if this is how it is recorded)

d. The final standardised scores

e. The mean and standard deviation values used to calculate the standard
scores.

 

You will find, therefore, a spreadsheet containing the anonymised
datasets for the 2016 tests (2017 entry).  There is one worksheet tab per
school/consortium.

 

I hope you find this information useful.  If you have any concerns about
the information provided, I would ask you to contact the ICO given that
the request is the subject of an ongoing complaint  

 

I am copying this response to Paul Warbrick, Senior Case Officer at the
ICO.

 

Kind regards

 

Elaine Richardson

Information Governance Officer

Durham University

 

 

James Coombs left an annotation ()

For anyone trying to follow this. In October 2016 CEM claimed parts a & c of my request were exempt under s43(2). They agreed to provide the rest of the information but didn’t release anything until six months later after the Information Commissioner became involved .
They say they've overturned their decision to refuse part b, although they've never said this was exempt from disclosure. They’ve not released part e of the request or ever provided a reason why they believe it should be exempt and I presume they still maintain that parts a & c are exempt under s43(2).

In summary the original request consisted of:
a. The raw test scores for each test before any age weighting is added
b. The candidate’s birth month (or age in months if this is how it is recorded)
c. The raw score for each test with age weighting added, (or the amount of age weighting if this is how it is recorded)
d. The final standardised scores
e. The mean and standard deviation values used to calculate the standard scores.

ACCESS I., Durham University

1 Attachment

Dear Mr Coombs

 

Further to our recent correspondence with the Information Commissioner’s
Office (ICO), I am writing to supply the information in response to part
2(e) of your request.

 

2. For each of the sub-tests set (numeracy, verbal reasoning and
non-reasoning) I would like to request

a. The raw test scores for each test before any age weighting is added

b. The candidate’s birth month (or age in months if this is how it is
recorded)

c. The raw score for each test with age weighting added, (or the amount of
age weighting if this is how it is recorded)

d. The final standardised scores

e. The mean and standard deviation values used to calculate the standard
scores.

 

I hope you find this information useful.  If you have any concerns about
the information provided, I would ask you to contact the ICO given that
the request is the subject of an ongoing complaint  

 

I am copying this response to Paul Warbrick, Senior Case Officer at the
ICO.

 

Kind regards

 

Elaine Richardson

Information Governance Officer

Durham University

 

 

James Coombs left an annotation ()

The latest information released is NOT what was requested and is of no use to anyone.

The Information Commissioner will not investigate whether CEM failed to comply with s10 although all the evidence needed is here in this request.

Despite new evidence which has come to light since FTT ruling EA/2015/0226 the Information Commissioner is taking that earlier ruling as having established a precedent that CEM may use s43 to protect their commercial interest in "tutor-proof tests".

CEM have never claimed to be able to provide tutor-proof tests although their application of s43(2) is based on them profiting from providing such tests.

This will need to go to appeal as soon as the Information Commissioner has published his decision.

S. Lalani left an annotation ()

Wishing you all the very best in the appeal.

Publishing the raw marks as well as their uniform equivalent ought not to be so contentious! This is common practice for other assessments (including GCSE, A Level etc) and the public interest is significant.

The process of converting from raw to uniform marks is meant to keep the test fair year on year, recognising that an exam paper can vary in difficulty from one year to the next.

Unless the CEM test is so unreliable that the raw marks are a source of embarrassment to the university, changing dramatically year on year, I don't understand why their release could affect them commercially.

The same could be said for other examining bodies, however. Yet there is transparency and raw marks are released. Sometimes grade boundaries at A Level or GCSE change significantly from one year to the next (in terms of raw marks). That's just how it is.

Looking for an EU Authority?

You can request documents directly from EU Institutions at our sister site AskTheEU.org . Find out more .

AskTheEU.org