Errors in the 2019 Secondary Transfer Test

Response to this request is long overdue. By law, under all circumstances, The Buckinghamshire Grammar Schools should have responded by now (details). You can complain by requesting an internal review.

Dear The Buckinghamshire Grammar Schools,

I would be very grateful if you could provide the time and details of communication by which TBGS were informed there were errors in the 2019 secondary transfer test.

I would be very grateful if , for each testing centre, you could provide me with the following information regarding the 2019 secondary transfer test:
i) The time at which the centre was notified that two questions in the Verbal Skills paper could not be answered correctly as the options on the answer sheet did not reflect those in the question paper
ii) The instructions given to each testin centre
iii) The time at which the testing centre acknowledged receipt of your instructions.
iv) The time or times when pupils started the secondary trasfer test in each testing centre.

By testing centre, I mean any venue at which pupils sat the secondary transfer test.

In addition, I would be most grateful if you could supply any report, analysis or correspondence you have sent to, or received from, GL Assessment in regards of the test errors and their impact, and any remedial measures considered. This would include any statistical analysis of the impact of the errors on test outcomes conducted by GL Assessment. I would also like you to supply any underlying data that was used or made available to GL Assessment in or for their analysis.

I would also be grateful if you could provide minutes or records of all meetings you have held with GL Assessment since 12th September 2019, and any associated correspondence.

Finally, I request the names of any independent bodies contacted in relation to the errors in the secondary transfer test, for example requesting independent verification of any analysis/conclusions etc conducted in-house by GL Assessment staff. This would include any independent bodies who blanked you/GLA or refused to provide such services.

All the best, Neal Skipper.

TBGS, The Buckinghamshire Grammar Schools

Dear Mr Skipper
Thank you for your email. We will respond within 20 working days.
Regards
Mark Sturgeon
Chair
The Buckinghamshire Grammar Schools

show quoted sections

TBGS, The Buckinghamshire Grammar Schools

Dear Mr Skipper

 

Response to Freedom of Information Requests in relation to Bucks 11+

 

Thank you for your email of 21 September 2019 and 5 October 2019. I am
writing to inform you of the extent to which we can comply with your
request.

 

In relation to the Buckinghamshire 11+ test taken in September 2019, you
have requested on 21 September 2019:

 

1.       The time at which each testing centre was notified that two
questions in the Verbal Skills paper could not be answered correctly as
the options on the answer sheet did not reflect those in the question
paper

2.       The instructions given to each testing centre

3.       The time at which each testing centre acknowledged receipt of
your instructions

4.       The time or times when pupils started the secondary transfer test
in each testing centre

5.       Copies of any report, analysis or correspondence (including any
statistical analysis or underlying data used in any such analysis) we have
sent to, or received from, GL Assessment in regards of the test errors and
their impact, and any remedial measures considered

6.       Minutes or records of all meetings you have held with GL
Assessment since 12th September 2019, and any associated correspondence

7.       Names of any independent bodies contacted in relation to the
errors in the secondary transfer test.

 

On 5 October 2019, you requested:

 

8.       The total time each candidate devoted to answering the remaining
VR questions, which are to contribute towards their final STT score

9.       The analysis that allowed the independent statistician to
conclude that the post hoc reengineering of the VR section of the transfer
test is fair for all candidates

 

You are making this request pursuant to the Freedom of Information Act
2000 (FoIA). The FoIA has certain exemptions to the right to have
information disclosed.  One of the absolute exemptions is provided for in
section 41 of the FoIA ‚Äď Confidentiality. Another relevant exemption is
section 43 ‚Äď Commercial interests.

In response to requests 1 and 2 above, we can provide the following
information: LA primary schools were notified at 9.29. The email was
recalled at 10.12 as it had been sent from a BCC individual email and
resent at 10.12. Grammar schools and out of county schools testing
individual EQA children were notified at 9.30. Partner schools and lead
invigilators at Bucks New University (used for some out of county testing)
were notified at 9.46. Test centres were instructed: Please reassure the
children that answers to these questions will not be taken into account
and children should ignore these questions and move on to the next one.

 

In relation to requests 3, 4 and 8 above, we do not know or hold that
information so we are unable to assist in this regard.

 

In relation to requests 5, 6 and 7, we cannot provide you with the above
requested information as this information has been supplied to us by our
supplier, GL Assessment Limited, subject to conditions of confidentiality.
In addition, elements of the information you have requested would also be
covered by the commercial interests exemption.

 

Section 41 - Confidentiality

 

If information is provided to a public body in confidence, and a breach of
that confidence would result in action being taken against the public
body, then the public body does not have to disclose that information. As
stated above, the information you have requested above has been provided
to us by GL Assessment Limited subject to conditions of confidentiality, a
breach of which would result in GL Assessment Limited taking action
against us.

 

Section 43 ‚Äď Commercial Interests

 

Section 43 provides an exemption to the right to have information
disclosed where:

‚Äʬ†the information is a trade secret; or
‚Äʬ†the release of information is likely to prejudice the commercial
interests of any person.

 

The statistical information you have requested relates to or discusses GL
Assessment’s standardisation methodology. This standardisation methodology
is a trade secret of GL Assessment’s which it keeps from its competitors
in the marketplace.  Releasing information would also prejudice the
commercial interest of GL Assessment.

 

Regarding request 9, we refer you to our explanation in the Frequently
Asked Questions page of our website, which can be found here -
[1]https://www.thebucksgrammarschools.org/f...

 

Yours sincerely

Mark Sturgeon
Chair
The Buckinghamshire Grammar Schools

show quoted sections

Dear The Buckinghamshire Grammar Schools,

Please pass this on to the person who conducts Freedom of Information reviews.

I am writing to request an internal review of The Buckinghamshire Grammar Schools's handling of my FOI request 'Errors in the 2019 Secondary Transfer Test', and in particular to your response to questions 5 and 9 (your numbering).

‚Äú5. I would be most grateful if you could supply any report, analysis or correspondence you have sent to, or received from, GL Assessment in regards of the test errors and their impact, and any remedial measures considered. This would include any statistical analysis of the impact of the errors on test outcomes conducted by GL Assessment. I would also like you to supply any underlying data that was used or made available to GL Assessment in or for their analysis‚ÄĚ.

You refused this information on the grounds of: 41 Confidentiality, and 43 Commercial Interests of GL Assessment. I am not completely clear as to which exemption you think applies to which piece of information requested, and I therefore think it might be useful if I clarify the motivation and details of my request. I believe that 41 does not apply, and 43 is outweighed by the public interest and need for transparency.

I am not interested in the method GL Assessment use to standardize the STTS scores, but rather to understand quantitatively the impact of the errors on the reliability and validity of the standardized test scores. This information is not available from the communications sent to parents, or the FAQs on your website. To highlight the public interest issues, I sent the information you have made available to an external expert. I will forward their comments to you separately.

You state that the Verbal Reasoning and English sections score well above 0.80 reliability. There are a number of different measures of reliability, and I would be very grateful if you could explain which one you are quoting. I would also be very pleased if you could provide the actual values for each section of the tests (not just VR and English), the method used to calculate these reliabilities, the factors included, and any assumptions/approximations involved. For example, is this figure just a measure of internal consistency, or does it include an estimate of occasion (test-retest) error? I would also like to request the reliability close to the cut-off score, in the approximate range STTS 115 ‚Äď 125. If possible, the underlying data on which these calculations are based.

GL Assessment have previously published a technical briefing on their VR tests (https://www.gl-assessment.ie/sites/gl/fi...), so I assume they must have investigated these issues in the current context. This GL Assessment report states that ‚ÄúThere are two reasons why you can be reassured that the tests are reliable. First, because of the controlled test delivery and automatic computer marking, there is very little room for the test to be administered in different ways and thus to obtain different results.‚ÄĚ From your response to questions 1-3, it is clear that the test delivery was not controlled, and that the test was administered in different ways in different schools. I would therefore be grateful if you could explain in detail how you have managed to conclude ‚Äúthat the overall reliability of the test has not been compromised‚ÄĚ.

‚Äú9. The analysis that allowed the independent statistician to conclude that the post hoc reengineering of the VR section of the transfer test is fair for all candidates‚ÄĚ

You have confirmed that you do not know how much time each child wasted answering the questions that were subsequently discarded. You have also confirmed that test centres were instructed: ‚ÄúPlease reassure the children that answers to these questions [Q10 and Q13] will not be taken into account and children should ignore these questions and move on to the next one‚ÄĚ. You did not tell children that the last six questions would also not be taken into account.

To take an example, by following your instructions, one child might have provided correct answers to VR questions 1, 3, 5, 11, 14, 15, 16, 17, 18 and 19. A second child of identical age might have provided correct answers to VR questions 1, 2, 3, 5 and 11. After you have applied your solution to the test, the verbal reasoning score/ability of these two children is (strongly) inverted. Could you please, therefore, explain how your independent statistician is able to conclude that the outcome of the test, without the last six questions, is still fair for all children. This information is not provided in your FAQs.

Very many thanks again,

Neal Skipper

TBGS, The Buckinghamshire Grammar Schools

Dear Mr Skipper
Thank you for your email. We will respond within 20 working days.
Kind regards
Mark Sturgeon
Chair
The Buckinghamshire Grammar Schools

show quoted sections

Dear TBGS,

i'd be most grateful if you could update me with progress on your response.

Verey many thanks,

Neal Skipper

Dear Mr Sturgeon,
Many thanks for your email of 16th December and your response to my request for an internal review, which I have pasted below. For the record, it is my view that there were no new requests in my message to you of 28th October 2019: you have the information, and it was therefore covered by my original message.

With regards to your point 2 below, you state on your website that the Verbal Reasoning and English sections score well above 0.80 reliability. I assumed that this was an attempt on your part to inform the public. I therefore asked you explicitly to explain which, of the many possible, measures of reliability GLA had used. You're now claiming that GLA would successfully take you to Court if you provided this necessary clarification...

I will be lodging a complaint with the Information Commissioner’s Office.

Many thanks again,

Neal Skipper

-------------------------------------
16 December 2019 15:23
Dear Mr Skipper

Thank you for your email of 28 October 2019 and further details provided on 2 November.

You have requested an internal review of our handling of your requests made pursuant to the Freedom of Information Act 2000 (FoIA). You also appear to have made new FoIA requests which we shall respond to within this letter.

Findings of Internal Review

Your request for an internal review is in respect of our response to questions 5 and 9 (as numbered in our email of 17 October). You have requested clarification on the exemption that we consider applies to our refusal to release that requested information.

Following the review it remains our position that the exemption of confidentiality by virtue of section 41 of the FoIA applies to the information you have requested.

Section 41 of the FoIA states that information is exempt information for the purposes of required disclosure if:

a) it was obtained by the public authority from any other person (including another public authority); and
b) the disclosure of the information to the public by the public authority holding it would constitute a breach of confidence actionable by that or any other person.

In order for Section 41 to be engaged, the following criteria must be fulfilled:

a) the authority must have obtained the information from another person;
b) its disclosure must constitute a breach of confidence;
c) a legal person must be able to bring an action for the breach of confidence to court; and
d) that court action must be likely to succeed.

GL Assessment has provided us with details of the statistician’s report commissioned by them, which included information in relation to the statistician’s analysis, methodology, conclusions and consideration of the impact of the errors on the candidates. This information, along with other information provided by GL Assessment on this matter and their correspondence with us, has been disclosed to us subject to binding written terms of confidentiality.

Any unpermitted disclosure of this information in contravention of these confidentiality obligations would constitute a breach of confidence. As a result, GL Assessment would have a contractual right to bring a claim against us for breach of confidence.

Section 41 is an absolute exemption, so there is no public interest test to be carried out under FoIA.
However, we have considered whether we would have a public interest defence to a claim made by GL Assessment if we were to disclose some or all of the information you have requested.

We appreciate your comments on why you think it would be in the public interest for this information to be released. However, we have arrived at the conclusion that the public interest defence is likely to fail (and therefore GL Assessment’s court action is likely to succeed) because of the wider public interest in preserving the principle of confidentiality, namely disclosure would undermine the relationship of trust between us and GL Assessment. GL Assessment is one of a very limited number of 11+ test providers and therefore this could adversely affect our ability to select this provider in the future. This has an adverse effect of limiting our choice of providers which is not in the public interest as we would have a significantly smaller selection of providers from which to judge cost, value for money and credibility.

New FoIA Requests

Turning to your additional requests for information, we believe you are making a FoIA request for:
1. information/explanation of the impact of the errors on the reliability and validity of the standardised test scores.
2. information/explanation of the reliability measurement(s) used.
3. the actual values for each section of the tests (not just VR and English), the method used to calculate these reliabilities, the factors included, and any assumptions/approximations involved. For example, is this figure just a measure of internal consistency, or does it include an estimate of occasion (test-retest) error?
4. the reliability close to the cut-off score, in the approximate range STTS 115 ‚Äď 125 and the underlying data on which these calculations are based.
5. explanation of how the independent statistician was able to conclude that the outcome of the test, without the last six questions, was still fair for all children.
We are unable to provide you with the above requested information as this information has been supplied to us by GL Assessment Limited, subject to the aforementioned conditions of confidentiality (section 41 of the FoIA exemption). We believe that our analysis given above as to why the section 41 exemption applies equally to your new requests.
You have a right to lodge a complaint with the UK supervisory body, the Information Commissioner’s Office (the ICO) here:- https://ico.org.uk/make-a-complaint/

Yours sincerely

Mark Sturgeon
Chair
The Buckinghamshire Grammar Schools