Algorithm for Computing Postcode Centroids in Code-Point Open
Dear Ordnance Survey Limited,
My understanding is that the Postcode "Centroids" in your Code-Point Open Product are computed as follows (at least where the PQI is 10): The AddressBase coordinates of each address within the Postcode unit are averaged to obtain an average position (Easting and Northing) for the unit. The coordinates assigned to the centroid are then those of the address in the unit that is closest to this average position.
However, in my use of the Code-Point Open dataset, I have encountered a number of anomalies, where it appears that this algorithm has either not been used, or there must be errors in the input data. I would like to better understand how these errors arise, and how common they might be in Code-Point Open.
To that end, could you please confirm if the basic algorithm I have stated above is correct, and also provide the precise technical method used in Code-Point Open? For the latter, I would like enough detail that it would be possible to precisely reproduce the calculations for each unit, given the necessary input data/coordinates.
Secondly, for each of the following postcode units, please provide a list of all AddressBase entries with that postcode (including their addresses and coordinates) together with any other data that is required for the computation of the Code-Point Open centroid for that unit. (I wish to be able to replicate your computations myself for these units in order to understand how/why the errors are occurring.)
* NE8 1AF
* E2 9RF
* NR20 5NF
* SG18 8PS
Yours faithfully,
Robert Whittaker
Dear Mr Whittaker
Ref No: FOI-21-1014 – Request for information - Algorithm for Computing Postcode Centroids in Code-Point Open
Thank you for your email of 18/06/2021.
As your request was received by Ordnance Survey on 18/06/2021 we will provide you with a response within 20 working days, as required by the Freedom of Information Act, therefore by 16/07/2021.
In exceptional circumstances it may be necessary for Ordnance Survey to extend this date to consider a complex public interest argument if engaging a qualified exemption. In such cases we will notify you of the revised date by which we would expect to provide a response.
Should you need to contact Ordnance Survey regarding your request at any time, please quote the allocated reference number in your correspondence.
Thank you for your enquiry.
Kind regards,
Jo Saunders
Legal & Contracts Administrator, Legal Services
Dear Mr Whittaker,
Thank you for your email of 18 June 2021, requesting information from
Ordnance Survey in accordance with the Freedom of Information Act (FOIA)
2000, as set out in the extract below:
“My understanding is that the Postcode "Centroids" in your Code-Point Open
Product are computed as follows (at least where the PQI is 10): The
AddressBase coordinates of each address within the Postcode unit are
averaged to obtain an average position (Easting and Northing) for the
unit. The coordinates assigned to the centroid are then those of the
address in the unit that is closest to this average position.
However, in my use of the Code-Point Open dataset, I have encountered a
number of anomalies, where it appears that this algorithm has either not
been used, or there must be errors in the input data. I would like to
better understand how these errors arise, and how common they might be in
Code-Point Open.
To that end, could you please confirm if the basic algorithm I have stated
above is correct, and also provide the precise technical method used in
Code-Point Open? For the latter, I would like enough detail that it would
be possible to precisely reproduce the calculations for each unit, given
the necessary input data/coordinates.
Secondly, for each of the following postcode units, please provide a list
of all AddressBase entries with that postcode (including their addresses
and coordinates) together with any other data that is required for the
computation of the Code-Point Open centroid for that unit. (I wish to be
able to replicate your computations myself for these units in order to
understand how/why the errors are occurring.)
* NE8 1AF
* E2 9RF
* NR20 5NF
* SG18 8PS”
Firstly, I confirm that the basic algorithm you state in the first
paragraph of your request is a correct summary of the process used to
derive the location of the postcode centroid in Code-Point Open.
I confirm that Ordnance Survey does hold the information you have
requested; however, we consider the information to be exempt from
disclosure under sections 43 and 21 of the FOIA. Taking each of your
questions in turn, this is explained further below: -
1. “provide the precise technical method used in Code-Point Open?”
Section 43(2) Prejudice to commercial Interests
You have requested the precise technical method used in Code-Point Open.
I confirm that Ordnance Survey does hold the information you have
requested; however, we are unable to comply with your request and provide
a copy of the information, as we consider the information to be exempt
from disclosure under section 43 of the FOIA.
Under section 43(2) of the FOIA, information is exempt if its disclosure
would, or would be likely to, prejudice the commercial interests of any
person. In this case, we consider that disclosure of the precise
technical method would be likely to prejudice the commercial interests of
Ordnance Survey. The precise algorithm comprises a substantial amount of
code, which has taken several weeks of work to produce. As you might
expect, this code comprises commercially valuable information; note that
parts or all of the code are likely to be useful in future commercial
applications.
This is a qualified exemption, and we are required to consider the public
interest.
Public Interest Test
Ordnance Survey recognises the need for transparency. However, this must
be balanced against the public interest in allowing the organisation to
protect its commercial information and not place itself at a disadvantage
in the competitive market in which it operates. Section 43(2) is a
prejudice based exemption, and there is a public interest inherent in
avoiding the harm specified. Ordnance Survey considers that the prejudice
would be likely to occur and would involve a material adverse effect on
Ordnance Survey’s revenues. We are not aware of a significant wider public
interest in disclosing the information.
Having considered the above, we are satisfied that the public interest in
maintaining the exemption outweighs the public interest in disclosure.
2. “for each of the following postcode units, please provide a list of all
AddressBase entries with that postcode (including their addresses and
coordinates) together with any other data that is required for the
computation of the Code-Point Open centroid for that unit.”
Section 21 (Information accessible by other means)
In relation to the second part of your request you have asked us to
provide a list of all AddressBase entries for the postcode units provided
together with any other data that is required for the computation of the
Code-Point Open centroid for that unit. I confirm that Ordnance Survey
does hold the list of AddressBase entries you have requested; however, we
consider the information to be exempt from disclosure under section 21 of
the FOIA, as it is reasonably accessible to you by other means. The
information is available from Ordnance Survey or one of our licenced
partners in our AddressBase products on payment of the appropriate fee.
Section 21(2)(a) states that information may be regarded as reasonably
accessible to the applicant even though it is accessible only on payment.
Section 21 is an absolute exemption and therefore not subject to the
public interest test.
Internal Review
Your enquiry has been processed according to the Freedom of Information
Act (FOIA) 2000. If you are unhappy with our response, you may request
an internal review with our Internal Review Officer by contacting them,
within 40 working days of receipt of our final response to your FOI
request, as follows:
Internal Review Officer
Customer Service Centre
Ordnance Survey
Adanac Drive
Southampton
SO16 0AS
E-mail: [1][Ordnance Survey Limited request email]
Please include the reference number above. You may request an internal
review where you believe Ordnance Survey has:
o Failed to respond to your request within the time limits (normally 20
working days)
o Failed to tell you whether or not we hold the information
o Failed to provide the information you have requested
o Failed to explain the reasons for refusing a request
o Failed to correctly apply an exemption or exception
The Internal Review Officer will not have been involved in the
original decision. They will conduct an independent internal review and
will inform you of the outcome of the review normally within 20 working
days, but exceptionally within 40 working days, in line with the
Information Commissioner’s guidance.
The Internal Review Officer will either: uphold the original decision,
provide an additional explanation of the exemption/s applied or release
further information, if it is considered appropriate to do so.
Appeal to Information Commissioner's Office (ICO)
If, following the outcome of the internal review you remain unhappy with
our response, you may raise an appeal, within three months of receiving
our response, with the Information Commissioner’s Office.
Further information can be found on the ICO website (ico.org.uk) or you
may wish to call the ICO helpline on 0303 123 1113.
Thank you for your enquiry.
Yours sincerely,
Elizabeth Cole
Freedom of Information Officer, Legal Services
Adanac Drive, Southampton, United Kingdom, SO16 0AS
T: +44 (0)3456 05 05 05
[2]www.os.uk | [3][Ordnance Survey Limited request email]
Follow us: [4]Twitter | [5]LinkedIn | [6]YouTube | [7]Instagram |
[8]Facebook
Our values are: Think Customer | Thrive Together | Stay Ahead | Seize the
Moment
This email and any attachments are intended only for the intended
recipient and may contain sensitive information. If you are not the
intended recipient, please immediately delete this email and inform the
sender.
OS email communications may be monitored to ensure the secure and
effective operation of our systems and for other lawful purposes. Subject
to contract: No rights are to be derived from any proposal contained in
this email until a written agreement containing all necessary terms is
executed between the relevant parties.
Thank you for your cooperation.
Ordnance Survey Limited (Company Registration number 09121572)
Registered Office: Explorer House
Adanac Drive
Southampton SO16 0AS
Tel: 03456 050505
[9]http://www.os.uk
References
Visible links
1. mailto:[Ordnance Survey Limited request email]
2. http://www.os.uk/
3. mailto:[Ordnance Survey Limited request email]
4. https://twitter.com/OrdnanceSurvey
5. https://www.linkedin.com/company/ordnanc...
6. https://www.youtube.com/user/OSMapping?b...
7. https://www.instagram.com/ordnancesurvey...
8. https://www.facebook.com/osmapping/
9. http://www.os.uk/
Dear Ordnance Survey Limited,
Please pass this on to the person who conducts Freedom of Information reviews.
I am writing to request an internal review of Ordnance Survey Limited's handling of my FOI request 'Algorithm for Computing Postcode Centroids in Code-Point Open'. A full history of my FOI request and all correspondence is available on the Internet at this address: https://www.whatdotheyknow.com/request/a...
First, please review your application of Section 43(2) to my request for the "precise technical method used in Code-Point Open". I can confirm that I am only requesting details of the algorithm you apply, and not the code that implements it. The prejudice you outline appears to relate only to the code. I see no reason why the details of the algorithm would create any harm to your interests. It would be quite possible for you to provide the information about the algorithm without releasing that actual code. (Moreover allowing others to review the algorithm, may help you discover a flaw in it, thus allowing you to improve the data you provide, and further your commercial interests.)
In addition to the general public interest in transparency, there is a strong public interest in understanding where errors may arise in the open data that you are publishing, particularly as the release of this open data is funded by public money. The summary of the algorithm I presented (which you agreed was accurate) does not explain the anomalies I have observed. In these cases, based on the data I can reasonably access, the centroid in Code-Point Open does not correspond to the location of an address within the postcode unit. If there is issue here, you are arguably failing to deliver what you have agreed to provide in Code-Point Open in return for the public funding. There is a strong public interest in understand what is going on here, and fixing any errors.
Secondly, please review your application of Section 21 to my request for the details of AddressBase entries for the specific postcodes. While this information would be accessible to me via AddressBase, this comes at a cost. You fail to specify what this cost would be in your response, but I understand it would be a significant amount of money, and I would have to pay for significantly more data than I am actually requesting. Given the negligible proportion of the AddressBase data I am requesting, I contend that it would not be reasonable for me to pay such fees, and hence the information is not reasonably accessible to me via those other means. If you wish to maintain reliance on this exemption, please outline precisely how the information could be obtained, and what the total cost to me would be in doing so.
Finally, under your general duty to provide support and assistance to applicants, even if you are not able to provide the precise information requested, I would argue that you could at least explain how the observed anomalies in Code-Point Open have arisen -- as that is what I am ultimately interested in. In particular, you could confirm whether or not the centroids in Code-Point Open for those four postcode units do correspond a current AddressBase entry in that unit, and if not where the centroid coordinates in these cases have come from.
Yours faithfully,
Robert Whittaker
Dear Mr Whittaker,
Original Ref No: FOI-21-1014 (Algorithm for Computing Postcode Centroids
in Code-Point Open)
Thank you for your email of 14 July 2021, requesting a formal review of
the decision made regarding FOI-21-1014. As requested, I have forwarded
this matter to our Internal Review Officer for his attention.
The Internal Review Officer will conduct an independent internal review
and will inform you of the outcome of the review normally within 20
working days (11 August 2021), but exceptionally within 40 working days,
in line with the Information Commissioner’s guidance.
The Internal Review Officer will either: uphold the original decision,
provide an additional explanation of the exemption/s applied or release
further information, if it is considered appropriate to do so.
Appeal to Information Commissioner's Office (ICO)
If, following the outcome of the internal review you remain unhappy with
our response, you may raise an appeal with the Information Commissioner’s
Office.
Further information can be found on the ICO website (ico.org.uk) or you
may wish to call the ICO helpline on 0303 123 1113.
Regards,
Elizabeth Cole
Freedom of Information Officer, Legal Services
Adanac Drive, Southampton, United Kingdom, SO16 0AS
T: +44 (0)3456 05 05 05
[1]www.os.uk | [2][Ordnance Survey Limited request email]
Follow us: [3]Twitter | [4]LinkedIn | [5]YouTube | [6]Instagram |
[7]Facebook
Our values are: Think Customer | Thrive Together | Stay Ahead | Seize the
Moment
This email and any attachments are intended only for the intended
recipient and may contain sensitive information. If you are not the
intended recipient, please immediately delete this email and inform the
sender.
OS email communications may be monitored to ensure the secure and
effective operation of our systems and for other lawful purposes. Subject
to contract: No rights are to be derived from any proposal contained in
this email until a written agreement containing all necessary terms is
executed between the relevant parties.
Thank you for your cooperation.
Ordnance Survey Limited (Company Registration number 09121572)
Registered Office: Explorer House
Adanac Drive
Southampton SO16 0AS
Tel: 03456 050505
[8]http://www.os.uk
References
Visible links
1. http://www.os.uk/
2. mailto:[Ordnance Survey Limited request email]
3. https://twitter.com/OrdnanceSurvey
4. https://www.linkedin.com/company/ordnanc...
5. https://www.youtube.com/user/OSMapping?b...
6. https://www.instagram.com/ordnancesurvey...
7. https://www.facebook.com/osmapping/
8. http://www.os.uk/
Dear Mr Whittaker,
FOI 211014 – Request for Internal Review
Thank you for your email of 14 July requesting an Internal Review of
Ordnance Survey’s response to your Freedom of Information request FOI
211014.
I set out below your original request, Ordnance Survey’s response to your
original request, your request for an Internal Review, and finally my
response following this Internal Review.
Original request 18 June 2021
Your original request is set out below:
Dear Ordnance Survey Limited,
My understanding is that the Postcode "Centroids" in your Code-Point Open
Product are computed as follows (at least where the PQI is 10): The
AddressBase coordinates of each address within the Postcode unit are
averaged to obtain an average position (Easting and Northing) for the
unit. The coordinates assigned to the centroid are then those of the
address in the unit that is closest to this average position.
However, in my use of the Code-Point Open dataset, I have encountered a
number of anomalies, where it appears that this algorithm has either not
been used, or there must be errors in the input data. I would like to
better understand how these errors arise, and how common they might be in
Code-Point Open.
To that end, could you please confirm if the basic algorithm I have stated
above is correct, and also provide the precise technical method used in
Code-Point Open? For the latter, I would like enough detail that it would
be possible to precisely reproduce the calculations for each unit, given
the necessary input data/coordinates.
Secondly, for each of the following postcode units, please provide a list
of all AddressBase entries with that postcode (including their addresses
and coordinates) together with any other data that is required for the
computation of the Code-Point Open centroid for that unit. (I wish to be
able to replicate your computations myself for these units in order to
understand how/why the errors are occurring.)
* NE8 1AF
* E2 9RF
* NR20 5NF
* SG18 8PS
Yours faithfully,
Robert Whittaker
Our response 14 July 2021
Ordnance Survey responded to your FOI request 21104, as set out below.
Firstly, I confirm that the basic algorithm you state in the first
paragraph of your request is a correct summary of the process used to
derive the location of the postcode centroid in Code-Point Open.
I confirm that Ordnance Survey does hold the information you have
requested; however, we consider the information to be exempt from
disclosure under sections 43 and 21 of the FOIA. Taking each of your
questions in turn, this is explained further below: -
1. “provide the precise technical method used in Code-Point Open?”
Section 43(2) Prejudice to commercial Interests
You have requested the precise technical method used in Code-Point Open. I
confirm that Ordnance Survey does hold the information you have requested;
however, we are unable to comply with your request and provide a copy of
the information, as we consider the information to be exempt from
disclosure under section 43 of the FOIA.
Under section 43(2) of the FOIA, information is exempt if its disclosure
would, or would be likely to, prejudice the commercial interests of any
person. In this case, we consider that disclosure of the precise technical
method would be likely to prejudice the commercial interests of Ordnance
Survey. The precise algorithm comprises a substantial amount of code,
which has taken several weeks of work to produce. As you might expect,
this code comprises commercially valuable information; note that parts or
all of the code are likely to be useful in future commercial applications.
This is a qualified exemption, and we are required to consider the public
interest.
Public Interest Test
Ordnance Survey recognises the need for transparency. However, this must
be balanced against the public interest in allowing the organisation to
protect its commercial information and not place itself at a disadvantage
in the competitive market in which it operates. Section 43(2) is a
prejudice based exemption, and there is a public interest inherent in
avoiding the harm specified. Ordnance Survey considers that the prejudice
would be likely to occur and would involve a material adverse effect on
Ordnance Survey’s revenues. We are not aware of a significant wider public
interest in disclosing the information.
Having considered the above, we are satisfied that the public interest in
maintaining the exemption outweighs the public interest in disclosure.
2. “for each of the following postcode units, please provide a list of all
AddressBase entries with that postcode (including their addresses and
coordinates) together with any other data that is required for the
computation of the Code-Point Open centroid for that unit.”
Section 21 (Information accessible by other means)
In relation to the second part of your request you have asked us to
provide a list of all AddressBase entries for the postcode units provided
together with any other data that is required for the computation of the
Code-Point Open centroid for that unit. I confirm that Ordnance Survey
does hold the list of AddressBase entries you have requested; however, we
consider the information to be exempt from disclosure under section 21 of
the FOIA, as it is reasonably accessible to you by other means. The
information is available from Ordnance Survey or one of our licenced
partners in our AddressBase products on payment of the appropriate fee.
Section 21(2)(a) states that information may be regarded as reasonably
accessible to the applicant even though it is accessible only on payment.
Section 21 is an absolute exemption and therefore not subject to the
public interest test.
Request for Internal Review, 14 July
Dear Ordnance Survey Limited,
Please pass this on to the person who conducts Freedom of Information
reviews.
I am writing to request an internal review of Ordnance Survey Limited's
handling of my FOI request 'Algorithm for Computing Postcode Centroids in
Code-Point Open'. A full history of my FOI request and all correspondence
is available on the Internet at this address:
[1]https://www.whatdotheyknow.com/request/a...
First, please review your application of Section 43(2) to my request for
the "precise technical method used in Code-Point Open". I can confirm that
I am only requesting details of the algorithm you apply, and not the code
that implements it. The prejudice you outline appears to relate only to
the code. I see no reason why the details of the algorithm would create
any harm to your interests. It would be quite possible for you to provide
the information about the algorithm without releasing that actual code.
(Moreover allowing others to review the algorithm, may help you discover a
flaw in it, thus allowing you to improve the data you provide, and further
your commercial interests.)
In addition to the general public interest in transparency, there is a
strong public interest in understanding where errors may arise in the open
data that you are publishing, particularly as the release of this open
data is funded by public money. The summary of the algorithm I presented
(which you agreed was accurate) does not explain the anomalies I have
observed. In these cases, based on the data I can reasonably access, the
centroid in Code-Point Open does not correspond to the location of an
address within the postcode unit. If there is issue here, you are arguably
failing to deliver what you have agreed to provide in Code-Point Open in
return for the public funding. There is a strong public interest in
understand what is going on here, and fixing any errors.
Secondly, please review your application of Section 21 to my request for
the details of AddressBase entries for the specific postcodes. While this
information would be accessible to me via AddressBase, this comes at a
cost. You fail to specify what this cost would be in your response, but I
understand it would be a significant amount of money, and I would have to
pay for significantly more data than I am actually requesting. Given the
negligible proportion of the AddressBase data I am requesting, I contend
that it would not be reasonable for me to pay such fees, and hence the
information is not reasonably accessible to me via those other means. If
you wish to maintain reliance on this exemption, please outline precisely
how the information could be obtained, and what the total cost to me would
be in doing so.
Finally, under your general duty to provide support and assistance to
applicants, even if you are not able to provide the precise information
requested, I would argue that you could at least explain how the observed
anomalies in Code-Point Open have arisen -- as that is what I am
ultimately interested in. In particular, you could confirm whether or not
the centroids in Code-Point Open for those four postcode units do
correspond a current AddressBase entry in that unit, and if not where the
centroid coordinates in these cases have come from.
Yours faithfully,
Robert Whittaker
Internal Review
I have now completed my Internal Review of our 14 July response to this
FOI request.
First, please review your application of Section 43(2) to my request for
the "precise technical method used in Code-Point Open".
DECISION: UPHELD
Upon investigation, I have found no documentation within Ordnance Survey
relating to a Code-Point Open centroid algorithm which provides more
detail than the methodology which you correctly summarised in your
original request. The algorithm is, essentially, two straightforward
steps:
o Compute the mean easting and northing values of all AddressBase
records relating to a given postcode.
o Assign the Code-Point (and corresponding Code-Point Open) location to
the nearest AddressBase record.
I note that you have specified that you are not requesting the computer
code.
Secondly, please review your application of Section 21 to my request for
the details of AddressBase entries for the specific postcodes.
DECISION: UPHELD
I concur that the information that you seek is available through
commercial channels, and is therefore exempt under Section 21.
We have numerous partners who can provide you with the AddressBase
information you seek, with details here: [2]Find a Business Partner | List
of Ordnance Survey Partners. I have approached [3]GB Group, one of our
partners, and established that the cost of obtaining AddressBase records
for the addresses relating to the four specified postcodes would cost in
the region of £60. However, it's possible that my response to your final
point below may resolve this requirement.
Finally, under your general duty to provide support and assistance to
applicants, even if you are not able to provide the precise information
requested, I would argue that you could at least explain how the observed
anomalies in Code-Point Open have arisen -- as that is what I am
ultimately interested in. In particular, you could confirm whether or not
the centroids in Code-Point Open for those four postcode units do
correspond a current AddressBase entry in that unit, and if not where the
centroid coordinates in these cases have come from.
After an initial investigation of the four postcodes you identified, we
believe that there are errors in some of the underlying AddressBase
coordinates which may be skewing the Code-Point Open locations for those
postcodes.
This is a matter which is better handled via the OS Data Hub [4]Error
Reporting tool rather than the FOI process. This tool is specifically
designed to handle this type of query; the flagging of potential errors
triggers a process to (a) resolve data quality issues and then (b) contact
the originator of the query with feedback. On this occasion, we will log
the queries on your behalf in the Error Reporting tool, and we will ensure
that you will be given feedback following the conclusion of the
investigation. If you have further queries of a similar nature in future,
I'd invite you to flag them directly to the Error Reporting tool.
Thank you for bringing this issue to our attention.
Appeal to Information Commissioner's Office (ICO)
If, following the outcome of the Internal Review you remain unhappy with
our response, you may raise an appeal, within three months of receiving
our response, with the Information Commissioner’s Office.
Further information can be found on the ICO website (ico.org.uk) under
‘[5]Report a concern’ or you may wish to call the ICO helpline on 0303
123 1113.
Thank you for your enquiry.
Your sincerely,
Mark Stileman
FOI Internal Reviewer
Adanac Drive, Southampton, United Kingdom, SO16 0AS
M: +44 (0)7789 927434
[6]www.os.uk | [7][email address] | [8]linkedin.com/in/markstileman
Our values are: Think Customer | Thrive Together | Stay Ahead | Seize the
Moment
Dear Ordnance Survey Limited,
I am writing again regarding an internal review of Ordnance Survey Limited's handling of my FOI request 'Algorithm for Computing Postcode Centroids in Code-Point Open'. A full history of my FOI request and all correspondence is available on the Internet at this address: https://www.whatdotheyknow.com/request/a... . While you do not need to respond to a second internal review request, I feel it may be more expedient to give you another chance to respond to some clarifications rather than me going directly to a s50 complaint to the Information Commissioner. (But if you do not regard this as a review request, please treat the final paragraph below as a new information request.)
Regarding my request for the precise algorithm, I am afraid that the information you have provided is not sufficient. I require enough information such that if I had a copy of AddressBase (and other other input data required) I would be able to reproduce the calculations of the centroids in Code-Point Open exactly. What you have supplied so far does not do this. In particular:
* You have not explained which AddressBase records are used. I believe that AddressBase also includes historic and street record UPRNs, along with UPRNs that relate to non-addressable objects. I think these mostly have postcodes assigned to them, but they are presumably not used in the centroid calculations. So how are the AddressBase records filtered, and what fields are used for this?
* You have not stated what level of precision you use when calculating the mean easting and northing. Is it done to integer precision only, or do you retain some decimal places also?
* You have not been clear about whether the nearest location that the centroid is snapped to must be one of the locations used in the calculation for that particular unit, or just one with the correct postcode, or just the nearest location of any record.
* You have not explained the method for computing the distance between the mean position and each of the candidate locations. (Perhaps for simplicity you simply use Pythagoras on the coordinate differences.)
* You have not explained what happens if the smallest distance from the mean position is shared by two or more candidate locations. Even with rounding, this will often occur when a unit contains only two locations, and may occur in other cases too. (Presumably there must be a deterministic way of resolving this, to avoid centroids flipping randomly in such cases between different releases of Code-Point Open.)
All of these details could presumably be found by inspecting the code, but it is the *information* rather than the code itself that is subject to FOIA. It is up to you how you provide the information to me.
Secondly, with regard to your explanation of the cause of the errors I have observed, this may explain two of the units -- NE8 1AF and NR20 5NF -- as I think these each only have two addresses in them. But I don't think it could explain the other two. With the algorithm you have presented, a single record with erroneous coordinates could not be responsible for a misplaced centroid where there are at least three records with that postcode. You would need either the majority to be misplaced, or at least one to be massively misplaced in the same direction as another misplaced location.
Looking at the OSN's UPRN Lookup dataset (which includes postcodes for each UPRN), it appears that the postcodes SG18 8PS and SG18 8NE may have been mis-assigned to a number of UPRNs. Royal Mail's Postcode Finder says SG18 8PS contains three addresses: Benson's for Beds, Halfords, and Pets at Home; which I think correspond to the UPRNs 10091572346, 10091572347 and 10091572345. Yet these three UPRNs are assigned to SG18 8NE in OSN's UPRN Lookup. This suggests that the error is not in the location/coordinates, but in the assigned postcodes.
Regarding E2 9RF, Royal Mail lists 11 addresses in that unit, all of which have addresses in the Poyser Street Railway Arches. ONS UPRN Lookup lists 17 UPRNs with that postcode. 15 of these have coordinates at the same point (534891,183008) in the middle of the Railway Arches on Poyser Street. Two are located at nearby distinct points on the Blackwall Trading Estate off Lanrick Road, close to the (incorrect) Code-Point Open centroid. (Presumably these two have either an incorrect postcode or an incorrect location.) Provided 3 or more of the 15 UPRNs located on Poyser Street are used in the calculation, the centroid calculated by the algorithm you have stated would necessarily result in the Poyser Street coordinates (534891,183008) being returned for this postcode unit. (This is a mathematical fact, since the greater number of records there would mean that the average position would be closer to the Poyser Street location than Lanrick Road location.) Since these coordinates are not returned, then either the algorithm you have stated is not what is actually being used, or there greater problems in the source data than "errors in some of the underlying AddressBase coordinates".
Finally, I'm afraid I still do not agree with your application of s21 to the AddressBase records, as I do not regard paying £60 is reasonable for the information requested. Moreover, since you have not said which AddressBase records have been used in your calculations, I wouldn't actually know which records I needed to ask for.
To resolve this last issue, could you please disclose the UPRNs of the AddressBase records that have been used in the centroid calculations for each of the four postcodes, together with details of which ones you suspect contain errors, and the nature of each error? (In fact, this would probably give me the information I need, as I could then reproduce you centroid calculations using the coordinates from OS Open UPRN. In which case I would be able to withdraw my request for the AddressBase records themselves.)
Yours faithfully,
Robert Whittaker
Dear Mr Whittaker,
We acknowledge receipt of your email of 3 August 2021, in response to our
Internal Review response of 31 July 2021.
We are currently considering the content of your email and will write to
you again in due course to confirm the appropriate route for Ordnance
Survey to respond to you.
Please note, in the event we do consider all or part of your request to be
valid under the Freedom of Information Act we will respond to you within
the statutory timeframes.
Should you need to contact us in the meantime, please quote the original
allocated reference number above.
Kind regards,
Elizabeth Cole
Freedom of Information Officer, Legal Services
Adanac Drive, Southampton, United Kingdom, SO16 0AS
T: +44 (0)3456 05 05 05
[1]www.os.uk | [2][Ordnance Survey Limited request email]
Follow us: [3]Twitter | [4]LinkedIn | [5]YouTube | [6]Instagram |
[7]Facebook
Our values are: Think Customer | Thrive Together | Stay Ahead | Seize the
Moment
This email and any attachments are intended only for the intended
recipient and may contain sensitive information. If you are not the
intended recipient, please immediately delete this email and inform the
sender.
OS email communications may be monitored to ensure the secure and
effective operation of our systems and for other lawful purposes. Subject
to contract: No rights are to be derived from any proposal contained in
this email until a written agreement containing all necessary terms is
executed between the relevant parties.
Thank you for your cooperation.
Ordnance Survey Limited (Company Registration number 09121572)
Registered Office: Explorer House
Adanac Drive
Southampton SO16 0AS
Tel: 03456 050505
[8]http://www.os.uk
References
Visible links
1. http://www.os.uk/
2. mailto:[Ordnance Survey Limited request email]
3. https://twitter.com/OrdnanceSurvey
4. https://www.linkedin.com/company/ordnanc...
5. https://www.youtube.com/user/OSMapping?b...
6. https://www.instagram.com/ordnancesurvey...
7. https://www.facebook.com/osmapping/
8. http://www.os.uk/
Dear Mr Whittaker,
Thanks for your response. I am happy to address the clarifications you
raise. Please bear with us while key colleagues are returning from leave,
and I will respond to you as soon as I can.
Regards,
Mark
Mark Stileman
FOI Internal Reviewer
Adanac Drive, Southampton, United Kingdom, SO16 0AS
M: +44 (0)7789 927434
[1]www.os.uk | [2][email address] | [3]linkedin.com/in/markstileman
Dear Mr Whittaker,
Thank you for your patience while we’ve been investigating the
clarification questions you raised, which I will address in turn.
Further details of the algorithm for generating Code-Point Open centroids
I can clarify that not every AddressBase record is necessarily used in
calculating the position of a Code-Point Open centroid, as I indicated in
the internal review response, for which I apologise. Specifically, I can
now confirm the following.
Filtering of AddressBase records
AddressBase records are filtered and used in the postcode centroid
calculation when:
· they have been matched to a Royal Mail Postcode Address File (PAF)
address
· they have a positional accuracy quality value of 1 or 2
Level of precision
The coordinates used for centroid calculations have centimetre-level
precision. Once the centroid coordinates have been calculated and snapped
to the nearest delivery point they are rounded to metre-level precision.
Address eligibility to be designated as a postcode centroid
The location of the centroid must be one of the locations used in the
calculation of that unit.
Method for computing the distance between the mean position and each of
the candidate locations
Coordinate differences are calculated using Pythagoras' Theorem.
Process for dealing with equidistant candidate locations
If there are two candidate addresses that are the same distance away from
the property, we will arbitrarily select whichever one is first to be
returned by the database.
Postcode queries
We are now able to provide information about the specific postcodes that
you have queried, as follows:
NR20 5NF and NE8 1AF
For both of these postcodes an error has been identified in the location
of one the constituent addresses. These have both been corrected, and both
postcode centroids will be in their correct position in the November
release of Code-Point Open.
SG18 8PS and SG18 8NE
The address records for these postcodes have now been referred for further
investigation with the relevant local authorities and Royal Mail to
ascertain whether there is an issue with their assigned postcode fields.
We will provide you with further information when we have concluded this
investigation.
E2 9RF
In calculating this postcode centroid, 9 of the 10 constituent AddressBase
records are omitted because they do not have a positional accuracy of
sufficient quality. As a result, in this instance only one address record
has been used to calculate the centroid location.
The ONS UPRN look-up includes provisional and historic records as well as
non-postal addresses which have been assigned a postcode.
Access to address data
We are handling this item as a new FOI request, our response to which is
being provided under separate cover.
Regards,
Mark Stileman
FOI Internal Reviewer
Adanac Drive, Southampton, United Kingdom, SO16 0AS
M: +44 (0)7789 927434
[1]www.os.uk | [2][email address] | [3]linkedin.com/in/markstileman
From: Mark Stileman
Sent: 17 August 2021 17:13
To: Robert Whittaker <[FOI #766145 email]>
Cc: FOI <[email address]>
Subject: RE: Internal review of Freedom of Information request - Algorithm
for Computing Postcode Centroids in Code-Point
Dear Mr Whittaker,
Thanks for your response. I am happy to address the clarifications you
raise. Please bear with us while key colleagues are returning from leave,
and I will respond to you as soon as I can.
Regards,
Mark
Mark Stileman
FOI Internal Reviewer
Adanac Drive, Southampton, United Kingdom, SO16 0AS
M: +44 (0)7789 927434
[4]www.os.uk | [5][email address] | [6]linkedin.com/in/markstileman
Dear Mr Whittaker,
Thank you for your email of 3 August 2021, in response to our Internal
Review response of 31 July 2021, relating to FOI-21-1014.
Mark Stileman, our Internal Review officer, has responded to you earlier
today in relation to the majority of the points you make in your email.
In relation to the final two paragraphs of your email, set out below, we
have treated this as a new request for information under the Freedom of
Information Act 2000 (FOIA).
“Finally, I'm afraid I still do not agree with your application of s21 to
the AddressBase records, as I do not regard paying £60 is reasonable for
the information requested. Moreover, since you have not said which
AddressBase records have been used in your calculations, I wouldn't
actually know which records I needed to ask for.
To resolve this last issue, could you please disclose the UPRNs of the
AddressBase records that have been used in the centroid calculations for
each of the four postcodes, together with details of which ones you
suspect contain errors, and the nature of each error? (In fact, this would
probably give me the information I need, as I could then reproduce you
centroid calculations using the coordinates from OS Open UPRN. In which
case I would be able to withdraw my request for the AddressBase records
themselves.)”
Our understanding is that the four postcodes referred to in the extract
above are the same four as were referred to in your previous FOI Request
(FOI-21-1014), namely: NE8 1AF, E2 9RF, NR20 5NF and SG18 8PS.
I confirm that Ordnance Survey does hold the information you have
requested. We note your point above that, since OS has not stated which
AddressBase records have been used in our calculations of the Code-Point
Open centroids, you would not know which records from AddressBase you
require. However, further to the information that has been provided in
Mark Stileman’s email of earlier today, in terms of the fact that OS only
uses those records which have a positional accuracy quality value of 1 or
2 (and noting that such quality value is included in our AddressBase
products), we consider that we would be able to rely on section 21
(information accessible by other means). We also consider that we would
be able to invoke section 43 (prejudice to commercial interests), subject
to the results of a public interest test.
Having said that, we acknowledge that our original response to
FOI-21-1-14, and our response to your request for an Internal Review, did
not provide you with the detailed information that you were seeking
regarding the algorithm, and we appreciate the patience you have shown to
date on this point. As such, on this occasion, we are happy to provide
you with the requested UPRNs, which are as follows:
UPRN Postcode
6352809 E2 9RF
100000069795 NE8 1AF
100000026370 NE8 1AF
100000026370 NR20 5NF
10011968997 NR20 5NF
100081310322 SG18 8PS
10002900239 SG18 8PS
100081252884 SG18 8PS
Internal Review
Your enquiry has been processed according to the Freedom of Information
Act (FOIA) 2000. If you are unhappy with our response, you may request
an internal review with our Internal Review Officer by contacting them,
within 40 working days of receipt of our final response to your FOI
request, as follows:
Internal Review Officer
Customer Service Centre
Ordnance Survey
Adanac Drive
Southampton
SO16 0AS
E-mail: [1][Ordnance Survey Limited request email]
Please include the reference number above. You may request an internal
review where you believe Ordnance Survey has:
o Failed to respond to your request within the time limits (normally 20
working days)
o Failed to tell you whether or not we hold the information
o Failed to provide the information you have requested
o Failed to explain the reasons for refusing a request
o Failed to correctly apply an exemption or exception
The Internal Review Officer will not have been involved in the
original decision. They will conduct an independent internal review and
will inform you of the outcome of the review normally within 20 working
days, but exceptionally within 40 working days, in line with the
Information Commissioner’s guidance.
The Internal Review Officer will either: uphold the original decision,
provide an additional explanation of the exemption/s applied or release
further information, if it is considered appropriate to do so.
Appeal to Information Commissioner's Office (ICO)
If, following the outcome of the internal review you remain unhappy with
our response, you may raise an appeal, within three months of receiving
our response, with the Information Commissioner’s Office.
Further information can be found on the ICO website (ico.org.uk) or you
may wish to call the ICO helpline on 0303 123 1113.
Thank you for your enquiry.
Yours sincerely,
FOI Officer
Adanac Drive, Southampton, United Kingdom, SO16 0AS
[2]www.os.uk | [3][email address]
Follow us: [4]Twitter | [5]LinkedIn | [6]YouTube | [7]Instagram |
[8]Facebook
Our values are: Think Customer | Thrive Together | Stay Ahead | Seize the
Moment
This email and any attachments are intended only for the intended
recipient and may contain sensitive information. If you are not the
intended recipient, please immediately delete this email and inform the
sender.
OS email communications may be monitored to ensure the secure and
effective operation of our systems and for other lawful purposes. Subject
to contract: No rights are to be derived from any proposal contained in
this email until a written agreement containing all necessary terms is
executed between the relevant parties.
Thank you for your cooperation.
Ordnance Survey Limited (Company Registration number 09121572)
Registered Office: Explorer House
Adanac Drive
Southampton SO16 0AS
Tel: 03456 050505
[9]http://www.os.uk
References
Visible links
1. mailto:[Ordnance Survey Limited request email]
2. https://eur01.safelinks.protection.outlo...
3. mailto:[email address]
4. https://eur01.safelinks.protection.outlo...
5. https://eur01.safelinks.protection.outlo...
6. https://eur01.safelinks.protection.outlo...
7. https://eur01.safelinks.protection.outlo...
8. https://eur01.safelinks.protection.outlo...
9. http://www.os.uk/
Dear Sir/Madam,
Many thanks for providing the additional information about how your centroid-calculating algorithm works. I believe this can now explain the anomalous points I was seeing in Code-Point Open. However, it does reveal that your initial simplistic description of the algorithm was incomplete.
I'm sure you'll forgive me for wondering if there are any other aspects of the algorithm that you have still not disclosed. In particular, the current description you have provided does not appear to say if/how the centroid is calculated when there is a postcode for which there are no AddressBase records that meet both criteria you have given. (I would imagine that in some cases the postcode might be omitted from Code-Point Open, but in others a location based on the lower positional accuracy points in AddressBase may be used.)
So please could you reconsider my original request in the light of this, and disclose the *full* algorithm that you use for computing the centroid locations in Code-Point Open. This could be done by releasing the computer code you use, or by providing a *complete* description of everything that the code does. You should provide enough information that if I had a full copy of AddressBase (and any other necessary source data) I could replicate the calculations *exactly*.
Yours sincerely,
Robert Whittaker
Dear Mr Whittaker
Thank you for your email of 16/09/2021.
As your request was received by Ordnance Survey on 16/09/2021 we will
provide you with a response within 20 working days, as required by the
Freedom of Information Act 2000, therefore by 14/10/2021.
In exceptional circumstances it may be necessary for Ordnance Survey to
extend this date to consider a complex public interest argument if
engaging a qualified exemption. In such cases we will notify you of the
revised date by which we would expect to provide a response.
Should you need to contact Ordnance Survey regarding your request at any
time, please quote the allocated reference number in your correspondence.
Thank you for your enquiry.
Kind regards,
Elizabeth Cole
Freedom of Information Officer, Legal Services
Adanac Drive, Southampton, United Kingdom, SO16 0AS
T: +44 (0)3456 05 05 05
[1]www.os.uk | [2][Ordnance Survey Limited request email]
Follow us: [3]Twitter | [4]LinkedIn | [5]YouTube | [6]Instagram |
[7]Facebook
Our values are: Think Customer | Thrive Together | Stay Ahead | Seize the
Moment
This email and any attachments are intended only for the intended
recipient and may contain sensitive information. If you are not the
intended recipient, please immediately delete this email and inform the
sender.
OS email communications may be monitored to ensure the secure and
effective operation of our systems and for other lawful purposes. Subject
to contract: No rights are to be derived from any proposal contained in
this email until a written agreement containing all necessary terms is
executed between the relevant parties.
Thank you for your cooperation.
Ordnance Survey Limited (Company Registration number 09121572)
Registered Office: Explorer House
Adanac Drive
Southampton SO16 0AS
Tel: 03456 050505
[8]http://www.os.uk
References
Visible links
1. http://www.os.uk/
2. mailto:[Ordnance Survey Limited request email]
3. https://twitter.com/OrdnanceSurvey
4. https://www.linkedin.com/company/ordnanc...
5. https://www.youtube.com/user/OSMapping?b...
6. https://www.instagram.com/ordnancesurvey...
7. https://www.facebook.com/osmapping/
8. http://www.os.uk/
Dear Mr Whittaker,
Thank you for your email of 16 September 2021, requesting information from
Ordnance Survey in accordance with the Freedom of Information Act (FOIA)
2000.
We have considered your FOI request, which was to: “disclose the *full*
algorithm that you use for computing the centroid locations in Code-Point
Open. This could be done by releasing the computer code you use, or by
providing a *complete* description of everything that the code does. You
should provide enough information that if I had a full copy of AddressBase
(and any other necessary source data) I could replicate the calculations
*exactly*.”
I confirm that Ordnance Survey does hold some of the information that you
have requested, where we consider information to be exempt or not held
this is explained in further detail below:-
Section 43(2) Prejudice to commercial Interests
You have requested the full algorithm that we use for computing the
centroid locations in Code-Point Open, which you state could be provided
by releasing the computer code that we use. I confirm that Ordnance
Survey does hold the information you have requested, in the form of the
computer code; however, we are unable to comply with your request and
provide a copy of the information, as we consider the information to be
exempt from disclosure under section 43 of the FOIA.
Under section 43(2) of the FOIA, information is exempt if its disclosure
would, or would be likely to, prejudice the commercial interests of any
person. In this case, we consider that disclosure of the code would be
likely to prejudice the commercial interests of Ordnance Survey. The code
which computes the centroid locations in Code-Point Open is part of a much
larger set of code which generates not only Code-Point Open but also
Ordnance Survey’s premium (paid for) products, Code Point and Code Point
with Polygons. As you may appreciate, since this code underpins Ordnance
Survey premium products, it has a commercial value. It is not possible to
extract solely the code with computes the centroid locations in Code-Point
Open, as the coding is inextricably tied to coding for the premium
products.
Section 43(2) is a qualified exemption, and we are required to consider
the public interest.
Public Interest Test
Ordnance Survey recognises the need for transparency. However, this must
be balanced against the public interest in allowing the organisation to
protect its commercial information and not place itself at a disadvantage
in the competitive market in which it operates. Section 43(2) is a
prejudice based exemption, and there is a public interest inherent in
avoiding the harm specified. Ordnance Survey considers that the prejudice
would be likely to occur and would involve a material adverse effect on
Ordnance Survey’s revenues. We are not aware of a significant wider public
interest in disclosing the information.
Having considered the above, we are satisfied that the public interest in
maintaining the exemption outweighs the public interest in disclosure.
Advice and assistance
We appreciate that our response to your request FOI 21-1014 was
incomplete. In that response, dated 14 July, we confirmed that the “basic
algorithm you state in the first paragraph of your [original] request is a
correct summary of the process used to derive the location of the postcode
centroid in Code-Point Open.” Having been prompted by your further
questions to further investigate your request, following which you were
provided with additional information, we understand why you are now asking
us to reconsider your original question. Further to this FOI request, we
have looked at your request afresh.
The result of our review is that we have ascertained that, outside of the
code itself, we do not hold the “full algorithm”, which you also describe
as a “*complete* description of everything that the code does” in a
recorded form. This is partly due to the fact that the process is not
solely an Ordnance Survey process, but involves a number of members from
the Gridlink consortium. The Gridlink consortium members comprise Royal
Mail, the Office of National Statistics (ONS), National Records of
Scotland (NRS), Land & Property Services, and Ordnance Survey, each of
whom supplies component data to create the Gridlink dataset, which is a
dataset of UK postcodes, administration areas and health authority codes.
Data from the Gridlink dataset is used in the creation and maintenance of
Code-Point Open.
In the hope that it provides some assistance to you, we summarise below a
number of the steps involved in the creation of postcode unit centroids in
Code-Point Open; this summary has been prepared following research by
colleagues within Ordnance Survey.
1. New postcode units created by Royal Mail have centroid co-ordinates
imputed by ONS; this is a process that is run by the ONS and the
detail of the process/code is not documented at Ordnance Survey.
These imputed co-ordinates are sent to Ordnance Survey. These
centroids are given a positional accuracy of 50 (note that this is a
different positional accuracy scale to the one used in AddressBase).
2. At the same time as step 1, NRS manually verifies postcode unit
centroids in Scotland from the previous month’s run of the Gridlink
dataset; these centroids are provided to Ordnance Survey, and given a
positional accuracy of 20 (again, using the different positional
accuracy scale). Since this process is run by NRS, Ordnance Survey
does not hold details of the process/code.
3. Ordnance Survey calculates postcode centroids using the process
previously disclosed to you, i.e. where there are addresses within the
postcode unit which have a positional accuracy, in AddressBase, of 1
or 2; the postcode unit centroids calculated from these addresses are
given a positional quality indicator of 10 (again, using the different
positional accuracy scale). Where a postcode unit does not contain
any qualifying addresses (i.e. addresses with positional accuracy of 1
or 2 within AddressBase), it is attributed an approximate location by
taking the mean of the qualifying addresses within the postcode
sector. This mean position is then snapped to the nearest qualifying
address; centroids derived in this way are given a positional quality
indicator of 60 (again, using the different positional accuracy
scale).
4. Ordnance Survey loads into the software the postcode centroid
information derived as described in the three paragraphs above. For
each postcode unit, the software selects the centroid with the best
postcode positional quality indicator.
Internal Review
Your enquiry has been processed according to the Freedom of Information
Act (FOIA) 2000. If you are unhappy with our response, you may request
an internal review with our Internal Review Officer by contacting them,
within 40 working days of receipt of our final response to your FOI
request, as follows:
Internal Review Officer
Customer Service Centre
Ordnance Survey
Adanac Drive
Southampton
SO16 0AS
E-mail: [1][Ordnance Survey Limited request email]
Please include the reference number above. You may request an internal
review where you believe Ordnance Survey has:
o Failed to respond to your request within the time limits (normally 20
working days)
o Failed to tell you whether or not we hold the information
o Failed to provide the information you have requested
o Failed to explain the reasons for refusing a request
o Failed to correctly apply an exemption or exception
The Internal Review Officer will not have been involved in the
original decision. They will conduct an independent internal review and
will inform you of the outcome of the review normally within 20 working
days, but exceptionally within 40 working days, in line with the
Information Commissioner’s guidance.
The Internal Review Officer will either: uphold the original decision,
provide an additional explanation of the exemption/s applied or release
further information, if it is considered appropriate to do so.
Appeal to Information Commissioner's Office (ICO)
If, following the outcome of the internal review you remain unhappy with
our response, you may raise an appeal, within three months of receiving
our response, with the Information Commissioner’s Office.
Further information can be found on the ICO website (ico.org.uk) or you
may wish to call the ICO helpline on 0303 123 1113.
Thank you for your enquiry.
Kind regards,
Elizabeth Cole
Freedom of Information Officer, Legal Services
Adanac Drive, Southampton, United Kingdom, SO16 0AS
T: +44 (0)3456 05 05 05
[2]www.os.uk | [3][Ordnance Survey Limited request email]
Follow us: [4]Twitter | [5]LinkedIn | [6]YouTube | [7]Instagram |
[8]Facebook
Our values are: Think Customer | Thrive Together | Stay Ahead | Seize the
Moment
This email and any attachments are intended only for the intended
recipient and may contain sensitive information. If you are not the
intended recipient, please immediately delete this email and inform the
sender.
OS email communications may be monitored to ensure the secure and
effective operation of our systems and for other lawful purposes. Subject
to contract: No rights are to be derived from any proposal contained in
this email until a written agreement containing all necessary terms is
executed between the relevant parties.
Thank you for your cooperation.
Ordnance Survey Limited (Company Registration number 09121572)
Registered Office: Explorer House
Adanac Drive
Southampton SO16 0AS
Tel: 03456 050505
[9]http://www.os.uk
References
Visible links
1. mailto:[Ordnance Survey Limited request email]
2. http://www.os.uk/
3. mailto:[Ordnance Survey Limited request email]
4. https://twitter.com/OrdnanceSurvey
5. https://www.linkedin.com/company/ordnanc...
6. https://www.youtube.com/user/OSMapping?b...
7. https://www.instagram.com/ordnancesurvey...
8. https://www.facebook.com/osmapping/
9. http://www.os.uk/
We work to defend the right to FOI for everyone
Help us protect your right to hold public authorities to account. Donate and support our work.
Donate Now