Non Government data collected and used by ONS
Dear Office for National Statistics,
Can you please publish a list of data (or documents that show the list of data) from non-government organisations that (a) is web scraped by ONS, (b) has been requested by the ONS, and (c) has been received and is actually used by ONS. Where data is being used by ONS, please say if it is published in its own right or, if not, which published variables each data set contributes to.
Yours faithfully,
Simon Briscoe
Workflow Notification
--------------------------------------------------------------------------
Thank you for your Freedom of Information request.
We can confirm that your request has been received and is being dealt
with. We will aim to respond to your request as soon as possible, and no
later than 20 working days from when we received your request. Should we
experience any delays with processing your request by this deadline, we
will keep you informed.
If you have any further questions please do not hesitate to contact us at
[1][ONS request email].
In the meantime, please take a look at our previously published FOI
requests via the following link:
[2]https://www.ons.gov.uk/aboutus/transpare....
If you find the information that you are looking for on our website,
please get back in touch with us to close your request.
Yours Faithfully
The Freedom of Information Team
Office for National Statistics
--------------------------------------------------------------------------
References
Visible links
1. mailto:[ONS request email]
2. https://www.ons.gov.uk/aboutus/transpare...
Our Reference: FOI/2022/3995
Dear Simon,
Thank you for your email requesting data pertaining to Non-government data
sources.
I am writing to confirm that the Office for National Statistics has now
completed its search for the information which you requested and the
response can be found here:
You asked:
FOI Reference: FOI/2022/3995
Can you please publish a list of data (or documents that show the list of
data) from non-government organisations that (a) is web scraped by ONS,
(b) has been requested by the ONS, and (c) has been received and is used
by ONS. Where data is being used by ONS, please say if it is published in
its own right or, if not, which published variables each data set
contributes to.
Our response:
Thank you for your request.
ONS uses multiple data sources from government and non-government
organisations to support the ONS statutory function. The following
information relates to some our non-government data sources.
Financial transaction data (business to consumer, business to business)
solely used for statistical analysis of UK card spending. You can find the
data sources here: [1]Statement on data sources
Point of sales data from UK retailers, and other commercial prices data
(used car sales and public transport fares) to support the transformation
of inflation and retail sales statistics. Several web scraped data
covering clothing, electronic items and package holidays are additionally
used for the consumer price statistics. An example of a retailer data used
is here: [2]Transformation of consumer price statistics : November 2021
Mobility data to support the impact of mobility restrictions during the
Covid-19 pandemic. How we used these data sources are listed
here: [3]Understanding mobility during the COVID-19 pandemic
UK Business websites text data from glass.ai to inform survey response
chasing efforts and gain insight into the impact of
Covid-19: [4]Extracting text data from business website Covid-19 notices.
Utilities, Students Hall, Holiday home, and Hospital & Care home address
data to validate the active addresses of households for the Census 2021
only.
Trade data from numerous sources including CAA, providers of road freight
surveys, Vessels Value data and UK Chamber of Shipping data which,
combined with other government data, help form the national
accounts: [5]UK National Accounts, The Blue Book
National expenditure data from commercial sources, including people's
spending on horses, private airplanes, and boats which is used within the
calculation of GNI within the national accounts: [6]All data related to
gross domestic product (gdp)
Data relating to the 17 sustainable development goals is provided from
open licence data in the public domain and is published on the SDG
website: [7]UK data for the Sustainable Development Goals
Data for weekly real-time Faster Indicators, including Adzuna job vacancy
data and Springboard retail footfall data: [8]Economic activity and
social change in the UK, real-time indicators
Other non-government data sources that ONS has acquired and contain
personal information are all published here: [9]External data sources
containing personal data
Please find information relating to our web scraped data sources in the
attached downloads (a_web_scraped).
We intend to publish the information requested in parts b) and c) of your
request. Therefore, the information is exempt under s.22(1) of the Freedom
of Information Act (FOIA) 2000, as there is an intention to publish this
information in the future.
This exemption is subject to a public interest test. We recognise the
desirability of information being freely available. The concept of
transparency has been seriously considered by the Office for National
Statistics when setting out to create and publish lists of our data
sources but also disclosing everything we reasonably can in response to
this request.
A large part of this project is discussing the publication of these data
sources with our suppliers, particularly balancing the concepts of
transparency and confidentiality. Early disclosure, prior to completion of
this project and appropriate discussions with our data suppliers, could
undermine our supplier relationships and our functions should access to
data sources be withdrawn.
As said previously, we obtain these data sources to fulfil our functions
as not only a producer of official statistics that inform the public good,
but as a supporter of statistical research and the research community. If
our access to such data sources were affected, this could undermine our
ability to fulfil our statutory functions, which is not in the public
interest.
You have the right to have this response to your Freedom of Information
request reviewed internally by an internal review process and, if you
remain unhappy with the decision, by the Information Commissioner. If you
would like to have the decision reviewed please email the Freedom of
Information Team at ‘[ONS request email]’ or write to The FoI Team, Office
for National Statistics, Segensworth Road, Titchfield PO15 5RR, and mark
your correspondence "Internal Review".
If you have any queries about this email, please contact the Freedom of
Information Team ('[ONS request email]').
Please remember to quote the reference number above in any future
communications.
The Freedom of Information Team
Office for National Statistics
For information on Coronavirus, please visit [10]ONS.gov.uk/coronavirus
References
Visible links
1. https://www.ons.gov.uk/news/statementsan...
2. https://www.ons.gov.uk/economy/inflation...
3. https://datasciencecampus.ons.gov.uk/und...
4. https://datasciencecampus.ons.gov.uk/ext...
5. https://www.ons.gov.uk/economy/grossdome...
6. https://www.ons.gov.uk/economy/grossdome...
7. https://sdgdata.gov.uk/goals/
8. https://www.ons.gov.uk/economy/economico...
9. https://www.ons.gov.uk/aboutus/transpare...
10. http://ons.gov.uk/coronavirus
Dear FOI Team,
You responded by saying "The following information relates to some (of) our non-government data sources."
Do you have a full list of data used? If you do have a full list can you please send it to me?
Elements of your response are vague. You say: "National expenditure data from commercial sources, including people's spending on horses, private airplanes, and boats which is used ........". This does not give the sources - are you able to give them?
Yours sincerely,
Simon Briscoe
Dear FOI Team,
Thank you for this response. The response contains some high level narrative about non-government data sources but does not answer the question. The starting point must be to present a list of all the non-government data that is used by the ONS.
For example, where you say "National expenditure data from commercial sources, including people's
spending on horses, private airplanes, and boats", there is a need to say exactly what is collected and the source.
If the ONS does not know what data it uses, and all the components of its published statistics, then please say so.
Yours sincerely,
Simon Briscoe
Dear Simon
Thank you for your email, and please accept my apologies for the delay and the confusion with this response.
To clarify, we have provided a list pertaining to web scraped data sources in response to part (a) of your request. Parts (b) and (c) of your request are exempt under s.22(1) and we have provided the reasoning in our response.
You have referred to a high-level narrative about non-government data sources. We have provided this as additional information that you may find of interest in order to satisfy our duty to assist.
Best wishes
Steph
Steph Turner | Legal Services
UK Statistics Authority | Awdurdod Ystadegau’r Deyrnas Unedig | @UKStatsAuth | [email address] | http://www.statisticsauthority.gov.uk
ONS.gov.uk/coronavirus
We work to defend the right to FOI for everyone
Help us protect your right to hold public authorities to account. Donate and support our work.
Donate Now