Building a foundational dataset of Third Sector organisations in the UK

Overview

This blog post describes the creation of a dataset which aims to provide an extensive list of third sector organisations operating in the United Kingdom. We do this by drawing on administrative and regulatory data sources for such organisations and by gathering all unique names, addresses, registration dates, and dissolution/removal dates for each organisation. This work has been carried out as part of the ‘Improving access to and use of organisation-level data on the third sector and civil society’ project (ES/X000524/1). This ESRC funded project aims to link a foundational dataset of third sector organisations to a wide range of data obtained through data sharing agreements (e.g., ONS Business Index) or gathered from open data sources (e.g., 360Giving grant data). Linking to such data sources will allow us to expand our understanding of the third sector in the UK. For example, linking our foundational data to the grant data curated by 360Giving opens up avenues for new analyses on the size, frequency, and distribution of grants awarded to third sector organisations. The overall aim of this work is to create a single, accessible resource for analyses of the capacities, funding, and organisational dynamics of third sector organisations in the UK. As such, the project seeks to address persistent calls for better-quality and timely data on third sector and civil society organisations. In the following sections we outline the creation of an open and flexible version of our foundational dataset of third sector organisations. This dataset can be used by future researchers to link many types of useful data and improve analyses in this field.

Data sources

We utilise relevant extant data sources that are generated and shared by UK bodies with regulatory oversight of third sector organisations (e.g., Charity Commission for England and Wales). Additionally, we incorporate data generated by membership bodies such as Co-operatives UK where available. To construct our foundational dataset, we use contemporary open data provided by each regulator or membership body and several historical data files held by members of the research team. The open-source data used to generate the full dataset are listed below, followed by details about processing carried out per source. We discuss the historical files used in later sections.

Companies House

Companies House provide a quarterly download of all currently active registered companies in the United Kingdom, and an API (Application Programming Interface) which can be used to access data of older organisations as well as other fields not available in the bulk downloads. For our purposes we accessed Companies House data from three sources:

1. Bulk downloads released during the period of this project (‘COH 2023’)

2. Historically collected data by researchers on our team containing organisations active up to 2014 (‘COH 2014’)

3. A search using the API of organisations which were active between 2014 and now, but dissolved prior to the date of the available download. (‘COH gap’)

This data was then filtered to only include those whose company category was one of the following:

• PRI/LTD BY GUAR/NSC (Private, limited by guarantee, no share capital)
• Charitable Incorporated Organisation
• Community Interest Company
• Registered Society
• PRI/LBG/NSC (Private, Limited by guarantee, no share capital, use of ‘Limited’ exemption)
• Scottish Charitable Incorporated Organisation
• Industrial and Provident Society

The three companies house data sources yielded 413,900 unique companies; with a per-source breakdown as follows. The October 2023 bulk download contains 234,828 organisations, the historical data file from 2014 research contains 215,298, and the companies house API search yielded 312,009.

The overlaps between the data from the three sources is visualised in the Venn diagram in Figure 1.

Figure 1: Intersections between the three sources of Companies House data

Charity Regulators

Among the key data sources for this project are the three charity regulators operating in the UK: The Charity Commission for England and Wales (CCEW), Office of the Scottish Charity Regulator (OSCR), and Charity Commission for Northern Ireland (CCNI). Each of these regulators maintain open data on registered charitable organisations including information on names, addresses, registration dates, and removal dates. We combine 2023 open data extracts with historical data files to maximise data coverage in terms of unique organisations as well as coverage of unique names and addresses for each organisation. We supplement the CCEW open data extract with 13 historical data iterations from 2001 to 2022. OSCR open data were supplemented by four historical data files from 2012 to 2021. The CCNI, as the newest of the three regulator data sources, is not supplemented by historical files. Figure 2 outlines these data sources.

Figure 2: Charity regulator data sources

In the cases of CCEW and OSCR, the open and historical data were internally linked using the charity ID assigned by the regulator. Using these IDs as a grouping variable, we isolated unique names and addresses by standardising the relevant variables (e.g. converting to lower case, remove leading blank spaces, collapsing consecutive internal blank spaces etc) and removed duplicate information. Use of supplementary historical files proved successful in significantly increasing data coverage. These files account for an additional 47,174 unique names and 344,268 unique addresses for England and Wales. For Scotland, we extracted an additional 3,523 unique names and 10,882 unique addresses. Table 2 below offers an overview of unique organisations, names, and addresses split by regulator.

Other data sources

Data from Companies House and the three charity regulators provide significant coverage of the third sector. However, we use several additional sources to supplement these data and create an extensive base of voluntary organisations. We provide full details of these sources and how they contribute to the dataset in table 3 below.

Combining sources

Each data source contains information in different forms. In order to normalise this for one dataset, a target set of fields was defined, and processes to ingest data were written per source. A unique identifier (UID) was created for each organisation using the conventions adopted by FindThatCharity.com, namely GB-{source}-{org_id}. Source identifiers are listed above.

Names were ingested as they appear in the source, and also converted to a normalised form, capitalising, removing punctuation and extra spaces. Addresses were mapped to fields for address lines, city and postcode (where possible). Dates were normalised to all be of the form dd/mm/yyyy.

Following this per-source process, data from all sources were concatenated to form one dataset. This dataset was then deduplicated where organisations appear in more than one dataset with the same information.

Organisations were linked between datasets using two deterministic metrics: exact matches of the companyid, and exact matches of the normalised name. Companyid matches yielded 35649 links between Companies House organisations and Charity Regulator organisations, 127 links within Charity Regulators, and 130 links between the Mutuals Public Regulator and Social Housing England.

Matching using the normalised name field yielded large numbers of linkages, due to names being indistinct. For example, the name ‘ART’ occurs in the CCEW register 12 times, each with a different charity number. Of these, five have unique corresponding company ids. It is unlikely all 12 of these are the same organisation. The name ‘SCOTTISH AUTISM’ is found once in Companies House data, once in OSCR, and 38 times in Care Inspectorate Scotland’s list of organisations. These are likely to be the same organisation, with different sites.

We determined that for our purposes it was useful to limit the number of matches to at most 3 sources per linkage, of which at most one can be from Companies House. This led to the pattern of overlaps seen in the upset plot in Figuer 3.

Figure 3: Upset Plot of linkages between data sources, where name and/or company number match exactly

Data limitations

It is worth noting that this dataset of organisations, while extensive, cannot be exhaustive. The nature of the third sector means we are reliant on high quality data being maintained and published by relevant regulators or membership bodies. As such, there are many third sector organisations operating in the UK that exist outside the scope of these datasets because they are too small or operate on a very constrained basis. Ideally, we would be able to capture all voluntary work carried out in order to provide a full overview of the sector. However, the available data does represent the overwhelming majority of the third sector in terms of income, expenditure, employment, and service provision. As such, this foundational dataset provides a valuable addition to the evidence base of this field and can be used to improve analysis of the third sector in the future.

Conclusion

In this blog we have detailed the creation of an open foundational dataset of UK third sector organisations, provided examples of how this can be linked to a diverse range of data, and have discussed some limitations of the available data sources. This work contributes to the first national database on the population of organisations forming the third sector through bringing together information about charities with information about different kinds of noncharitable civil society organisations (including Community Interest Companies, Co-operatives and Mutuals, and non-profit Companies Limited by Guarantee). As such, this resource improves the information base of third sector research, will open up new avenues of investigation, will help to inform decisions by government and other significant funders,  and will refine the ability of sector stakeholders to obtain a research-led and sector-wide overview of voluntary organisations.