AdvocacyAPI — Data Overview

Current Database

Representatives by Source

Recent Syncs

Data Sources & Coverage

Representatives

Postcode Mapping

Councillor Email Coverage (Modern.gov)

~200 UK councils use the Civica Modern.gov platform for their democracy portals. These expose a mailing list endpoint (/mgCommitteeMailingList.aspx?EM=1&ID=0) that lists all councillor names and emails as CSV-formatted text.

Two URL patterns:

• https://{council}.moderngov.co.uk — hosted by Civica (most common)
• https://democracy.{council}.gov.uk — self-hosted by the council

Verified working

Councils not using Modern.gov

Some councils use other platforms where a different scraping strategy would be needed:

Scaling approach

The enricher at lib/advocacy_api/enrichers/moderngov.ex maps council names (as they appear in the Open Council Data CSV) to their Modern.gov base URL. Adding a council is one line:

@council_urls %{
  "Brighton and Hove" => "https://democracy.brighton-hove.gov.uk",
  "Doncaster" => "https://doncaster.moderngov.co.uk",
  # add more here...
}

Name matching between Open Council Data and Modern.gov is near-perfect — tested with Brighton and Hove where 53/53 names matched exactly (only the “Vacant” seat was unmatched).

To build out the full mapping, the URL slug can often be guessed from the council name, but not always (e.g., “Brighton and Hove” → brighton-hove, “South Cambridgeshire” → scambs). A manual mapping or a discovery script that tries both URL patterns would be the next step.

Councillor Email Coverage (CMIS)

~60 UK councils use CMIS by Astech Consultants for their democracy portals. Unlike Modern.gov (which exposes a single mailing-list endpoint), CMIS requires a two-step scrape: first fetch the councillors list page to discover individual councillor URLs, then visit each detail page to extract the email address.

The enricher at lib/advocacy_api/enrichers/cmis.ex handles this automatically with concurrent detail-page fetching.

Three URL patterns:

• https://{slug}.cmis.uk.com/{slug}/Councillors.aspx — hosted by Astech (most common)
• https://cmis.{council-domain}/cmis5/Councillors.aspx — self-hosted CMIS 5
• https://cmispublic.{council-domain}/cmis/Councillors.aspx — self-hosted public

Scaling approach

Adding a council requires one line in the @council_urls map:

@council_urls %{
  "Birmingham" => "https://birmingham.cmis.uk.com/birmingham/Councillors.aspx",
  "Dudley" => "https://dudley.cmis.uk.com/Councillors.aspx",
  # add more here...
}

The council name must match exactly what appears in the Open Council Data CSV. The scraper visits each councillor’s detail page concurrently (4 at a time) so enrichment takes longer than Modern.gov but is still practical.

Area-Level Datasets

The API attaches statistical datasets to each postcode via its geography codes (LSOA, MSOA, constituency, local authority). Each dataset has typed headline columns for fast queries and a JSONB detail column for full breakdowns.

LSOA-Level Datasets (Lower Layer Super Output Areas)

LSOAs are small statistical areas of roughly 1,500 people / 650 households. England has 32,844 LSOAs. These provide the most granular neighbourhood-level data.

MSOA-Level Datasets (Middle Layer Super Output Areas)

MSOAs are larger areas of roughly 7,500 people. England has 6,791 MSOAs. Used for data that is too volatile at LSOA level.

Constituency-Level Datasets

Local Authority-Level Datasets

Data Freshness

Key Limitations & Caveats

1. England-centric: Most LSOA-level datasets cover England only (or England & Wales). Scotland and Northern Ireland publish equivalent data but using different geographies and definitions, which are not yet integrated.

2. Boundary mismatches: IMD 2019 and some other datasets use 2011 LSOA boundaries, while Census 2021 introduced new 2021 LSOA boundaries. Most LSOAs are unchanged, but ~5% were split or merged. We join on LSOA code which handles this for matching LSOAs, but some 2011-only LSOAs may not match 2021 codes.

3. Point-in-time snapshots: Most datasets are annual or less frequent snapshots. The API returns the most recently imported version — it does not track changes over time (except house prices which include historical data in the detail JSONB).

4. Modelled vs measured: Several datasets (fuel poverty, child poverty, air quality) are modelled estimates, not direct measurements. They carry inherent uncertainty that increases at smaller geographies.

5. Small area suppression: DWP benefits data suppresses counts below 10 to prevent identification. This means the most deprived areas (few claimants) may show as null rather than a small number.

6. No real-time data: All datasets are retrospective. Crime data has a ~2 month lag, benefits data ~1 month, house prices ~3 months from transaction to publication.