But how is the B&NES region defined?

In this post I wanted to explore that question, paying particular reference to geography, or rather geographies, as there are several.

B&NES: a region and a council

B&NES as a district has only existed since 1996. As the Wikipedia page explains, the district was created after Avon was abolished.

The district is managed by B&NES council which is a Unitary Authority. Unitary authorities are essentially a combination of a county and district council and have responsibility for managing all functions in a region.

When we are referring to B&NES we may sometimes need to distinguish between the region - an administrative district of the UK and the council - the organisation that administers that district.

The District

The Bath and North East Somerset district covers 220 square miles (570 km2). The Ordnance Survey Linked Data site provides a nice overview of the area complete with a map showing the boundary as well as a list of the areas which it touches and contains. All of that data can be downloaded and explored directly from the site.

B&NES is neighboured by six other districts. B&NES is also part of Somerset, but confusingly Somerset is defined as both a “ceremonial county” which contains B&NES and a smaller administrative district which borders it.

Wards and Parishes

Electoral wards are the basic building blocks of the national administrative geography.

B&NES is broken down into 37 electoral wards. The list of wards and parishes is again listed at the official Ordnance Survey URL for B&NES, which links to each ward with details including a GML boundary region. The council also publishes a list of the wards, with links to PDF maps of the areas.

B&NES is also partially divided up into parishes. Bath itself doesn’t have any parishes, but the surrounding area is divided up into 51 parishes. Civil parishes are the smallest type of area in the UK administrative geography.

They also don’t necessarily correspond with the electoral wards. For example Keynsham is a Civil Parish but the same area is divided up across several wards, including Keynsham North and Keynsham South.

This highlights why we sometimes need to be careful when using statistics about local areas: the boundaries may be different depending on how the information was collected and reported.

Parishes and Wards are not the only type of area that overlap in the region

Postcodes

Postcodes are the geographical areas that we tend to bump into most often in our daily lives. Its the geographical identifier that we all typically have to hand. But postcodes are managed by the Royal Mail and are designed to support their mail delivery operations. This means that they regularly change as new houses are built, or Royal Mail alters its distribution. Post codes can also be changed and may be re-used in the future. So as stable geographical areas they aren’t necessarily ideal. Unfortunately its impossible to get post code boundaries as open data so things are even worse.

The Ordnance Survey do make some post code information available as open data. So from their Linked Data site we can find the list of 5750 postcode units in B&NES and this data has been added to the Bath Hacked data store. Postcodes are also organised into higher-level groupings. For example your home post code is a “postcode unit” whereas BA2 is a “postcode district”.

Postcodes don’t necessarily line up with electoral wards so they consist of yet another way to divide up the B&NES area.

Health

Like Royal Mail, the NHS also divides up the UK into regions in order to support its operations. There are two types:

• Area teams which are responsible for GPs, dental services, pharmacies, etc. B&NES is covered by NHS Bath, Gloucestershire, Swindon and Wiltshire area which is part of NHS South

• Commissioning Groups who are responsible for planning and buying of local NHS services. B&NES is covered by Bath and North East Somerset Commissioning Group. In fact the boundary for the group is aligned with the B&NES region.

Statistical Areas

The Office of National Statistics uses its own geography for the purposes of publishing population statistics. The census geography is another hierarchical organisation of the UK. The smallest unit of which is the output area. The areas are sized so that they contain roughly equal numbers of people. This generally means around 125 households, although some are smaller. By dividing the UK into small regions, it becomes easier to identify changes to the population over time, whilst avoiding giving away any personal information.

Because output areas are sized based on the number of people living in them, they vary widely in how much actual land they include. Obviously rural output areas will tend to be larger, while the more densely populated city areas will be much smaller.

It’s possible to browse the statistical geography for B&NES using a tool provided by the ONS. On that page you can elect to see how the area is divided up into output areas, workplace zones, and middle and lower layer output areas. The

The ONS recently published a national address lookup dataset to help map between addresses and the various stiatistics, electoral, health and other geographies in use across the UK. The B&NES subset of this is available from the Bath: Hacked datastore.

Fire, Police, Ambulance Services

To be comprehensive its also worth noting that the fire, ambulance and police services in the UK also have their own geographical breakdowns. Again, like the health and mail services, these reflect organisational jurisdictions.

As a quick reference, B&NES is covered by

• Avon Fire and Rescue covers B&NES and also North Somerset and South Gloucestershire
• Our police force is a division of Avon and Somerset Constabulary. They define several neighbourhoods across the region
• South West Ambulance Service provides the ambulance services across the area and the entire south west of the UK.

Summary

There are a number of different national geographies that cover our local area. When working with local data it’s important to understand the differences between them, especially when it comes to comparing statistics published by different organisations.

Hopefully this blog post provides some useful insight and pointers to further reading.

How does the map fit with your understanding of Bath & North East Somerset?

The map is embedded below, but you can explore the full map on Cartodb. For some of the denser population areas you’ll need to zoom in.

The map consists of a number of colour coded regions. Each region is what is known as an output area. Bath & North East Somerset consists of 583 output areas.

Each output area is colour coded based on the population cluster that lives in that region. The population clusters all have names which are shown on the key.

The rest of this post provides some background on the data and how to read the map.

Output Areas

An output area is the smallest geographical region for which data is available in the UK census. The areas are sized so that they contain roughly equal numbers of people. This generally means around 125 households, although some are smaller.

By dividing the UK into small regions, it becomes easier to identify changes to the population over time, whilst avoiding giving away any personal information.

Because output areas are sized based on the number of people living in them, they vary widely in how much actual land they include. Obviously rural output areas will tend to be larger, while the more densely populated city areas will be much smaller.

Population Clusters

The UK census collects a variety of information about the population. The ONS have assigned each of the output areas in the UK to a population cluster. A cluster is defined by a number of variables, including the demographics of the people living in the area, whether they own their homes, socio-economic indicators and employment status. The ONS have published a list of the variables used to calculate the clusters.

Each cluster has a name, e.g. “Comfortable Cosmopolitans” and a description. These are intended to be a memorable short hand for the more complex description that fully describes that cluster of people.

The colours on the map identfy the cluster to which the output area belongs. The key shows the name and colour associated with each cluster.

Because of limits in Cartodb, some of the less frequently occuring clusters have the same colour and are just listed as “Other” in the key. But if you click on an area it will show the actual name of the cluster.

Because clusters share a definition, then when you see the same colours on the map, then this indicates that there is a similar mix of people living in those areas.

Clusters in B&NES

Here is the full list of clusters that appear in B&NES, in alphabetical order along with a count of how many times they appear.

  6 Ageing City Dwellers
  7 Ageing Rural Dwellers
 78 Ageing Urban Living
 22 Challenged Diversity
 11 Challenged Terraced Workers
 25 Comfortable Cosmopolitans
  1 Constrained Flat Dwellers
  3 Ethnic Dynamics
 16 Farming Communities
 40 Hard-Pressed Ageing Workers
 25 Industrious Communities
  7 Inner-City Students
 29 Migration and Churn
  8 Rented Family Living
 42 Rural Tenants
 62 Semi-Detached Suburbia
 54 Students Around Campus
 46 Suburban Achievers
 96 Urban Professionals and Families
  5 White Communities

Here’s that same listed sorted by frequency:

 96 Urban Professionals and Families
 78 Ageing Urban Living
 62 Semi-Detached Suburbia
 54 Students Around Campus
 46 Suburban Achievers
 42 Rural Tenants
 40 Hard-Pressed Ageing Workers
 29 Migration and Churn
 25 Industrious Communities
 25 Comfortable Cosmopolitans
 22 Challenged Diversity
 16 Farming Communities
 11 Challenged Terraced Workers
  8 Rented Family Living
  7 Inner-City Students
  7 Ageing Rural Dwellers
  6 Ageing City Dwellers
  5 White Communities
  3 Ethnic Dynamics
  1 Constrained Flat Dwellers

This immediately gives an overall sense of the mix of the local area. Exploring the map should help build up a picture of where each cluster can be found.

Pen Portraits

The names of the clusters are interesting in themselves, but as I noted above there’s a description associated with each of them. The ONS have published their full list of portraits in a PDF document.

I’ve not included them here as there’s some useful context in the document. For example each of the clusters is included in a higher-level grouping that defines some common characteristics.

For example there is a super group called “Urbanites” which is defined as follows:

The population of this group are most likely to be located in urban areas in southern England and in less dense concentrations in large urban areas elsewhere in the UK. They are more likely to live in either flats or terraces, and to privately rent their home. The supergroup has an average ethnic mix, with an above average number of residents from other EU countries. A result of this is households are less likely to speak English or Welsh as their main language. Those in employment are more likely to be working in the information and communication, financial, public administration and education related sectors. Compared with the UK, unemployment is lower.

This group contains both “Urban Professionals and Families” and “Ageing Urban Living”, which are the two most frequent clusters in Bath. These are defined as:

Urban Professionals and Families - The population of this group shows a noticeably higher proportion of children aged 0 to 14 than the parent supergroup and a lower proportion aged 90 and over. There is also a higher proportion of people with mixed ethnicity. Households in this group are more likely to live in terraced properties and to live in privately rented accommodation. Unemployment is slightly higher than for the parent supergroup.

Ageing urban living - The population of this group shows a higher proportion of people aged 65 and over than the parent supergroup. Residents are more likely to live in communal establishments, detached properties and flats than the supergroup, with a higher proportion of households living in socially rented accommodation.

Notes on creating the Map

The complete 2011 area classifications can be downloaded from the ONS website.

After exploring the data at a recent Bath: Hacked curators night, Mark Owen extracted the Bath & North East Somerset data from the national dataset and combined it with the boundary information for each output area. The resulting shape file was then uploaded to Cartodb.

The resulting shape file was then uploaded to Cartodb to create the online map.

The full set of B&NES specific data, without the boundaries, is available from the Bath: Hacked datastore. This provides some additional context, as the clusters are organised into supergroups that can provide some additional ways to explore the area.

However the full map is something like 20GB, eating up a fair bit of disk space and memory and making it difficult to use it on older hardware. So I wondered whether there was a way to cut out a limited section of the map focusing on a smaller area. Like Bath for example.

Here’s how I did it.

After I’d failed to convince the OS to do the work for me my first port of call was to look at whether they had published any code that would let me re-run their world generation. Unfortunately they’ve decided not to do that which is a shame. As far as I can tell the world generation is based on open data so would have made a nice showcase for re-purposing their data in a new way. I also didn’t want to re-write it from scratch, although it would be an interesting project for someone to tackle. Especially if they were to use additional open sources.

So I started to dig through the plethora of Minecraft level editing tools to find something that I could use. Ideally I wanted to be able to take the geometry for B&NES and use the OS Minecraft co-ordinator finder to cut out the relevant region from the map.

However I struggled to find a command-line tool that would do exactly what I needed. This Python library for reading Minecraft levels looked like what I wanted. It has a command-line application for interacting with a world, including extracting bounded areas into schematics. However I had mixed results. While I was able to load the OS level and extract a schematic I couldn’t seem to successfully import it into the empty world I’d created.

So I turned to MCEdit which is a graphical Minecraft world editor which also happens to be written in Python. It seems to be well used by people creating large complex Minecraft builds and it has lots of features for copying and pasting sections of levels. I’d been reluctant to use it though as I figured it would be slow dealing with such a large world. I was also hoping to script something that could be tailored by other people interested in generating sections of the map covering their own area.

Anyway, I found that MCEdit, while slow to open the file, does provides a nice 2D way to navigate through the world (press Tab when you’ve loaded a level to switch perspective). So by jumping to the co-ordinates above Bath (around X:15000, Y:100, Z:45600 will do it) I was able to see the region I wanted to copy.

The MCEdit selection tool allows you to grab an area which you can then extract. You can also choose to “prune” the rest of the world leaving just your selection. I doubt this is the most efficient way to do it as it takes a couple of hours on my laptop with some fiddling around. But the end result is a level containing just the region surrounding Bath, Keynsham and Bathampton. Its also a lot smaller, about 20Mb.

If you’d like to download a copy of the level then you can download it here. The map contains Ordnance Survey Data © Crown Copyright and Database Distribution 2014.

I’ve set the spawn point to be on Beechen Cliff overlooking the centre of Bath.

Note that the scale isn’t 1:1 with normal Minecraft. IIRC a Minecraft block is essentially a 1m cube. For the OS map they explain that a block is more like a cuboid measuring 25 m x 25 m x 12m.

The OS documentation indicates which block type maps to which type of geographical feature. You can also follow the instructions in their readme to install the level.

What can you do with it? Well, I was wondering whether it might provide a fun and alternative way to visualise some local data. Let me know if you do anything with it!

Going into the hackday I realised several things.

Firstly, there was a lot of data available. BANES council had made an archive of 13 years of air quality data available. I’d also discovered that the DEFRA Air Quality data archive had data for one sensor dating from 1997. That’s a lot of data to get to grips with. I decided that I was going to focus on trying to summarise the dataset rather than build anything on top of it.

Secondly, I also didn’t really know anything about air quality. To get prepared I produced some documentation for everyone attending the hackday that summarised the main data source, and provided some useful pointers. I also did some reading around on both the BANES and UK-AIR websites.

Air quality data analysis is a complex area. There are a lot of factors to take into account including a variety of sources of pollutants, complex interactions between pollutants and impacts from the prevailing weather conditions. BANES publish some summary reports but while informative they didn’t really give me a sense of where the pollution was coming from, or how bad it was at different types of the day or year.

I also discovered the Open Air project which provides an R package to support air quality analysis. It also comes with an amazing set of documentation: the manual is over 200 pages and includes a short introduction to R.

So I read the manual and went into the hack day with a goal of trying to answer two questions:

1. Can we provide Bath citizens with more insight into the air quality for Bath?
2. Can we provide the local council with new ways to generate meaningful visualisations and summary reports?

I didn’t get as far as I’d hoped, but I managed to do enough to create what I think is an interesting summary of the data.

Using R and openair I was able to quickly import, normalise and explore the data. In fact R makes it so easy to generate diagrams that I spent a lot of the day just playing with graphs.

The judges also liked the results and I was lucky enough to win the Most Educational Project prize. The report is now featured on the BANES website.

I’ve also published the code on github if you’d like to explore. The main report code could easily be customised to use an alternate DEFRA location if you want to try it on some data from your local area.

There’s more to explore here, not just around the data analysis, but also around the concept of having reproducable data analytics.

Reproducability is an important part of scientific research and analysis. At least one driver of the growing adoption of open source and open data in the research community is to make science more reproducable: it should be possible for someone else to pick up your research to easily check the results and maybe go a step further.

I’ve not yet seen this idea extended to publishing of analysis of open (statistical) data, but the concepts are the same. Reproducability is another way to increase transparency. Open data has been shown to help people find data errors, but open source can also help people find and fix errors in an analysis itself.

I’ll certainly be playing more with R over the coming months. I’m sold on the ease with which it’s possible to really quickly explore a dataset.

The other hacks produced on the day were all really interesting. I recommend you read the run down of the entries on the BathHacked blog.

I wondered where the data for this analysis came from and whether it would be possible to repeat it using open data. So I picked up a copy of the Chronicle, opened my web browser and explored further.

Firstly before we dig deeper I’ll state up front that I have no political agenda here, or any particular interest in criticising the Chronicle. What I want to do is see whether the data reported in the article is available and whether the facts are correct. If we can highlight some inaccuracies or come up with some interesting insights along the way, then great.

To summarise, the key statements facts from the article are as follows:

• The ratio of house price to salary in 1997 was 4.62
• The ratio of house price to salary in 2013 the ratio was 8.74 (89% higher than 1997)
• Across the South West the average ratio is now above 5
• The Bank of England will now be limiting the number of risky mortages, assessed as those with a ratio higher than 4.5
• Cotsworld is the most unaffordable area in the South West with a ration of 11.6

The article also includes some notes on most and least affordable places in the UK. But it was the following quote that made my raise my eyebrow:

“The TUC has been unable to release the figure for the average house price or the average salary in B&NES, because that data, which the study is based on, is not available from the Government”

Unfortunately that statement is simply not true. I can’t tell whether this is a misunderstanding on behalf of the reporter in the Chronicle or a mis-communication from the TUC. Like most newspapers the Chronicle doesn’t link to its sources. But after some searching I found a TUC report on prices in the North West, but not one specifically on the South West.

But I know that the statement is not true because all of the raw data required to calculate these averages and the resulting ratio is available as open data:

• The Land Registry have been publishing their price paid data as open data for some time. The available data goes back to 1995. So we have the price of every house sale in B&NES for nearly 20 years.
• The ONS Annual Survey of Hours and Earnings (ASHE) provides information on earnings across the UK and data is available from 1998 to 2013, although the latest figures are still provisional. There’s a visualisation of changes in weekly earnings by region.

So the raw data necessary to re-calculate the figures reported by the Chronicle and the TUC is available as open data for anyone to re-use.

But not only that, the Department of Communities and Local Government (DCLG) actually publishes detailed figures on the housing market which includes the ratio of house prices to salary! This appears to be the actual source of the data and is actually linked to from the TUC report on the North West. Reading the Chronicle article though you might be lead to assume that the TUC have produced some analysis using unreleased data, when in fact the source is the government themselves.

The data we’re interested is included in Table 577: ratio of median house price to median earnings by district, from 1997. There’s also a PDF containing charts that plots the changes across the entire country. Looking at that data we can confirm all of figures reported in the article.

Not only that, but because we have access to the actual underlying data we can also take into account a few caveats that ONS and DCLG include in their supporting notes. None of these invalidate the figures, but may be useful when interpreting them. For example we can learn that:

• the ASHE is based only on 1 per cent sample of employee jobs
• the ASHE only includes data from employers, so it doesn’t cover self-employed people
• the reporting regions are based on district boundaries from 2009, so changes to size/shape of districts since that date won’t be reflected
• the 2013 figures are still provisional and are subject to change

Perhaps most importantly though we can see the entire set of data on the ratio of house prices to salary for B&NES from 1997 to 2013. I’ve taken a copy of that data and uploaded it to the Bath Hacked data store. I’ve used that to create the graph shown below:

As you can see the ratio has increased significantly since 1997 with the largest increases in the period between 1999-2004. Interestingly the ratio seems to have dropped slightly over the last few years: the provisional figure for 2013 is similar to that for 2004.

So as a result of the exercise I’ve learnt something about the data that wasn’t reported in the Chronicle. Clearly the ratio is still very high but somewhat encouragingly the local trend in prices and earnings suggests the ratio is flattening out, so we’re not suffering from a runaway increase over the last few years. Whether that is based on changes to house prices or salaries isn’t immediately clear, but we can dig into the data further to find out.

I was also able to highlight an inaccuracy in the article: far from withholding figures the government is providing a lot of useful data in this area.

Hopefully this is useful for others too and highlights some of the possibilities for using open data from a local perspective. The Bath Hacked store could become a useful source of information for adding important context to local news stories and economic trends.

I’ve never completely given up on the idea behind DataSulis though. While I’ve not had much time to do any visible work I’ve continued to research open datasets that might be useful to the local community. My work with the Open Data Institute and others has also given me some useful experience.

I was about to kick-start this project again when I discovered Bath: Hacked. It turns out that I’m not the only person passionate about open data in Bath and they’ve been extremely busy!

The @BathHacked team have been working with B&NES to open up some datasets. After a succcessful hack day earlier this year they’ve now launched a beta data store based on Socrata.

As a result of this I’ve been reworking the code in the DataSulis github account to add support for posting the data to Socrata. I’ve also been spending time exploring the Socrata platform. I volunteered to help out with BathHacked and have agreed to help manage the data store, to help people get the most out of it and ensure that the datasets are well-published.

There’s some interesting datasets in the store already, including some continuously updated air quality data taken from sensors around the city. B&NES are running an air quality hack day in a few weeks to encourage developers to use the data to build some interesting applications.

While I’ll be blogging on the Bath Hacked website, contributing to the BathHacked github project, etc. I’ve decided to revive this project. My plan is to use this site to write about my own personal perspective on open data in Bath, publish investigations of useful datasets, and share updates on my own hacking with local open data.

One question I got asked is: how do we move this forward? I thought I’d post some ideas about next steps, as well as some suggestions for how people can get involved.

First of all though lets consider why we might want to collate some local data. There are a number of good reasons. The primary reason is to have free, Open Data that can be used to create some useful applications. Those app can be useful not just for local people, but also for the thousands of tourists that visit the city each year.

Those applications could be built by local startups and provide a way for them to earn a few quid. The apps could also help local businesses increase their revenues, by driving attracting more customers. If we’re provided with better data from the local council, or central government, we can all be better engaged and help it deliver better services to us.

Open Data is pure win.

With these kinds of uses in mind, what kinds of data might we want to collect:

• Government data – There’s a whole range of potentially useful, interesting statistics published by central government, or by the likes of the NHS, that could be re-purposed for local use
• Local Government data – Planning applications, health & safety inspections, etc.
• Business listings – How about a free, locally maintained database of local businesses, with contact details, up to date information on opening-closing times, twitter accounts, blogs, and calenders?
• Travel information – Bus and train times. Cycle paths
• News – Searchable indexes of news sourced both locally as well as from national newspapers, e.g. the Guardian
• Reviews – local restaurant, business, and event reviews
• Jobs – local job adverts
• Social Network – WeLoveBath is becoming the focal point of Bath’s social network. Who is in it and how do they connect up? What about Facebook, Foursquare, etc?
• Media – Flickr and Geograph are full of pictures of Bath. Wouldn’t it be nice to have a dataset of Creative Commons licensed photos for use in local applications? Maybe to power a new kind of tourist guide? What about sound and video? Music?
• Events – There’s a lot going on in Bath for a relatively small city. I’d love to see a central database and calendar of events across the city, not just in the Theatre, or Komedia, but all of the pub nights, reading groups and other local events.

…and there’s a whole lot more. Cultural heritage, local walking routes, weather, etc. I’ve even had crazy ideas about mapping out all the species of trees in the Arboretum in Victoria Bath. Maybe a school project?!

So how can you (yes, you!) get involved?

• Are you a geek? Yes? Then why not:
  • Share some code showing how to query, collect or scrape some data together for other developers to build on? A lot of data is out there already, we just need to make it easier to find and access
  • Use some of the data showcased here to create an useful application or visualisation?
  • Help build an application or service to help crowd-source some data?
• Are you a non-geek (aka Normal Person)? Then how about:
  • Sharing some thoughts on what kinds of local application or service you’d like to see?
  • Curating some data using a Google Spreadsheet? No coding required, but the data is still easy to share with others
• Are you a local business or firm? Then how about:
  • Getting in touch to see how you could help share some of you data for others to use?

I’ll happily point to whatever people hack, build, or share from this blog, just send in pointers. I’ll try and showcase as much as possible whilst continuing to add more content and collated data myself.

I’m keen to take a lo-fi approach and just do the simplest things necessary to help collect some data and make it easily accessible to whoever needs it. There are lots of useful free tools available. For example Google Spreadsheet Forms are a quick and dirty way to crowdsource some basic information.

To really get things going I suggest we first try and collate some more data, and then maybe have a local hackday to try build something on what we’ve collected. And/or have a focused hacking session to collect together even more data. Or both.

Oh, and the cider. Don’t forget the cider.

As the Land Registry website explains, this data relates to:

residential property sales in England and Wales that are lodged with us for registration. The data includes:

• the full address of the property (Primary addressable object name (PAON), Secondary addressable object name (SAON), street, postcode, locality (if available), town, district, county)
• the price paid for the property
• the date of transfer
• the property type (Detached, Semi, Terraced, Flat/Maisonette)
• whether the property is new build or not
• whether the property is freehold or leasehold.

We can filter their data to grab just the Bath prices.

There is a simple script in this github project which looks at the Land Registry website and grabs whatever CSV files are available. The CSV files are then read and filtered to just grab the data that relates to properties in the BA1 and BA2 area.

Currently the Land Registry are publishing this data on a monthly basis, so there is only a single month available currently. I expect that more data will appear over time. Here’s how it looks today.

For more background on the codes used in the data then read the Land Registry FAQ.

Their data is published under the Open Government License so can be freely reused. Lets take a look at what it contains and how we can grab a local extract.

If you visit the ODS Website you can see that they publish a large number of CSV files that contain data about organisations and their relationships. The structure of the CSV files is well-documented and the files are well-normalised so they’re easy to process.

Some of the data is updated on a weekly basis. This is the core NHS organisational data. Other data, such as that listing GPs and Branch Surgeries is updated less frequently.

The data is useful for two reasons:

• If you want to draw maps showing, e.g. location of pharmacies then there’s well normalised address data that can be geocoded.
• If you want to link up government statistics with individual medical practices or service providers, then you’ll need the unique identifiers

I’ve previously taken all of the ODS data and loaded it into Kasabi. Kasabi is my day job: its a data marketplace for hosting and publish data. If you want to query the data online then you could use any of the range of APIs available from there. Sign-up is free.

For the purposes of the DataSulis project we’re only really interested in the data about the NHS in Bath, i.e. BA1 and BA2. I wrote some Ruby scripts to download the latest ODS files and then extract those organisations that have the relevant postcodes. The filtered CSV files are then cached locally. You could use them as the basis for further processing.

To illustrate the output I’ve put a snapshot of the data into github too. For example here’s a CSV file containing a list of the Dental Practices in Bath

So lets look at how we can work with their data to query it and extract it for local use.

Here’s a quick primer on postcodes. Postcodes have a structure to them. Each of the different parts of a postcode refer to a different area and those areas have a hierarchical relationship. Here are some examples along with the name that the Ordnance Survey (OS) uses to describe them:

• BA = Post Code Area
• BA2 = Post Code District
• BA2 3 = Post Code Sector
• BA2 3PL = Post Code Unit

So “BA2 3PL” is within a sector called “BA2 3”, and so on. The OS publish their data in various ways, including as Linked Data. Without going into details, Linked Data is just a way to publish data to the web by giving everything a unique URL.

So based on a postcode or part of a postcode you can build a URL to the OS website and use it to grab some data. For example here’s a JSON description of BA23PL. That means that you can quickly lookup some useful data such as the lat/long which is the centre of a post code, or to discover in which electoral ward it lies. More on those alternate geographic regions in another post.

Sometimes though you just want a to grab some data for local processing. Having a list of local postcodes can help drive some address matching or other data processing task. So how can we get a complete list of local postcodes?

The OS allow you to download data from their site, so you could grab all of the postcode dataset and process it to extract what you need. But there’s a simpler way. The OS also provide an API called a “SPARQL Endpoint” for their data. SPARQL is a query language for working with RDF, its basically a way to query a graph of Linked Data to extract the bits you need.

Here’s a SPARQL query that will fetch data about all of the Post Code Units that are within the BA1 or BA2 Post Code Districts:

If we submit that query to the OS SPARQL Endpoint then extract just the data we need.

Here’s some simple Ruby code that does exactly that. It requests that the SPARQL API return the data as JSON and then spits it out as a simple CSV file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

require 'rubygems'
require 'json'
require 'net/http'
require 'cgi'
require 'csv'

dir = File.dirname(__FILE__)

query = File.read( File.join(dir, "..", "rq", "list-bath-postcode-units.rq") )

Net::HTTP.start('api.talis.com') do |http|
  req = Net::HTTP::Get.new("/stores/ordnance-survey/services/sparql?output=json&query=#{CGI.escape(query)}")
  response = http.request(req)
  postcodes = JSON.parse( response.body )
  CSV.open( File.join(dir, "..", "data", "bath-postcodes.csv"), "w") do |csv|
    postcodes["results"]["bindings"].each do |postcode|
      csv << [ postcode["id"]["value"], postcode["code"]["value"], postcode["latitude"]["value"], postcode["longitude"]["value"] ]
    end
  end
end

Here’s another version that generates a JSON description instead.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

require 'rubygems'
require 'json'
require 'net/http'
require 'cgi'

dir = File.dirname(__FILE__)

query = File.read( File.join(dir, "..", "rq", "list-bath-postcode-units.rq") )

Net::HTTP.start('api.talis.com') do |http|
  req = Net::HTTP::Get.new("/stores/ordnance-survey/services/sparql?output=json&query=#{CGI.escape(query)}")
  response = http.request(req)
  postcodes = JSON.parse( response.body )
  output = {
    "postcodes" => {}
  }
  postcodes["results"]["bindings"].each do |postcode|
    output["postcodes"][ [ postcode["code"]["value"] ] ] = {
      "id" => postcode["id"]["value"],
      "latitude" => postcode["latitude"]["value"],
      "longitude" => postcode["longitude"]["value"]
    }
  end
  File.open( File.join(dir, "..", "data", "bath-postcodes.json"), "w") do |file|
    file.puts JSON.pretty_generate(output)
  end
end

And here are the generated files as both CSV and JSON. Those files are provided to save you re-generating them yourself but you shouldn’t assume they’re always going to be up to date.

This site is an experiment in creating a collection of useful pointers, blog posts, datasets and maybe even APIs for local hackers in Bath. As I’ve explained a little in the about page I’m interested in exploring how a more local, city level view of Open Data could help support some interesting innovative hacking amongst the local geek community.

As the recent [Bath Digital Festival] has shown Bath has developed a really amazing local tech community. [WeLoveBath] also shows that Bath has a great community of engaged citizens. But I’ve not seen any efforts to start curating local datasets that can help both of those communities do interesting, innovative things in our local area.

This site sets out to see if we can remedy that. Basically, I’ve spent a lot of time over the last few years doing data modelling and munging and I decided its time to use those skills for good :)

I expect this will be something of a journey and I’m hoping that some of you out there will want to join in and help out. I’m hoping that if its successful then maybe we can run a BathCamp or hackday dedicated to local data projects.

It doesn’t matter if your a geek or not I’m hoping that we can explore lots of ways to curate and collect local data. I’m as interested in running some local crowd-sourcing experiments as I am in wrangling data from various APIs.

The project is intended to be open from top to bottom, so all of the content, source code and data should be up for re-use.

Lets see what we can make.