Billy Ehrenberg-Shannon; Aleks Wisniewska; John Burn-Murdoch; Steve Bernard; Ændrew Rininsland; Callum Locke; Valentina Romei; Avtar Rai; Haluka Maier-Borst.
Throughout 2017 and 2018, the Financial Times has developed a specialism in producing rapid-response overnight analyses of elections, leveraging our data collection and visualisation skills to turn around insightful and visually striking reports on several elections across Europe, responding faster than other news organisations both in the UK and even those based in the countries where these elections have taken place.
Over and above simply providing the top-line results, we have focused on adding insight by identifying and explaining salient voting patterns, highlighting significant associations between the characteristics of people and places, and the political causes they support.
We have produced detailed analyses of the French (May 2017), British (June 2017), German (September 2017) and Italian (March 2018) general elections in this way, providing readers with a framework for understanding the political shifts taking shape across Europe — in some cases before the last votes have even been counted.
What makes this project innovative?
Delivering work of this quality and timeliness for different countries, socio-political contexts and electoral data formats has only been possible due to four key ingredients:
First, the team has developed highly versatile skills in data scraping and cleaning. This has allowed us to quickly obtain, process and analyse results regardless of whether results are received in a neatly formatted feed — as in the UK — or scraped from unfamiliar or noncompliant websites — as with Italy, where a new results website meant the scrapers that would be relied on to pull in tens of thousands of pages of results could only be developed as voting began.
Second, we have carrying out “election rehearsals” — practice runs of election night to make sure our workflows for obtaining, cleaning and visualising data are all polished, and robust to any glitches that might come up on the night of the count.
Third, our data and graphics specialists have pored over news reports and academic literature from the countries whose elections we have analysed, ensuring that the analyses and graphics we have produced on the night have fully-developed theories to match. This has ensured that despite the quick turnaround of our work, we have not been highlighting spurious or uninteresting patterns in the data, but have instead contributed to the ongoing debate in every country we have covered.
And finally, members of the team have committed — voluntarily — to going well beyond the call of duty in terms of the number and timing of hours worked, in order to have our insights on the page by the time our readers wake up.
What was the impact of your project? How did you measure it?
The work has demonstrably paid off, with readers from continental Europe outnumbering those from Britain and the United States — typically far larger audiences for the FT — for the data team’s analyses of the French, German and Italian elections, and large numbers of social media users from these countries also promoting the FT’s work.
Source and methodology
For each election, the team identified official data sources at the most granular possible level, with the guidance of local academic experts and the FT’s network of correspondents. This included official election results from interior ministries or election authorities (or the Press Association elections service in the case of the United Kingdom), demographic data from census and other official statistics sources and exit poll data where this was available.
R scripts were written in advance to scrape the electoral results services in real time and join them to the static, pre-sourced demographic data. Based on pre-election reporting about the likely social divisions that would be revealed by the election, potential stories and their associated graphics were pre-prepared to allow rapid overnight production once the electoral data became available.
Since each country’s data formats and electoral system is very different, little of the material could be reused between elections, but the workflows and processes were refined based on the previous election experience.
Scraping and analysis was primarily conducted in R, with most final projection graphics created in D3 — often adapting the Financial Times’ Visual Vocabulary library of data visualisation formats. Maps were initially produced in QGIS.