With all the hype surrounding big data it’s difficult to consider what other alternative approaches there may be to gain insight into what is happening within a society. In government the traditional approach of data collection focuses on survey data or data generated from government information management systems that have been aggregated into a centralised data warehouse. These are then analysed and used for informing policy. The problem is people lie on surveys all the time and operational information management systems only capture a small part of what’s really going on.
Anecdotes and stories are rich in insights. Unfortunately collecting anecdotes using traditional methods doesn’t scale well and people’s biases tend to get in the way during the collection and analysis process. If we could scale the capturing of anecdotes and make sense of it at scale as well as develop a methodology to account for bias it would greatly enhance the sense making ability of policy makers. That’s what I’ll outline below and show you a demonstration of an app to do just that.
The Idea
We build a large scale human sensor network in the Pacific based on stories using a technique called distributed ethnography. The aim is to creating a mass sensor network using citizens to inform politicians and policy makers. We use smartphones and apps to capture what’s know as micro narratives (aka anecdotes) which are then quantified by the storytellers themselves (to filter out biases from researchers, consultants etc).
The Science
The science behind the approach is based on the fact that people evolved to transmit knowledge and meaning through stories and anecdotes. Stories provide a much more richer source of information with multiple layers of meanings. The problem traditionally has been how to scale this. The solution to scaling the approach would be to equip people with electronic diaries. The techniques described below come from anthropology is based on the pioneering work by Dave Snowden who’s the Chief Scientist at the Cognitive Edge. You can watch his fascinating talk here.
How it works
We create a diary app to capture micro narratives in real time via diary entries. The diary entries are short anecdotes of a 140 characters or less of any problems that people encounter in places where they live and/or work.
The anectdote is then quantified and categorised by the person telling it (very important because we don’t want the biases of researchers, facilitators or consultants coming through on the feedback which is what typically happens).
The quantified anecdote then gets fed through to a centralised database and plotted on what’s known as a fitness landscape, providing near realtime unfiltered feedback into what people are saying on the ground.
Executives / Managers / Analysts are then able to visualise at scale people’s stories as they happen. The impacts of any legal, policy, operational, economic, biological or technological changes can be identified using weak signal detection methods as the stories begin to shift on the landscape maps in response to the changing environment. The stories can then be used to cross verify:
- Electronic sensor data
- – IMS data
- – Consultant / Advisors Reports
- – Policy decisions etc.
The prototype app in the screenshots above is called problems and proposals. It captures anecdotes about people’s problems and their proposed solutions to those problems. These can then be plotted on a fitness landscape map to see what’s going on