With all the hype surrounding big data it’s difficult to consider what other alternative approaches there may be to gain insight into what is happening within a society. In government the traditional approach of data collection focuses on survey data or data generated from government information management systems that have been aggregated into a centralised data warehouse. These are then analysed and used for informing policy. The problem is people lie on surveys all the time and operational information management systems only capture a small part of what’s really going on.
Anecdotes and stories are rich in insights. Unfortunately collecting anecdotes using traditional methods doesn’t scale well and people’s biases tend to get in the way during the collection and analysis process. If we could scale the capturing of anecdotes and make sense of it at scale as well as develop a methodology to account for bias it would greatly enhance the sense making ability of policy makers. That’s what I’ll outline below and show you a demonstration of an app to do just that.