Simon Rogers

Twitter has hired its first data editor. Simon Rogers, one of the leading practitioners of data journalism in the world, will join Twitter in May. He will be moving his family from London to San Francisco and applying his skills to telling data-driven stories using tweets. James Ball will replace him as the Guardian’s new data editor.

As a data editor, will Rogers keep editing and producing something that we’ll recognize as journalism? Will his work at Twitter be different than what Google Think or Facebook Stories delivers? Different in terms of how he tells stories with data? Or is the difference that Twitter has a lot more revenue coming in or sees data-driven storytelling as core to driving more business? (Rogers wouldn’t comment on those counts.)

The gig clearly has potential and Rogers clearly has demonstrable capacity. As he related to me today, in an interview, “what I’m good at is explaining data, simplifying it and making it accessible.”

That’s a critical set of skills in business, government, or media today. Data-driven journalists have to understand data sources, quality, context, and underlying biases. That’s equally true of Twitter.  Pew Research reminded us in 2013 that Twitter is not representative of everyone and is often at odds with public opinion.

Tweets aren’t always a reliable source to understand everything that happens in the world but it’s undeniable that useful insights can be found there. It has become a core component of the  set of digital tools and platforms that journalists apply in their work, connected to smartphone phones, pens, water bottles and notebooks. News frequently breaks on Twitter first and is shared by millions of users independently of any media organization. Journalists now use Twitter to apply a trade that’s well over a century old: gather and fact-check reports, add context, and find the truth of what’s happening. (Picking up the phone and going to location still matter, naturally.) The amount of misinformation on Twitter during major news events puts a high premium on the media debunking rumors and sharing accurate facts.

Will the primary difference in Rogers’ ability to find truth and meaning in the tweets be access to Twitter’s full firehose, developers, and processing power? His work will have to be judged on its own merits. Until he starts his new gig in May, the following interview offers more insight into why he joined Twitter and how he’s thinking about what he’ll be doing there.

Why leave the paper now?

Simon Rogers: I love the Guardian and have always wanted to work here. I grew up in a house where we read two papers: The Guardian during the week and the Observer on Sundays. I’ve had offers but this is the first job where it’s become a serious possibility.

There are a few reasons.

Firstly, Twitter is an amazing phenomenon. It’s changed every level of how we work as reporters. We really saw that during the “Reading the Riots” project. There we had 1.6 million riot-related tweets which Twitter gave us to analyze.

London Riots

That was important because politicians were agitating about the ‘role’ of Twitter during the disturbances. The work that our team did with academics at Manchester and the subsequent interactive produced by Alastair Dant and the interactive team here opened my eyes to the facts that:

• Twitter and the way it’s used tells us a lot about every aspect of life
• The data behind those tweets can really shine a light on the big stories of the moment
• If you can combine that data with brilliant developers you have a really powerful tool

Secondly, Twitter is an amazing place from what I’ve seen so far. There’s a real energy about the place and some brilliant people doing fascinating things. I love the idea of being part of that team.

Thirdly, I’ve been at the Guardian nearly 15 years. I am so comfortable and confident in what I do there that I need a new challenge. This all just came together at the right time.

As a data-driven journalist, you’ve had to understand data sources, quality, context, and underlying biases. How does that apply to Twitter?

Simon Rogers: Absolutely. Mark Twain said “a lie can be halfway around the world before the truth has got its boots on.” All social media encourages that.

I think the work we did with the riot tweets shows how the truth can catch up fast. What interested me about Boston was the way that people were tweeting calmness, if you like.

I think we’ve seen this with the Datablog in general: that people used to worry that the masses weren’t clever enough to understand the data that we were publishing. In fact, the community self-rights itself, correcting errors other readers or even ourselves had perpetrated. That’s really interesting to me.

What will you be able to do at Twitter with data that you couldn’t do at the Guardian data desk?

Simon Rogers: Just to be there, in the midst of that data will be amazing. I think it will make me better at what I do. And I hope I have something to offer them too.

Will you be using the same tools as you’ve been applying at the Guardian?

Simon Rogers: I’m looking forward to learning some new ones. I’m comfortable with what I know. It’s about time I became uncomfortable.

Twitter has some of the world’s best data scientists. What makes being a data editor different from being a data scientist?

Simon Rogers: I’m not the world’s best statistician. I’m not even very good at maths. I guess what I’ve been doing at The Guardian is acting as a human bridge between data that’s tricky to understand; and a wider audience that wants to understand it. Isn’t that what all data journalism is?

My take on being a data editor at the Guardian was that I used it as a way to make data more accessible – crucially the understanding of it. I need to understand it, to make it clear to others and I want to explain that data in ways that I can understand. Is that the difference between data editors and data scientists? I don’t know – I think a lot of these definitions are artificial anyway.

It’s like people getting data journalism and data visualization mixed up. I think they are probably different things and involve different processes, but in the end, does it matter anyway?

This interview was edited and condensed.

Crawford explored many of these same topics in our interview, which follows.

What research are you working on now, following up on your paper on big data?

Kate Crawford: I’m currently researching how big data practices are affecting different industries, from news to crisis recovery to urban design. This talk was based on that upcoming work, touching on questions of smartphones as sensors, on dealing with disasters (like Hurricane Sandy), and new epistemologies — or ways we understand knowledge — in an era of big data.

When “Six Provocations for Big Data” came out in 2011, we were critiquing the very early stages of big data and social media. In the two years since, the issues we raised are even more prominent.

I’m now looking beyond social media to a range of other areas where big data is raising questions of social justice and privacy. I’m also editing a special issue on critiques of big data, which will be coming out later this year in the International Journal of Communications.

As more nonprofits and governments look to data analysis in governing or services, what do they need to think about and avoid?

Kate Crawford: Governments have a responsibility to serve all citizens, so it’s important that big data doesn’t become a proxy for “data about everyone.” There are two problems here: first is the question of who is visible and who isn’t represented; the second is privacy, or what I call “privacy practices” — because privacy means different things depending on where and who you are.

For example, the Streetbump app is brilliant. What city wouldn’t want to passively draw on data from all those smartphones out there, a constantly moving network of sensors? But, as we know, there are significant percentages of Americans who don’t have smartphones, particularly older citizens and those with lower disposable incomes. What happens to their neighborhoods if they generate no data? They fall off the map. To be invisible when governments make resource decisions is dangerous.

Then, of course, there’s the whole issue of people signing up to be passively tracked wherever they go. People may happily opt into it, but we’d want to be very careful about who gets that data, and how it is protected over the long term — not just five years, but 50 years and beyond. Governments might be tempted to use that data for other purposes, even civic ones, and this has significant implications for privacy and the expectations citizens have for the use of their data.

Where else could such biases apply?

Kate Crawford: There are many areas where big data bias is a problem from a social equity perspective. One of the key ones at the moment is law enforcement. I’m concerned by some of the work that seeks to “profile” areas, and even people, as likely to be involved in crime. It’s called “predictive policing” (more here). We’ve already seen some problematic outcomes when profiling was introduced for plane travel. Now, imagine what happens if you or your neighborhood falls on the wrong side of a predictive model. How do you even begin to correct the record? Which algorithm do you appeal to?

What are the things, as David Brooks listed recently, that big data can’t do?

Kate Crawford: There are lots of things that big data can’t do. It’s useful to consider the history of knowledge, and then imagine what it would look like if we only used one set of tools, one methodology for getting answers.

This is why I find people like Gabriel Tarde so interesting — he was grappling with ideas of method, big data and small data, back in the late 1800s.

He reminds us of what we can lose sight of when we go up orders of magnitude and try to leave small-scale data behind — like interviewing people, or observing communities, or running limited experiments. Context is key, and it is much easier to be attentive to context when we are surrounded by it. When context is dissolved into so many aggregated datasets, we can start getting mistaken impressions.

When Google Flu Analytics mistakenly predicted that 11% of the US had flu this year, that points to how relying on a big data signal alone may give us an exaggerated or distorted result (in that case, more than double the actual figure, which was between 4.5-4.8%). Now, imagine how much worse it would be if that data was all that health agencies had to work with.

I’m really interested in how we might best combine computational social science with traditional qualitative and ethnographic methods. With a range of tools and perspectives, we’re much more likely to get a three-dimensional view of a problem and be less prone to serious error. This goes beyond tacking on a few focus groups to big datasets, but conjoining deep, ethnographically-informed research with rich data sources.

What can the history of statistics in social science tell us about correlation vs causation? Does big data change that dynamic?

Kate Crawford: This is a gigantic question, and one that could be its own talk! With big datasets, it’s very tempting for researchers to engage in apophenia — seeing patterns where none actually exist — because massive quantities of data can point to a range of correlative possibilities.

For example, David Leinweber showed back in 2007 that data mining techniques could show a strong but spurious correlation between the changes in the S&P 500 stock index and butter production in Bangladesh. There’s
another great correlation between the use of Facebook and the rise of the Greek debt crisis.

With big data techniques, some people argue you can get much closer to being able to predict causal relations. But even here, big data tends to need several steps of preparation (data “cleaning” and pre-processing) and several steps in interpretation (deciding which of many analyses shows a positive result versus a null-result).

Basically, humans are still in the mix, and thus it’s very hard to escape false positives, strained correlations and cognitive bias.

This interview originally appeared on O’Reilly Radar.

That focus caught my attention, given that my colleague, Alistair Croll, had published several posts on Radar looking at the ethics around big data. The increasing use of data mining and algorithms by government to make decisions based upon pattern recognition and assumptions regarding future actions is a trend worth watching. Will guaranteeing government data quality become mission-critical, once high profile mistakes are made? Any assessment of the perils and promise of a data-driven society will have to include a reasoned examination of the growing power of these applied algorithms to the markets and the lives of our fellow citizens.

Given some of those concerns, I called Goldsmith up this winter to learn more about what he meant.

Our interview, lightly edited for content and clarity, follows.

When you say “preemptive government,” what do you mean?

Stephen Goldsmith: I’m thinking about the intersection of trends here. One is what we might talk about as big data and data analytics. Inside, government has massive amounts of information located in all sorts of different places that if one looked at analytically, we could figure out which restaurants are most likely to have problems; which contractors are most likely to build bad buildings and the like.

For the first time, through the combination of digital processes, mobile tools and big data analytics, government can make preemptive solutions. Government generally responds to problems and then measures its performance by the number of activities it conducts, as contrasted to the problems it solves. New York City and Chicago have begun to take the lead in this area in specific places. When I was in New York City, we were trying to figure out how to set up a data analytics center. New York has started to do some of that, so that we can predict where the next event’s going to occur and then solve it. That eventually needs to be merged with community sentiment mining, but it’s a slightly different issue.

What substantive examples exist of this kind of approach making a difference?

Stephen Goldsmith: We are now operating a mayoral performance analytics initiative at the Kennedy School, trying to create energy around the solutions. We are featuring people who are doing it well, bringing folks together.

New York City, through a fellow named Mike Flowers, has begun to solve specific problems in building violations and restaurant inspections. He’s overcoming the obstacles where all of the legacy CIOs say they can’t share data. He’s showing that you can.

Chicago is just doing remarkable stuff in a range of areas, from land use planning to crime control, like deciding how to intervene earlier in solving crimes.

Indiana has begun working on child welfare using analytics to figure out best outcomes for children in tough circumstances. They’re using analytics to measure which providers are best for which young adults that are in trouble, what type of kid is most successful with what type of mental health treatment, drug treatment , mentoring or the like.

I think these are all just scratch the surface. They need to be highlighted so that city and state leaders can understand how they can have dramatically better returns on their investments by identifying these issues in advance.

Who else is sharing and consuming data to do predictive data analytics in the way that you’re describing?

Stephen Goldsmith: A lot of well-known staff programs, like ComStat or CityStat, do a really good job of measuring activities. When combined with analytics, they’ll be able to move into managing outcomes more effectively. There are a lot of folks like San Francisco beginning to look at these issues, but I think really, New York City and Chicago are in the lead.

Based upon their example, what kinds of budget, staffing and policy decisions would other cities need to make to realize similar returns?

Stephen Goldsmith: The most restrictive element in government today is the no-longer-accurate impression that legacy data can’t be easily integrated. Every agency has a CIO who often believes it’s his or her job to protect that data. I’m not talking about privacy; I’m just talking about data integrity. We know that there’s a range of tools that will allow relatively easy integration and data mining.

Another lesson is that this really needs to be driven by the mayor or the governor. The answers to problems come from picking up data across agencies, not just managing the data inside your agency. Without city hall or gubernatorial leadership, it’s very difficult to drive data analytics.

What about the risks of ‘preemptive government’ leading to false positives or worse?

Stephen Goldsmith: There is a risk, but let me talk about it in the following way: Government can no longer afford to operate the way it operates. You cannot afford to regulate every business as if it’s equally bad or equally good. Every restaurant is not equally good or equally bad. Every contractor’s not bad or good. There are bad guys and good guys, and good performers and bad performers. There are families that need help and families that don’t need help. We need to allocate our resources most effectively to create solutions. That means we need to look at which solutions work for which problems.

What do we know about which contractors have a history of being bad? I don’t mean “bad” like just how they build — I mean have they paid their taxes right, do they discriminate in the marketplace, whatever those factors are in order to target our resources.

That means that when Flowers did this in New York, we got several hundred buildings that were the most likely to burn down. We knew that from analytics. We’re going to go out and mitigate those buildings. Could we make a mistake and say that ten of those 300 buildings really aren’t that bad? Absolutely, but it’s a much better targeting of resources and it’s the only way government can afford to effectively operate.

There are other issues, too, like personalization, where we have a lot of privacy issues, and “opt-in” and “opt out” where people may want a personal relationship with their government. That’s a little different than predictive analytics, but it raises privacy issues.

Then we have a fascinating question, one that social work communities and criminal justice communities worry about, which is, “Okay, you can predict the likelihood that somebody can be hurt, or that somebody will commit a crime and adjust resources accordingly – but we better be pretty careful because it raises a lot of ethics questions and profiling questions.”

My short answer is that these are important, legitimate questions. We can’t ignore them, but if we continue to do business the way we do it has more negative tradeoffs than not.

Speaking of personalization and privacy, has mining social media for situational awareness during national disasters or other times of crisis become a part of the toolkit for cities?

Stephen Goldsmith: The conversation we’ve had has been about how to use enterprise data to make better decisions, right? That’s basically going to open up a lot of insight, but that model is pretty arrogant. It basically ignores crowd sourcing. It assumes that really smart people in government with a lot of data will make better solutions. But we know that people in communities can co-produce information together. They can crowd source solutions.

In New York City, we actually had some experience with this. One thread was the work that Rachel [Haot] was doing to communicate, but we were also using social media on the operations side. I think we’re barely getting started on how to mine community sentiment, how to integrate that with 311 data for better solutions, how to prioritize information where people have problems, and how to anticipate the problems early.

You may know that Indianapolis, in the 2012 Super Bowl, had a group of college students and a couple of local providers looking at Twitter conversations in order to intervene earlier. They were geotagged by name and curated to figure out where there was a crime problem, where somebody needed parking, where they were looking for tickets and where there’s too much trash on the ground. It didn’t require them to call government. Government was watching the conversation, participating in it and solving the problem.

I think that where we are has lots of potential and a little bit immature. The work now is to incorporate the community sentiment into the analytics and the mobile tools.

This change has been a long time coming, although more needs to be done to fully open the People’s House to the People. In April 2011, Speaker of the House John Boehner and Majority Leader Eric Cantor sent a letter to the House Clerk regarding legislative data release. In September 2011, a live XML feed for the House floor went online. In September 2012, Congress launched a beta version of Congress.gov but failed to open the data.

“Thanks to GPO, all House bills for this Congress will be available in one XML file that can be downloaded by anyone,” said Speaker Boehner, in a joint statement with Majority Leader Cantor at Speaker.gov. “This is a win for every American who believes in open government. Making legislative data easier to use for third parties, developers, and anyone interested in how Congress is tackling current challenges is a priority for House leaders. We’re going to keep working to make the legislative process more transparent and to better connect lawmakers with the people we serve.”

In a post on Tuesday at Speaker.gov, Don Seymour, digital communications director for the Speaker of the House, detailed the progress made during the 112th Congress:

This project is the first of several to be rolled out in the 113th Congress that were coordinated or initiated by the Legislative Branch Bulk Data Task Force. The task force was created to expedite the process of providing bulk access to legislative information and to increase transparency for the American people. It includes the House Clerk, legislative branch agencies such as the Government Printing Office and Library of Congress, representatives from House leadership and key committees, and the House Chief Administrative Officer.

Open government is and has been a priority for House leaders. In fact, the Clerk began offering real-time updates on House floor proceedings in XML back in 2011. The feed of real-time information complemented HouseLive.gov, a new video streaming feature they set up for desktop and mobile devices. The House also began utilizing new low-cost video conferencing tools, streaming committee hearings online, working with developers and transparency advocates, and more.

As Speaker Boehner said, this is good news for every American. Despite the abysmal public perception of Congress, genuine institutional changes in the House of Representatives driven by the GOP embracing innovation and transparency have been happening over the last three years. Open government in the House also enjoys a rare status in Washington these days: bipartisan comity. These improvements built upon bipartisan progress made while Representative Nancy Pelosi held the Speaker’s gavel, including putting committee hearings online, putting expenditures and lobbying disclosure online, and changed the franking rules to allow for the use of new media, like YouTube, Facebook and Twitter.

Democratic Whip Steny Hoyer praised the GPO and House Clerk for providing bulk access to House bills in XML:

“The actions this week by the GPO and the House Clerk are significant steps towards making the legislative branch more open and transparent,” said Whip Hoyer. ”Congress has a duty to share information about legislation being developed and deliberated, and this new effort will allow the public to follow and engage with Congress in innovative new ways.  I commend GPO and the House Clerk for their actions, and hope that other legislative branch entities like the Library of Congress and the Senate will follow suit by including additional legislative information that is already publicly available, yet not accessible, on Thomas.gov.”

A more open road ahead?

As Tim O’Reilly observed in 2011, the current leadership of the House do seem to be doing a better job on transparency and open data than their predecessors. Jim Harper’s analysis of the government’s data publication process substantiates that progress. Writing at the Cato Institute, where he is the director of information policy studies, Harper praised the House for this step forward:

I believe the public has an Internet-fueled expectation that they should understand what happens in Congress. It’s one explanation for rock-bottom esteem for government in opinion polls. Access to good data help produce better public understanding of what goes on in Washington and also, I believe, more felicitous policy outcomes — not only reduced demand for government, but better administered government in the areas the public wants it.”

Harper also offered some constructive criticism for improvement:

That I’ve been able to find, the XML is not well documented. What each of the technical codes means is understood by several people in Washington’s transparency community, but the idea is to make it available very broadly, so the documentation should be very strong. The information at xml.house.gov should be updated, tightened up, and made easily available to the people gathering bill data on FDsys.

The XML data structures put in bills are limited in terms of what they convey. There is rudimentary information about who introduced and cosponsored bills, what committees they were referred to, and other procedural information. That’s good. But the effects of bills—on agencies, existing law, programs, places—this is not available in machine-readable code. That would be great.

Josh Tauberer, the author of Open Government Data, added some caveats on the House’s move to bulk bill XML. Tauberer is the civic hacker behind Govtrack.us, which has been scraping and making legislative data more open for years. He also contributed a chapter to Open Government in 2010.

In his comments, excerpted below, he notes that “there’s no new data here, and thus not the data that the bulk legislative data advocates have been asking for.” In other words, this is evolutionary change, not revolutionary change.

What we’re seeing with the bills bulk data project is how the wave of culture change is moving through government. Over the last two years the House Republican leadership has embraced open government in many ways (my 112th Congress recap | the new House floor feed). With this bills XML project, we’re seeing more legislative support agencies being involved in how the House does open government.

This isn’t a technical feat by any means, but it is a cultural feat. The House and GPO worked together to institutionalize a new way for the House to publish bulk data.

Because of the way Data.gov is managed in the executive branch, we’ve become accustomed to big announcements. The bills bulk data project and the other recent projects show that the House is taking a different approach, an incremental approach, to open government data: publish early and often, gather feedback, then go on to bigger projects. This is something open government advocates have been asking for.

Daniel Schuman, the legislative counsel of the Sunlight Foundation, echoed Harper and Tauberer’s balanced praise for improved access, including a recent history of progress and suggestions for what remains to do:

“Ultimately, this path should take us past the point where all legislative information published on THOMAS (or its successor Congress.gov) is available online, in real time, as structured data that is capable of being downloaded in bulk. The most requested data is legislative status information, which is held by the Library of Congress and still is not available today as structured data, bulk or otherwise. That includes when the bill was introduced, who co-sponsored it, a summary of the legislation, and so on.

Status information is prepared by the Library of Congress, which has been historically recalcitrant to make this information available to the public in any other formats besides a series of web pages. But we know based on a March 2008 memo that the hurdle here is political will, not technology. That’s why this new announcement is encouraging. The task force is starting to crack open the vault. Let’s hope that the Senate and the Library of Congress are coming to share the House’s enthusiasm for transparency.”

As we head further into 2013, here’s hoping that the entire Congress takes more steps to make the content and status of proposed laws more accessible to the hundreds of millions of people its members represent around the country. As more progress is made toward freeing the data, it will enable the nation to track the progress of legislation in real time in the media and civic entrepreneurs to build better interfaces for understanding the proposal before Congress.

Related Stories:

• Congress launches Congress.gov in beta, doesn’t open the data
• Can the People’s House become a platform for the People?
• Bulk data downloads: a breakthrough for government transparency

This post is part of our series investigating open data. An earlier version of this post appeared on the O’Reilly Radar Tumblr.

While the story that Flowers told is a compelling one, the role of big data in the public sector was in evidence in several other sessions at the conference. Here are three more ways that big data is relevant to the public sector that stood out from my trip to New York City.

There’s justice in the data

“Data and technology will change how criminal justice works in America,” said Anne Milgram. Milgram, the former attorney general of New Jersey, is now the vice president of criminal justice at the Arnold Foundation, where she’s working on building better risk assessment tools for courts, including a pilot in Manhattan.

“You can’t fix a problem you don’t know you don’t have,” said Migram. “We need to understand when and where violent crime is happening.” Most institutional actors, she asserted, are not using tech effectively. They need access to information other actors have and face cultural hurdles to data sharing.

When she asked a judge which criminals his court was actually putting in jails, he said it was the most dangerous and violent offenders. The data shows that the majority of people being incarcerated are low-level, non-violent offenders, said Milgram. The judges she talked with, however, couldn’t believe that outcome, given that their intention in sentencing was otherwise. What she showed them also highlighted an important system error: highly violent people were ending up out on the streets, not incarcerated.

Applying a data-driven lens to improve the efficiency and effectiveness of the criminal justice system has a potentially huge incentive: improving the bottom lines of state budgets around the country. According to Milgram, corrections ranks among the highest expenditures in state budgets. (As of 2012, corrections was No. 4 in California, for instance.) It costs $45,000 p er year in New Jersey to incarcerate someone, she said, and the cost of local systems nationally is estimated by the Department of Justice at more than $130 billion dollars a year. The costs of incarceration just prior to trial alone is $9 billion.

Milgram spoke about “moneyballing” criminal justice earlier this year at the Code for America Summit. Video of that talk is embedded below:

http://www.youtube.com/watch?v=_XN5RaInhD64

The returns for better applying technology in criminal justice extend far beyond reducing crime or costs to something that government officials are sworn to uphold: justice.

“It’s too expensive” is no longer an excuse

While Milgram is focused on getting a pilot up and running in New York City, Chicago’s data-driven approach to open government has been underway since Mayor Rahm Emanuel was elected in February 2011. At Strata + Hadoop World, Q Ethan McCallum, the author of the newly published Bad Data Handbook, joined Chicago chief information officer and chief data officer Brett Goldstein to talk about text mining and civic engagement.

.@chicagocdo: how could we gather data to complement existing inputs? #strataconf cc @oreillymedia twitter.com/digiphile/stat…

— Alex Howard (@digiphile) October 25, 2012

There’s much that can and should be said about what Chicago has accomplished with data in the past years, from the launch of Open 311 in Chicago, which creates public data infrastructure for civic apps like this flu shot finder, to the city’s embrace of its civic hacking community, which creates useful apps like Was my car towed?.

What jumped out for me at Strata however, wasn’t the quality of Chicago’s data, how it has consumed it internally or the ecosystem of nonprofits, developers and startups that the city has worked to create around it. Instead, it was the importance of political leadership, results and the pursuit of internal capacity, not just the number of open datasets published online. “Mayor Emanuel wanted Chicago to be the standard for open data, analytics and prediction,” said Goldstein.

Chicago is a political place, he allowed with a laugh at the press conference at the Strata Conference, but the mandate from the mayor is to do it right. “We’re not creating pretty pictures,” he said. “We’re building a solid foundation.”

Goldstein, who added “CIO” to his title earlier this year, has spent a lot of time working on architecture since. The question has been how to bring the data together. Chicago has taken an open source approach to doing so, trying to make it universally easier to use and standardizing across the enterprise. He’s also been working with the community: Chicago is not only making municipal data available, but it’s also sponsoring R classes to help people understand how to put it to good use.

Goldstein’s team is dealing with short-term deliverables. Traditionally in IT projects, cities send out a request for proposals and then spend money on a big box solution — and that can take years. “My mandate is to give our residents every value for their taxes,” he said at the press conference. “By having an agile team, we could stand up in weeks what would take months or years. By having an agile mentality, you can get a rapid return. There are classic IT things that should go to RFP — like an ERP system — but why not build other things?”

For Goldstein, “showing that you can use R in a municipal government is a game changer.” As a result of his team’s work in Chicago, “it’s too expensive” to use big data in the public sector is no longer an excuse not to so so.

To help other cities use Hadoop, MongoDb and R, Goldstein is collaborating with Michael Flowers on G-Analytics, a group focused on building capacity in this nascent field of urban data analytics around the United States and beyond.

“I have a close relationship with Flowers,” said Goldstein, at the press conference. “We trade code. If I write something, I want someone to be able to download and use it.”

Balance public good with human rights protection

In August, my colleague Alistair Croll provocatively wrote that big data is our generation’s civil rights issue.

Robert Kirkpatrick, director of U.N. Global Pulse, broadened his concerns when he delivered remarks at Strata: “Big data is a human rights issue,” he said. “We must never analyze personally identifiable information, never analyze confidential data, and never seek to re-identify data.”

He described three big opportunities in big data for the United Nations and governments in general:

1. better early warning, to enable faster response
2. real-time awareness, to know what’s happening on the ground now
3. real-time feedback, perhaps “most important,” to can see what’s not happening versus what was intended

You can view Kirkpatrick’s presentation at Slideshare. In his talk on big data and development, Kirkpatrick appealed to a packed room for help on the challenges and big questions that U.N. Global Pulse faces, a need he articulated in an op-ed in the first issue of “Big Data”, a peer-reviewed journal that launched at Strata:

How does the United Nations gain access to the data it needs in order to do the research necessary to answer those other questions? We believe the answer to the latter, crucial question is what we call “data philanthropy,” where data-rich companies donate to research projects. For example, I have been spending a lot of time lately talking to private sector companies about how they can safely and anonymously share with Global Pulse some of what they know about customers, to help give us a badly needed leg-up in our quest to better protect the vulnerable. The companies that are most open to the message are the ones that recognize that data philanthropy is not charity. These companies know that population well-being is key to the growth and continuity of business. No company wants to invest in a promising emerging market only to find out it is being threatened by a food crisis that could leave customers unable to afford products and services. And it would be sadly ironic if it turned out that expert analysis of patterns in a company’s own data could have revealed that people were headed for trouble while there was still time to act.

Kirkpatrick hopes that the data science community and corporations will donate their time, expertise and data for the public good, enabling better crisis tracking using crowdsourced information.

He’s particularly interested in the potential of social and mobile data for their predictive value. For instance, said Kirkpatrick, Twitter data accurately predicted the cholera outbreak in Haiti two weeks earlier than official records. Mobile networks can also act as drought sensors in the Sahel Desert in Africa.

The tools keep improving, too: the U.S. Geological Survey’s (USGS) Twitter earthquake detector (@USGSted) has a less than 10% false positive rate.

Such systems complement existing systems, not replace them, said Kirkpatrick. After an alert, the USGS can wake up seismologists in that part of the world to go check their data centers. By doing so, they can reduce the time to detect the epicenter of a quake from 20 minutes to four minutes.

This kind of data analysis has considerable potential for more than natural disasters or epidemics. In Jakarta, the “tweetingest city on Earth,” with more than 9 million tweets sent daily, UN Global Pulse analyzed 14 million tweets during a period of inquiry and found that people talk differently about food when they’re not getting it, leading to potential predictive value for food security. Social media can be used to predict food price inflation, though Kirkpatrick warned that signals in the data are culturally contextual. For instance, for every 5,000 more tweets from Indonesians about eggs, UN Global Pulse found a 2-3% decrease in food consumer price index.
signals in data.

UN Global Pulse is now working on building its networks around the world, partnering with government, the private sectors, agencies and academia on doing research. When they find something that works, they turn to the open source paradigm to help local populations tap into distributed intelligence.

We expect to see more map layers of proxy indicators, said Kirkpatrick. “The signals are getting stronger, with an increase in the volume of relevant conversations,” he said. “The social media food index and food price index started matching up in October 2011. We believe more people using social media to talk about basic needs is leading to a correlation with official statistics. We think there’s a huge opportunity to have socioeconomic weather stations that show trends in poverty and food in every community in the world.”

Strata Conference Santa Clara — Strata Conference Santa Clara, being held Feb. 26-28, 2013 in California, gives you the skills, tools, and technologies you need to make data work today.

Save 20% on registration with the code STRATA20

After he correctly called all 50 states, Silver is being celebrated as the “king of the quants” by CNET and the “nerdy Chuck Norris” by Wired. The combined success of statistical models from Silver, TPM PollTracker, HuffPost Pollster, RealClearPolitics Average, and the Princeton Election Consortium all make traditional “horse race journalism” that uses insider information from the campaign trail to explain what’s really going on look a bit, well, antiquated. With the rise of political data science, the Guardian even went so far as to say that big data may sound the death knell for punditry.

This election season should serve, in general, as a wake-up call for data-illiterate journalists. It was certainly a triumph of logic over punditry. At this point, it’s fair to “predict” that Silver’s reputation and the role of data analysis will continue to endure, long after 2012.

“As of this writing, the only thing that’s ‘razor-thin’ or ‘too close to call’ is the gap between the consensus poll forecast and the result” — Randall Munroe

The data campaign

The other big tech story to emerge from the electoral fray, however, is the how the campaigns themselves used technology. What social media was to 2008, data-driven campaigning was in 2012. In the wake of this election, people who understand math, programming and data science will be in even higher demand as a strategic advantage in campaigns, from getting out the vote to targeting and persuading voters.

For political scientists and campaign staff, the story of the quants and data crunchers who helped President Obama win will be pored over and analyzed for years to come. For those wondering how the first big data election played out, Sarah Lai Stirland’s analysis of how Obama’s digital infrastructure helped him win re-election is a must-read, as is Nick Judd’s breakdown of former Massachusetts governor Mitt Romney’s digital campaign. The Obama campaign found voters in battleground states that their opponents apparently didn’t know existed. The exit polls suggest that finding and turning out the winning coalition of young people, minorities and women was critical — and data-driven campaigning clearly played a role.

For added insight on the role of data in this campaign, watch O’Reilly Media’s special online conference on big data and elections, from earlier this year. (It’s still quite relevant.) The archive is embedded below:

For more resources and analysis of the growing role of big data in elections and politics, read on.

If you’re new to the topic, this list of videos and articles should be useful.

In October, I joined other journalists and digital media experts to discuss Al Jazeera’s “The Stream” for a show on “data mining the U.S. Election.”

The PBS News Hour collaborated with Frontline to produce a feature on this year’s digital campaigns, including an interactive on targeting the electorate.

For a look back at some of the best news and commentary on big data and politics, read the following links:

Patrick Ruffini: “Goodbye, polling. Hello, Big Data.”

Tech President: “Election 2012: It’s Not Facebook. It’s The Data, Stupid.”

Slate: “Project Dreamcatcher: How cutting-edge text analytics can help the Obama campaign determine voters’ hopes and fears.”

Guardian: “Obama, Facebook and the power of friendship: the 2012 data election”

Campaigns & Elections: “Big Data Is A Big Factor in 2012.”

Guardian: “Obama campaign leaves Mitt Romney trailing as focus shifts to November”

Politico: “Obama’s Data Advantage”

Guardian: “Mitt Romney’s campaign closing gap on Obama in digital election race”

Tech President: “Zac Moffatt Talks Digital Strategy”

New York Times: “Campaigns Mine Personal Lives to Get Out Vote”

Slate: “Are You Going To Vote? Do You Promise?”

Slate: “Obama does it better.”

Strata Conference Santa Clara — Strata Conference Santa Clara, being held Feb. 26-28, 2013 in California, gives you the skills, tools, and technologies you need to make data work today.

Save 20% on registration with the code STRATA20

[Photo Credit: NASA}

In that context, it’s key to understand that it’s government weather data, gathered and shared from satellites high above the Earth, that’s being used by a huge number of infomediaries to forecast, predict and instruct people about what to expect and what to do. In perhaps the most impressive mashup of social and government data now online, an interactive Google Crisis Map for Hurricane Sandy pictured below predicts the future of the ‘Frankenstorm’ in real-time, including a NYC-specific version.

If you’re looking for a great example of public data for public good, these maps like the Weather Underground’s interactive are a canonical example of what’s possible.

Matt Lira, the director of digital for the Majority Leader in the U.S. House of Representatives, made an important, clear connection between open government, weather data and the a gorgeous wind visualization that has been getting passed around today.

This dynamic wind map is an example of how open government data can be utilized in effective & creative ways: hint.fm/wind/ #opengov

— Matt Lira (@MattLira) October 29, 2012

In the context of the utility of weather data, it will be interesting to see if Congress takes action to fund weather satellite replacements.)

In New York City, as the city’s websites faced heavy demand when residents went to its hurricane evacuation finder on Sunday, residents could also go and consult WYNC’s beautiful evacuation map. (Civiguard also activated an instant evacuation zone checker for smartphones and modern browsers.) WNYC data news editor John Keefe is responsible for the map below that puts the city’s open government data in action.

By releasing open data for uses in these apps, New York City and the U.S. federal government are acting as a platform for public media, civic entrepreneurs and nonprofits to enable people to help themselves and one another at a crucial time. When natural disasters loom, public data feeds can become critical infrastructure.

For one more example of how this looks in practice, look at WNYC’s storm surge map for New York and New Jersey.

If you’re a coder interested in working with the tech community, MIT Media Lab Director Joi Ito is helping to coordinate #HurricaneHackers working on projects and resources for Hurricane Sandy. The group has made a timeline of events, a list of livestreams, along with aggregating links to official data and social streams, like Instacane, a site that aggregates Instagram images about the hurricane.

Stay safe, keep informed

Hurricane Sandy has meteorologists scared, and for good reason. The federal government is providing information on Hurricane Sandy at Hurricanes.govs and NOAA and sharing news and advisories in real-time on the radio, television, mobile devices and online using social media channels like @FEMA.

As the storm comes in, FEMA recommends m.fema.gov to mobile users and ready.gov for desktops. The Wall Street Journal and Reuters are both live-blogging the news. Like WNYC, the Associated Press Reuters used weather data to populate interactive Hurricane Tracker maps.

People in the path of the storm can download smartphone apps from the RedCross: http://rdcrss.org/R4gjDV and FEMA on Android: http://bit.ly/ToDgqB iOS: http://bit.ly/sNZNJI or BlackBerry: http://bit.ly/wUiqHL

If you do not have a smartphone, save 43362 (4FEMA) to your mobile phone and charge it up. If, after #Sandy, you cannot return home and have immediate housing needs, text SHELTER + ZIP code to 43362.

UPDATE: Bob Rudis (@hrbrmstr wrote in to share his R code for live tracking at Github, which he blogged about here. David Smith has already put the code to use in tracking Hurricane Sandy with open data and R at Revolution Analytics.

If you have more examples of data, maps, apps, code or services relevant to the hurricane or its aftermath, please share them in the comments or write to alex@oreilly.com. And if you’re in the path of the storm, please stay safe.

The process of converting paper forms into structured data still hasn’t been significantly disrupted by rapid growth of the Internet, distributed computing and mobile devices. Fields that range from research science to medicine to law to education to consumer finance to government all need better, cheaper bridges from the analog to the digital sphere.

Enter Captricity. The startup, which was co-founded by Jeff J. Lin and Kuang Chen, has its roots in the fieldwork on rural health Chen did as part of his PhD program.

“I was looking at the information systems that were available to these low-resource organizations,” Chen said in a recent phone interview. “I saw that they’re very much bound in paper. There’s actually a lot of efforts to modernize the infrastructure and put in mobile phones. Now that there’s mobile connectivity, you can run a health clinic on solar panels and long distance Wi-Fi. At the end of the day, however, business processes are still on paper because they had to be essentially fail-proof. Technology fails all the time. From that perspective, paper is going to stick around for a very long time. If we’re really going to tackle the challenge of the availability of data, we shouldn’t necessarily be trying to change the technology infrastructure first — bringing mobile phones and iPads to where there’s paper — but really to start with solving the paper problem.”

When Chen saw that data entry was a chokepoint for digitizing health indicators, he started working on developing a better, cheaper way to ingest data on forms.

Captricity’s approach to that paper problem is fascinating, as I saw in an online demo of the technology. They’ve found a way to use the Internet to quickly and cheaply digitize handwritten forms into structured data.

“Our target user is an office admin who’s really good at, let’s say, Excel and doing mail merges in Word,” said Chen. “We make sure that person or a school teacher or an existing database administrator can just go and do it themselves.”

The process looks relatively simple for the end user, at least on the surface. Scan a form and then outline fields on it to create a template. When you subsequently scan a batch of forms, the system breaks up the designated fields into images that crowdsourced workers can identify online. The fields from each form are then output as structured data, exportable as a CSV file or into Google documents. Each digitized document represents a row. The provenance of the data is preserved, with the original image connected to a given cell. Most jobs take 10 to 20 minutes. (The demo I saw took 11 minutes or so.)

“Under the covers, the approach is to take a page and cut it up into little pieces,” said Chen. “We reorganize each little piece and then we send it out to workers on the Internet. They don’t see the context of anything else. They give us a set of answers that we make sure are correct by essentially doing it in triplicate. Then we take a small set of these triple-verified values and build essentially a machine-learning vision engine that predicts the value for that box. ”

Captricity has effectively positioned itself as a “digitization-as-a-service” provider. Instead of buying equipment, organizations can pay as they go.

“It’s a place on the Internet, like a tap, that can turn on digitization,” Chen said. “You pay for exactly how much you use. You don’t have to spend $300,000 and buy a very high-speed scanner and self-service on premise to get what you need to get done. What we have runs on Amazon AWS and uses its Elastic Computing Cloud to crowdsource labor from Mechanical Turk. We are in talks to use other more private and specialized crowds as well.”

The retail cost to use Captricity is about $0.20 per page, after the first 25 pages, which are free. Larger volume customers will pay a bit more, based on the type of data and the volume of data. By comparison, OCR-only solutions are in the ballpark of $0.01 to $0.03 per page, said Chen, but require an expensive software license and equipment.

After the recent launch of an iOS app, Captricity went mobile. It’s now possible to scan using an iPhone, iPad or iPod Touch, integrate with an existing template and then sync the information to a Dropbox account. An Android app is in development and will be launched early next year. They’ve also rolled out integration with Salesforce in the mobile app, along with Box.com, and Constant Contact.

The startup, however, holds the potential to be something bigger than just a better mobile digitization provider: Captricity’s application programming interface (API) will help tap into their digitized data more easily.

“We have an API that has been in private beta for about a month-and-a-half now,” said Chen. “We extract away complexity and allow an application developer to just say, ‘We have forms. Let’s enter in the data. Go.’ And you’re up and running in a day.”

And now, developers can share that information. On Tuesday, Captricity announced a new open data platform that will publicly share digitized, structured datasets. The first dataset that they’ve published comes from the U.S. Census, as a demonstration of the concept.

This technology is of particular interest to health care, which is full of forms. That’s why Chen will join representatives from West Health and ElationEMR at this month’s StrataRx Conference to talk about the untapped potential of structured data on paper. The ability for users to only scan certain parts of forms — which would enable fields containing personally identifiable information to be left out — could be a key component of health data infrastructure. There will be special challenges, given HIPAA rules that govern patient data, but selective digitization might be a way to address them.

“We have fairly strict and cautious controls over this process, so it’s not automatic, but rather handheld, to make sure that only data intended to be public becomes public,” said Chen.

The idea of breaking up jobs for many people to work on online has been around for a while, of course, with crowdsourcing breaking big in the middle of the last decade. The innovation here is applying it to something that machines still can’t do as well as humans — reading handwriting — and then solving a problem that has been a real chokepoint for digitizing data. Even if this particular startup doesn’t end up taking off, they’ve approached a critical need in a way that has implications for multiple industries. For the data economy to grow, it will need more feedstock.

“This roughly falls under this flag of human-guided machine learning,” said Chen. “I think that with the advent of crowdsourcing, this is going to be a really powerful force in improving what machine learning algorithms can do and the types of problems it can solve. In six months — plus the time I spent on my PhD — we built a production system to do OCR that is better than any OCR system out there today, by probably an order of magnitude. I have the most respect for the computer vision researchers that came up with those OCR algorithms. It’s just that we solved a different problem than they did. This is the approach that we’re taking to solve the other problems of this domain as well.”

Strata Conference + Hadoop World — The O’Reilly Strata Conference, being held Oct. 23-25 in New York City, explores the changes brought to technology and business by big data, data science, and pervasive computing. This year, Strata has joined forces with Hadoop World.

Save 20% on registration with the code RADAR20

Today, the founder and CEO of DataMarket told the audience of the first White House Energy Datapalooza that his company would make energy data more discoverable and usable. In doing so, Datamarket will be be tapping into an emerging data economy of businesses using open government data.

“We are honored to have been invited to take part in this fantastic initiative,” said Gislason in a prepared statement. “At DataMarket we focus on doing one thing well: aggregating vast amounts of heterogeneous data to help business users with their planning and decision-making. Our new energy portal applies this know-how to the US government’s energy data, for the first time enabling these valuable resources to be searched, visualized and shared through one gateway and in combination with other domestic and worldwide open data sources.”

Energy.datamarket.com, which won’t go live officially until mid-October, will offer search for 10 thousand data sets, 2 million time series and 50 million energy facts. DataMarket.com is based upon data from thirteen different data providers including the U.S. Department of Energy’s Energy Information Agency (EIA), Oak Ridge National Laboratory, Energy Efficiency and Renewable Energy program, National Renewable Energy Laboratory, the Environmental Protection Agency (EPA), the Bureau of Transportation Statistics, the World Bank and United Nations.

Last week, I interviewed Gislason about his company and why they’re focusing on energy data.

What itch were you scratching when you founded DataMarket?

Gislason: When we wanted the best data to base sales plans and decision making upon, we would search on Google. We’d go to websites, where we found heterogeneous Excel files and Power Points. We had to spend time cleaning it up, manipulating it, before we could get a trend line. Wouldn’t it be great if there were a “Google for numbers?” We aggregate quantifiable data into a single database that enables comparisons. Most of the business is licensing the technology to other companies. It’s used by more than 40,000 people a month.

Why add energy data?

Gislason: DataMarket is a ‘nice to have’ for many people but not a must-have for anybody. We realized that we need to go for narrower audiences, starting to do with more vertical approach. Energy is the first target. The Energy Datapalooza is a great venue to kick off this first initiative.

What service are you providing?

Gislason: This is an aggregation service, mostly based upon public data. In the energy space, a key data provider is the Energy Information Administration (EIA). They have their own systems and way of publishing. EIA has 8 different systems and no unified way to search through it. That’s the service that we’re selling: aggregating, making it super easy to search, download, and compare. All the providers out there have been super helpful in helping.

What value are you adding to the open data?

Gislason: Lots of data has been made available already but there are two issues: discoverability and usability. If you’re a person that uses energy data, you have to be able to find it. You also have to spend a lot of time to clean it up. Now you can use our technology. We add value on top of it: we normalize it and provide services. We’re not taking away anything. We’re adding value for tens of thousands of people using this data every day.

What are the value added services?

Gislason: The magic actually happens in search. Take “oil production in Colorado,” with 16 data sets. You can visualize data, do charts or export the data. You can download the chart or the data view. You can use the API. You can connect in R. Data from the EIR, EPA, or UN all comes through the same interface.

You can get all these data sources for free. The business case is to gather a large audience by having data for free. The business model is a subscription service, on an annual basis. Pricing is not set yet. For professional use, it will probably be a few thousand dollars a year. We make a lot more data usable. It’s not initially commercially available but people can sign up. There will be a free 2-week trial for anybody.