ENTRIES TAGGED "Facebook"

Here are a few stories from the data space that caught my attention this week.

Big obstacles for big data

For the latest issue of Foreign Policy, Uri Friedman put together a summarized history of big data to show “[h]ow we arrived at a term to describe the potential and peril of today’s data deluge.” A couple months ago, MIT’s Alex “Sandy” Pentland took a look at some of that big data potential for Harvard Business Review; this week, he looked at some of the perilous aspects. Pentland writes that to be realistic about big data, it’s important to look not only at its promise, but also its obstacles. He identifies the problem of finding meaningful correlations as one of big data’s biggest obstacles:

“When your volume of data is massive, virtually any problem you tackle will generate a wealth of ‘statistically significant’ answers. Correlations abound with Big Data, but inevitably most of these are not useful connections. For instance, your Big Data set may tell you that on Mondays, people who drive to work rather than take public transportation are more likely to get the flu. Sounds interesting, and traditional research methods show that it’s factually true. Jackpot!

“But why is it true? Is it causal? Is it just an accident? You don’t know. This means, strangely, that the scientific method as we normally use it no longer works, because there are so many possible relationships to consider that many are bound to be ‘statistically significant’. As a consequence, the standard laboratory-based question-and-answering process — the method that we have used to build systems for centuries — begins to fall apart.”

Pentland says that big data is going to push us out of our comfort zone, requiring us to conduct experiments in the real world — outside our familiar laboratories — and change the way we test the causality of connections. He also addresses issues of understanding those correlations enough to put them to use, knowing who owns the data and learning to forge new types of collaborations to use it, and how putting individuals in charge of their own data helps address big data privacy concerns. This piece, together with Pentland’s earlier big data potential post, are this week’s recommended reads.

Read more…