In the following interview, Rogers discusses the changes he’s seen in data journalism over the last five years and how new tools and increased notoriety will shape the data journalism space.

Why has data become the story for some journalists like yourself?

Simon Rogers: It’s a big change for reporters, to go from being suspicious of numbers to noticing that often data journalism is the only way to get stories from them. I think it’s a combination — the huge growth in published data out there combining with things like WikiLeaks, which changed the game for news editors to realize this was a new way to get stories.

Why do you think readers like data-centric stories?

Simon Rogers: I think it’s about trust. The public doesn’t trust reporters any more, but if you can show the workings behind your stories, to be transparent, then it makes your stories stronger. After years of unfettered comment online, there’s a real desire for facts.

You’ve been at data journalism for a while. How has it changed over the last five years?

Simon Rogers: It’s become much more mainstream. The work that I do used to be regarded as a little eccentric by the news desk; now it’s part of the fabric of The Guardian.

Are the tools of data journalism getting better?

Simon Rogers: Yes and no. If you can code, they are becoming brilliant, with things like The Miso Project really changing how we can work. But if you’re a reporter in a hurry, I’m getting a sense of stalling. We had a flurry of great tools a couple of years ago — Google Fusion Tables, Refine and so on — but we’ve also lost previously helpful things like Many Eyes. Datawrapper is a great new way to generate charts, and I’m liking CartoDB, but we still need more that anyone can use.

How can journalism schools prepare student journalists for our increasingly data-centric world?

Simon Rogers: I think it’s less about tools, as these are always changing. It’s more about helping create an attitude that looks for stories in numbers in a journalistic way — asking the same questions as you would in person to a contact.

Given the role of data analysis in this year’s US election, and the near mythic status of Nate Silver, what does this mean for data journalism moving forward?

Simon Rogers: It helps with the general feel that data is something to be embraced rather than avoided in journalism. And not all data work has to be as amazing as Nate’s. Much of what we do is very simple analysis: Has something become bigger or smaller? How does it compare? That sort of thing. But the key point holds: it helps you get stories.

Strata Conference Santa Clara — Strata Conference Santa Clara, being held Feb. 26-28, 2013 in California, gives you the skills, tools, and technologies you need to make data work today.

Save 20% on registration with the code STRATA20

Related:

• Investigating data journalism: O’Reilly Radar series
• The work of data journalism: Find, clean, analyze, create … repeat
• Before you interrogate data, you must tame it
• Data journalism profiles and coverage
• The Data Journalism Handbook (book)

If a game maker like EA is considered a big data company, it could have implications for other companies we might not think of as typical big data generators. With that in mind, I got in touch with Rajat Taneja, chief technology officer at EA and a keynote speaker at the upcoming Strata Conference in California. Since Taneja came on board with EA in 2011, he’s helped steer the company’s technological initiatives, including understanding the impact this growing data store will have on the firm — both from a processing standpoint and how to use it to provide games and services customers want most. He says no matter what your company does, if you have constantly connected online services, you are very likely going to be dealing with lots of data.

Our interview follows.

How does big data apply to a game company like EA?

Rajat Taneja: When thinking about big data, video games may not immediately come to mind but in reality, video games today are always-on, digital interactive entertainment that transmits huge amounts of data across the network as you play. Our network processes terabytes of data every day just from gameplay events. As our games become even more interconnected and cross-device enabled, the experiences become even richer and telemetry will only increase.

Where does your data come from and how do you collect it?

Rajat Taneja: Gaming is a multi-user online experience now, where friends and the gaming community come together across the globe to play online together. Everything from the game’s content, to a player’s progress, social functionalities, payment processes to account information all contribute to the amounts of data that we have to process on our EA network.
 
We have a very sophisticated internal data platform that ingests, processes and efficiently stores the data across a wide variety of systems spanning structured and unstructured data bases. The platform leverages open-source technologies (like the Hadoop stack) as well as vendor technology. This data is then made available through a variety of analytic tools for automated actions, deeper analytics and reporting. This is where data turns into information and insight.

What kinds of tools do you use to process and view the data?

Rajat Taneja: Our data platform was predominantly architected around a very structured pipeline and conventional data warehouse systems. This was primarily geared for complex OLAP and OLTP workloads. However, the nature of data we collect is now changing dramatically, and its use and importance in personalizing game play experiences is becoming critical. We, therefore, have to rethink fundamentally how the pipeline is architected and the scale it has to support. Our new architecture is now focused on far more agile, near real-time processes that are now being predominantly built using off-the-shelf open source tools for storage, compute, modeling and analytics.

How do you make data accessible to your employees, partners and even your customers? And how can presenting this data enhance your relationships with these parties?

Rajat Taneja: Origin, our direct-to-consumer gaming platform, brings the customer’s gaming experience into one single application, so no matter what game or device you play on, you can connect all of your experiences. Your friend list lives there, your achievements, and your payment information. This makes it easy for our consumers to control their own data and track their progress, managed securely on our network.
 
Inside the company, the data and derived insight are made available to game developers and studios so they can gain deep insight into playing patterns and feature usage. This helps them tailor the game experiences and capabilities to enhance the enjoyment by our consumers. Our live operations team gets a near real-time view of various game functionality, which provides them with critical information that is used to manage the day-to-day operations of the game. Our customer experience and marketing teams get relevant snap shots of data critical to their efforts and initiatives.
 
Overall, data is a key connective tissue that allows all functions to operate with more confidence and deeper insight into their specific areas.

If EA is a big data company, what does that mean for other companies? What lessons can other companies take from yours?

Rajat Taneja: So much of our everyday lives today are connected to online data and interactions, and games are becoming no different.  Our industry is going through a massive transition — a shift from packaged goods to digital delivery and constantly-connected online services — that has vaulted us into the sphere of big data. 

We have all the big data challenges, but also many, many opportunities to transform games from something you buy to a place that you go. The big key for us is looking at our customer base and wanting to meet and exceed their high expectations for an entertainment experience. They won’t tolerate network downtime, their data must be secure, and they don’t want to have multiple logins and passwords. We’re building the infrastructure for seamless experiences that take them from device to device, platform to platform, where their experience and connection is consistent across time and devices. By putting the consumers at the center of their gaming world, we have set into motion a technical challenge where using big data is a critical component of the solution. Think about your customers, put their needs first, and evaluate if big data is part of your solution as well.

This interview was edited and condensed.

Strata Conference Santa Clara — Strata Conference Santa Clara, being held Feb. 26-28, 2013 in California, gives you the skills, tools, and technologies you need to make data work today.

Save 20% on registration with the code STRATA20

Related:

• Big data goes to work

It’s easy to dismiss this as a problem for companies like Google and Facebook, which are gathering mountains of data from users. However, as GoodData CEO Roman Stanek (@RomanStanek) points out in the following interview, the growing amounts of data from a variety of sources makes big data an issue that has an impact on every company, regardless of size.

Stanek, who has been an entrepreneur for more than 20 years, started GoodData in 2007 as a way to simplify business intelligence by putting it in the cloud. Today, he sees big data as more than a business intelligence problem, and as he has watched his business evolve, he believes companies like his can take big data out of the realm of data scientists and put it into the hands of ordinary business users.

There is a perception that big data is a big company problem. What role does big data have in small- to medium-size organizations?

Roman Stanek: Big data comes from hundreds of sources, most of which are outside a company’s firewalls, such as customer interactions, social media and emails. A company’s size is irrelevant to the volume of big data it has to manage and understand. For example, a company with 100 employees may have to answer thousands of customer-support calls coming in from Facebook, Twitter, email and telephone. That’s a massive amount of data it has to deal with.

In addition, big data represents tremendous potential wealth for all companies, no matter how small or large those enterprises are. When businesses are smart about leveraging data, they can make better and faster business decisions.

What factors are contributing to the growing amount of business data?

Roman Stanek: To name just a few: inventory levels, sales results, negative comments on Facebook, positive comments on Twitter, shopping habits on Amazon, playlists on Pandora and online search habits. No matter what you call the information or what it describes, it’s all data being collected.

IDC predicts digital data will grow to 2.7 zettabytes in 2012 (PDF). Thanks to new technologies like Hadoop, once-unquantifiable data — like Facebook conversations and tweets — can now be quantified. Nearly everything is measurable. The result is that companies are spending big dollars to collect, store and measure astronomical amounts of data.

What’s the difference between today’s big data and yesterday’s business intelligence (BI)?

Roman Stanek: Traditional BI is antiquated and broken. Current tools cannot cope with the massive amounts of unstructured data coming in from social networks and the cloud. Commonly used BI tools left a long trail of failed implementations and frustrated customers because they required such heavy lifting from IT departments.

In my opinion, big data’s only real value lies in the ability of businesses to transform data into insights they can act on. Sales managers, for example, can quickly analyze sales reps’ results, view new and lost contracts and compare team performance to the plan they set months earlier.

Help desk staff can see how individual customers affect sales and profit, so they know when to go above and beyond to retain certain customers while allowing the low fliers to churn. Insurance agents can predict the cost and nature of impending damage as hurricanes hurtle toward their region.

How do you get to a point where big data is not just in the realm of data scientists asking the big questions, but where business users find the answers to do their jobs and drive business growth?

Roman Stanek: You shouldn’t need a PhD in statistics to interpret data. I believe people already know what data they need to dive into to make strategic decisions. If I’m a chief marketing officer, for example, I’m dying to learn if Facebook is really driving my sales and, if so, to what extent. To find that out, I need a modern app that pulls in data from myriad sources and then presents that information in a simple, visually intuitive way that lets anyone inside a company make sense of his or her data. The growth and maturity of cloud computing technologies have finally made this combination possible.

How does business get to the point where big data is driving business strategy looking forward as opposed to what has happened, looking back?

Roman Stanek: Using big data to drive business strategy is the next stage of maturity. No company will jump from being anecdotal to analytical in one day. Companies must focus on becoming metrics driven so that they reach that level of maturity in their data analytics.

Traditional BI tools look at historical trends, allowing you to analyze what’s already happened. For example, I can see how my sales trended in the previous two to three quarters. The challenge with this approach is it doesn’t enable you to react to today’s information so you can influence tomorrow’s business performance.

In contrast, next-generation analytics leverage the scalability and processing power of the cloud to find insight people can use now. I recently heard about HR apps that analyze big data to determine which personality types are best suited for different types of jobs. That kind of information enables HR departments to create simple but revealing questions to find — and retain — the best employees for each position. That reduces churn, which has a direct impact on a company’s bottom line.

That’s just an example. The point is that a company’s size truly doesn’t matter. Big data offers the key for any company, regardless of size, to find new sources of revenue, increase profit and make smarter decisions, faster. Thanks to new real-time, cloud-based technologies, that ability is already a reality.

This interview was edited and condensed.

Strata Conference Santa Clara — Strata Conference Santa Clara, being held Feb. 26-28, 2013 in California, gives you the skills, tools, and technologies you need to make data work today.

Save 20% on registration with the code STRATA20

Bitsy Bentley (@bitsybot) is the director of data visualization at GfK Custom Research, where she works with information designers to craft meaningful data experiences for a variety of business audiences. In the following interview, she discusses the space between a “wow” response and an “aha” moment, how her team addresses privacy concerns, and why practice is vital for both visualization creators and viewers.

Bentley will explore related visualization topics during her presentation at Strata Conference + Hadoop World in New York City later this month.

Why are data visualizations an effective way to understand the underlying data?

Bitsy Bentley: There is so much beauty and richness in big datasets, and now that we have enough processing power to harness that richness, it’s little wonder that interest in data visualization is exploding. To quote John Tukey: “The greatest value of a picture is when it forces us to notice what we never expected to see.” My clients find that, whether they’re more concerned with numbers or more concerned with stories, an appropriate visual is integral to their understanding of the data.

Visualization unlocks the serendipity of data analysis. It provides a language that is less intimidating than an overwhelming array of digits. Something as simple as a set of histograms breaking down the distribution of a data store makes it easy to find irregularities and outliers in the data.

How do you balance technical versus non-technical requirements when creating data visualization applications?

Bitsy Bentley: From my perspective, it all comes down to communication. I enjoy the luxury of working with a brilliant team of developers that is very open to working in an agile, iterative process. It’s not possible to craft meaningful data experiences in a vacuum. Multi-disciplinary teams are essential to building the best data interactions.

When we’re developing new products, we include the business and technical teams in our user profile and wireframing discussions. These workshops are key to helping the entire team take ownership of the business problems we’re addressing in our data analysis applications. I find that by keeping the end users at the center of all our discussions, we are able to avoid the common pitfall of allowing the technology to wag the design and vice versa. All the requirements hinge on meaningful use cases. It allows the data interactions to push the boundaries of the technical solution, and it allows the technical solution to push the boundaries of the data interactions.

While it’s clear that creating data visualizations provides a way to understand big data in a non-threatening way, what are the challenges around creating a good one?

Bitsy Bentley: The best data visualization is the one that meets the audience’s need, and identifying that need is often a big challenge. I subscribe to the Tukey philosophy that it’s better to approximate an answer to the right question than provide an exact answer to the wrong question. It’s easy to get distracted by all the possible ways of describing a big dataset and forget that we’re often dealing with an audience of statistical novices.

My goal is that the users of products I design will take the data interactions for granted. It’s difficult to balance the “wow” of a visually striking presentation (that tends to sell well) with the “aha” moment when the user is able to easily and simply understand the problem at hand. At the beginning of my design process, I like to do a problem discovery phase to articulate the specific user needs we want to meet. It provides a good framework for measuring the success of the data experience and challenges us to keep the user at the center of the solution.

One of the big issues around using data is privacy. How does visualization provide a way to present big data without revealing details about the underlying data store? Do you typically provide access to source data when showing a visualization?

Bitsy Bentley: The market research industry has a long history of being concerned with data privacy. My company adheres to the Council of American Survey Research Organizations’ (CASRO) code of standards and ethics. These standards outline our responsibilities to our respondents, our clients, and to the public. With those responsibilities in mind, my team and I look to show patterns, not individual people, as we visualize things like a generalized customer journey or overall foot traffic in an area.

There are times that I like using deliberately fuzzy visualization methods, such as heatmaps, to keep the focus on high-level patterns. Much of our work is proprietary, so data access is usually restricted to our clients. In our visual data analysis products, we allow users to download the underlying aggregated data of the charts and graphs in the application. Occasionally our clients request SPSS files, and every so often we provide data through an API.

We are clearly at the beginning of this journey when it comes to understanding big data and illustrating it through visualizations. How can we become better at creating and understanding visualizations?

Bitsy Bentley: Like many things in life, it’s all about practice. I find the visual and statistical literacy of many audiences to be very low, but with just a little exposure, their standards rapidly change.

The best ideas come from dialogue.  Every data graphic we make is an opportunity to learn something new and challenge our audience to see something unexpected. Every data interaction we design is an opportunity to change the way our users think about the role data plays in their decision-making process. Every program we write helps us gain a deeper understanding of the relationship between data management and data visualization. If we want to be better at creating and understanding visualizations, we need to make more of them.

Strata Conference + Hadoop World — The O’Reilly Strata Conference, being held Oct. 23-25 in New York City, explores the changes brought to technology and business by big data, data science, and pervasive computing. This year, Strata has joined forces with Hadoop World.

Save 20% on registration with the code RADAR20

This interview was edited and condensed.

Related:

• Walking the tightrope of visualization criticism
• Why data visualization matters
• How to create a visualization
• Stories over spreadsheets

Of course, if social sites start selling data to the highest bidder that leaves open questions of data ownership and privacy and finding ways to strip personal identifiers.

Marie Wallace (@marie_wallace) is social analytics strategist for the IBM Collaboration Solutions division. She has spent more than a decade at IBM working on content analytics, and her experience uniquely positions her to address questions regarding big data, social media and analytics. Our interview follows.

Social media’s real value might not be in selling ads, but in the data they are collecting. Why do you think that is?

Marie Wallace: The reason ad targeting has worked so well for search is because it’s aligned and supportive to that particular activity; when I am searching for information about products or services I am happy to get ads that may help direct my search. Ads are somewhat analogous to a value-added service and social search makes the ads more personalized and relevant, which is why Google has invested so heavily in Google+.

The key is that in most cases ads only work in a search-like context, however with most social media sites people are not going there to search. They are going to converse with friends and family, which makes ads interruptive and frequently invasive. This is further exacerbated by mobile, where limited real estate makes ads even more offensive as they are distracting and clutter the screen. Social search is one example of a service that sits on top of social data, but there are a whole plethora of other services that social data can drive — from market research to consumer/brand engagement, social recommenders, information filtering, or expertise location.

It’s one thing to recognize the value of data, but how do you extract that value?

Marie Wallace: Extracting value from data requires a well-described set of scenarios with a clear understanding of what facts would be considered valuable for those scenarios. For example; when looking for a job there are a very specific set of questions that people want asked and answered: employee sentiment, corporate success (revenue, customers, products, growth), location, demographics, technologies, industries, skills, competitors, values, culture.

These are very different to the questions (and hence analysis) that might be pertinent to a different scenario. For example; when deciding where to go on holidays people are likely more interested in the location, activities, accommodation, weather, cost, demographics, or visitor sentiment. The key here is that analysis has to be not only domain-, but scenario-specific, which is why targeted specialist services like LinkedIn or Tripadvisor are always going to be able to deliver greater analytics value for the specific scenarios they support.

There are concerns on social networks about the sites abusing the data users are contributing. Is there a reliable way to anonymize data and deliver it in aggregate form that strips out individual user information?

Marie Wallace: I think the issue of privacy is a more complex problem, and while anonymizing user information is part of the solution, I don’t believe it’s at the heart of the problem. I believe the key social media challenges moving forward will be those of permission, trust, and transparency. People need to know exactly how their data is being used so that they can give permission for that use and that use only. For example; if I have a Tesco loyalty card and I trust them to respect my data, then I might be happy for them to see my Facebook Likes so they can provide me with more relevant special offers. Or if I register on LinkedIn I know that my data is going to be provided to recruiters and hiring companies, but I most definitely don’t want them to use it for any other undisclosed purpose.

There is also a likelihood that in the future we will see information brokers emerge, which provides a level of indirection (perhaps even obfuscation or anonymization) where they act as mediators on our behalf. This simplifies the authorization model, but does assume that we trust the information brokers and the models that they use for controlling access to our information.

Have the tools caught up with the amount and variety of data so that services like social networks can begin to manipulate the data they collect?

Marie Wallace: Having spent the last decade working on content analytics and semantic technologies, I can confidently say that many of the required tools have been around for years waiting for demand to catch up with supply. The advent of social media, alongside the growth of a new generation of big data platforms, now gives them the perfect business problem, dataset, and execution platform through which to shine. However, I believe the industry does have one significant gap in this otherwise rich landscape of technologies, and it’s a gap that I believe will impact the value that we can derive from these social networks.

It’s our handling of massive-scale networks that I believe is going to become a technological challenge as we move rapidly toward massive-scale graphs with social, semantic, temporal, and geospatial characteristics and as we look to apply complex analytics across these networks. There are a number of existing technologies from the linked data world that could morph to fill this gap, or alternatively there is a new generation of graph databases and analytics algorithms emerging focused on tackling this specialized problem. Only time will tell in terms of which technologies will emerge the winners.

What kinds of uses could you envision social sites finding for their data?

Marie Wallace: For the medium-term, I suspect that we will continue to see social analysis being driven by the marketing, sales, and support organizations. Social data will be used for market research, to help expand sales channels, and to improve how brands interact with customers.

As we move from marketing to sales to support, the type of analysis becomes more complex and this will put pressure on the algorithms being used to evaluate the data and derive insights; identity and entity disambiguation, micro-segmentation, influence analysis, sentiment, intent, network information flow, and community dynamics. A growing number of social applications will emerge, each delivering niche value to consumers and generating specialist data for brands. This ecosystem of social networks will drive consumer-brand engagement; everything from consumer feedback systems, customer support, to product and service innovation. Brands will move away from a focus on passive listening/monitoring to one of active engagement, and this will require a broader range of analytics in order to optimize and operationalize those interactions.

Further out I see us expanding the personalization that can be realized. Social data will become increasingly important for personalizing every search and navigation experience from Google, Amazon, Netflix, to Expedia, however search is only the tip of the iceberg. I anticipate that in the longer term social data will be used to personalize a whole range of experiences that cross the physical/digital divide; transforming how we shop, what we think, how we learn, and ultimately how we live.

Just imagine what will happen when we intersect the social web, the semantic web, with the web of data. Then we will really see personalization take on a whole new form!

This interview was edited and condensed.

Strata Conference + Hadoop World — The O’Reilly Strata Conference, being held Oct. 23-25 in New York City, explores the changes brought to technology and business by big data, data science, and pervasive computing. This year, Strata has joined forces with Hadoop World.

Save 20% on registration with the code RADAR20

Related:

• Social data is an oracle waiting for a question
• Social network analysis isn’t just for social networks
• Data markets aren’t coming. They’re already here
• Tim O’Reilly: My Contrarian Stance on Facebook and Privacy
• Data, Not Ads, Will Make Social Networks Profitable