Because readers of this blog have been grappling with these concerns for some time. I’ll provide the barest summary of topics covered in Mayer-Schönberger and Cukier’s extensive overview, then provide some complementary ideas of my own.

Summary of book topics

Some of the themes of Big Data that grabbed my interest include:

• New tools for measuring the world and people’s activities provide data sets that are many orders of magnitude higher than we are used to having, and computers tied together in clusters are run novel techniques to find insights never available to us before.
• Data crunchers are finding correlations that provide useful guidance for actions. Mere correlation cannot tell us why something is happening, but often it doesn’t matter. The authors cite numerous examples where correlations by themselves suggested valuable actions.
• Because big data opens up efficiencies to those savvy enough to use it, the future of business belongs to huge organizations (including middlemen and aggregators) with the resources to collect both data and experts to manipulate it, or to smaller organizations who are nimble enough to make hay from open or cheaply available data sets.
• Control over our own lives may slip more and more from our hands as institutions use statistical insights to determine not only what we are doing, but what we are likely to do in the future. One chapter in Big Data is devoted to policy-related remedies.
• Old-timers’ intuitions are challenged by the findings of big data, just as Deep Blue’s brute-force processing of chess moves can overcome the world’s best human chess masters. Nevertheless, the authors end affirming the importance of human insight and choice.

These represents a grand agenda for one book (nor have I exhausted all its topics), but I’d like to jump ahead to ideas that the Big Data stimulated for me, leaving it up to readers to get the book for themselves if they want to study all its conclusions. In the interest of full disclosure, I’ll mention that one author– Cukier, the data editor of The Economist–helped me get an article published several years ago.

Other aspects of big data

Mayer-Schönberger and Cukier’s view of traditional statistical techniques deserves a bit of examination. They tend to place these in opposition to newer techniques of crunching big data. According to their thesis, the old techniques were developed to deal with small samples and all the uncertainties they presented about representing the whole population. Those outdated assumptions compromise their applicability to a new age, where computers just iterate over the whole population. The authors even recount a suspicion of traditional statisticians made by one of their big data experts, New York City’s Mike Flowers, who was put off by statisticians’ interest in “arcane concerns about mathematical models” (p. 186).

Certainly, the authors say, there is a place for traditional statistics. It can even be used to run traditional experiments in order to validate suggestions made by big data crunching. But I think the relationship between old and new techniques is much tighter. This question has an important bearing on the power exerted by big data, because I believe proper techniques will be harder to learn and accurately apply than Mayer-Schönberger and Cukier suggest. While they expect the skills soon to become “commonplace” (pp. 125-126), I think there will be a crying shortage for some time, allowing a few large institutions with deep pockets to corner the market.

Let’s take the common big-data task of clustering, which might help in such situations as an art dealer trying to determine that Leonardo da Vinci is closer to El Greco in style than to Andy Warhol. Clustering algorithms can take a very long time to run, and choosing good starting points is important to reduce compute time. In fact, characteristics of the data can help a data scientist choose which algorithm to run in the first place. So what can provide with a starting point for the big data venture? A traditional statistical analysis of a random sample could be a good choice.

This extends throughout the field of data. Even the choice of the best sorting technique–a common exercise in the first classes for programming students–can vary depending on the characteristics of the data being sorted.

Big Data is not oriented toward this sort of technical discussion. The authors chose quite reasonably to avoid equations or other accoutrements of a mathematical explanation, which I’m sure would have scared off readers. And yet without some such background (to be sure, I’m no mathematician or statistician), one can’t determine the real strengths and weaknesses of the big data movement.

Let’s turn to the critical question of transparency, which Mayer-Schönberger and Cukier consider a necessity to help people challenge the decisions that others derive from data analysis. Transparency is no panacea, in my view. First, algorithms are incredibly complex. Second, as we’ve seen, the choice of algorithm (as well as the data to be analyzed) requires some subjective judgment, which is hard to challenge.

Worse still, any calculation affecting humans has winners and losers, and therefore makes some people eager to game the system. Big Data mentions the trivial example of orange used cars being found to be in better shape than other used cars (pp. 66-67), and points out how ridiculous it would be for car owners to paint the cars orange before selling them. These kind of dueling incentives apply across the board. It’s one reason Google doesn’t publish its search rank algorithms–and in fact, one reason it changes that algorithm on a daily basis.

Thus, instead of asking banks or insurers to reveal their decision-making processes, it may be better to give individuals access to data about themselves, and a process such as the “external algorithmist” proposed by Mayer-Schönberger and Cukier (p. 181) to allow individuals to present exculpatory evidence.

Both the external algorithmist and the internal algorithmist (similar to an ombudsman) envisioned by the authors are good additions to an organization. Dr. Brigitte Piniewski (whose work with an NIH-funded experiment in the community collection of health data, was covered in a Strata article) suggests that the role of an algorithmist is a very difficult one, and in fact too big for a single person to fill. The algorithmist, in her assessment, must have some basic knowledge not only of statistics but of real-world disciplines such as physics and biology, in order to have a sense of what is possible and what analytic results are absurd on their face.

Even more important for algorithmists is a correct attitude: an inate skepticism that may be in-born trait rather than something teachable. And she says this willingness to constantly challenge accepted beliefs must run throughout the organization, which cannot rely on a single expert to provide this corrective.

Because many institutional decisions take place in the background where the affected individuals never find out they took place at all (when have you ever learned that a marketer decided not to offer you a great deal?), the reactive approach has grave limitations. Furthermore, it’s hard to trust institutions to take self-corrective actions to preserve privacy and individual automomy at this historical moment when we’re reeling from revelations that the IRS targeted institutions based on their political positions and the Justice Department gathered phone records across the board from Associated Press reporters.

My last cluster of concerns relates to the role of human intuition and creativity in an age of big data, where the authors end their book on a high note. It’s important to recognize that big data analysis consists of applying what happened in the past to what will happen in the future. Had there been no recent influenza outbreaks, Google could not have run the tests that produced their famous flu prediction algorithm.

And thus the data we collect in the past hangs over us. Suppose we analyze arrest and conviction records to determine whom we should target most heavily for policing? Guess what? African-Americans and Hispanics will get the bulk of policy scrutiny, because the police have targeted them disproportionately for generations. In short correlations could turn into textbook cases of self-fulfilling prophecies. As Mayer-Schönberger and Cukier say, we still need human intervention to think outside the box.

I think big data will accentuate today’s trend to differentiate between the commoditized and the innovative. Like manufacturing, we will see more and more decision-making calling on big data–but with crucial human correctives, as noted before.

The relation between invention and standardization is a bit like the promise of 3D printers such as the Makerbot. On the one hand, they allow flights of invention by clever tinkerers like never before. But the printers depend on microchips made in sterile labs by the millions, and other materials from large manufacturers. (The biggest US producer of polylactide filament is a subsidiary of Cargill.) Used properly, big data could similarly be the greatest contributor in history to personal innovation.

I pre-date the Hadoop crew at Yahoo!, but I intimately understood the grid engineering that made Hadoop possible. For years, the working title of this book was The Art and Craft of Platform Engineering, and when I started working on Hadoop after a stint in the Red Hat kernel group, many of the ideas that were jammed into my head, going back to my experience with early supercomputers, all seem to make perfect sense for Hadoop. This is why I frequently refer to big data as “commercial supercomputing.”

In Disruptive Possibilities, I discuss the implications of the big data ecosystem over the next few years. These implications will inundate vendors and customers in a number of ways, including:

1. The disruption to the silo mentality, both in IT organizations and the industry that serves them, will be the big story of big data.
2. The IT industry will be battered by the new technology of big data because many of the products that pre-date Hadoop are laughably unaffordable at scale. Big data hardware and software is hundreds of times faster than existing enterprise-scale products and often thousands of times cheaper.
3. Technology as new and disruptive as big data is often resisted by IT organizations because their corporate mandate requires them to obsess about minimizing OPEX and not tolerate innovation, forcing IT to be the big bad wolf of big data.
4. IT organizations will be affected by the generation that replaces those who invested their careers in Oracle, Microsoft, and EMC. The old adage “no one ever gets fired for buying (historically) IBM” only applies to mature, established technology, not to immature and disruptive technology. Big data is the most disruptive force this industry has seen since the introduction of the relational database.
5. Big data requires data scientists and programmers to develop a better understanding of how the data flows underneath them, including an introduction (or reintroduction) to the computing platform that makes it possible. This may be outside of their comfort zones if they are similarly entrenched within silos. Professionals willing to learn new ways of collaborating, working, and thinking will prosper. That prosperity will be as much about highly efficient and small teams of people as it is about highly efficient and large groups of servers.
6. Civil liberties and privacy will be compromised as technology improvements make it affordable for any organization (private, public or clandestine) to analyze the patterns of data and behavior of anyone who uses a mobile phone.

Much more is covered in Disruptive Possibilities: How Big Data Changes Everything. Download it for free here.

While social media placed graph data on the radar of many companies, similar data sets can be found in many domains including the life and health sciences, security, and financial services. Graph data is different enough that it necessitates special tools and techniques. Because tools were a bit too complex for casual users, in the past this meant graph data analytics was the province of specialists. Fortunately graph data is an area that has attracted many enthusiastic entrepreneurs and developers. The tools have improved and I expect things to get much easier for users in the future. A great place to learn more about tools for graph data, is at the upcoming GraphLab Workshop (on July 1st in SF).

Data wrangling: creating graphs
Before you can take advantage of the other tools mentioned in this post, you’ll need to turn your data (e.g., web pages) into graphs. GraphBuilder is an open source project from Intel, that uses Hadoop MapReduce¹ to build graphs out of large data sets. Another option is the combination of GraphX/Spark described below. (A startup called Trifacta is building a general-purpose, data wrangling tool, that could help as well. )

Data management and search
Once you have a graph, there are many options for how to store it. The choice of database largely depends on amount of data (# of nodes, edges, along with the size of data associated with them), the types of tasks (pattern-matching and search, analytics), and workload. In the course of evaluating alternatives to MySQL (for storing social graph data), Facebook’s engineering team developed and released Linkbench – a data set that can be used to study how graph databases handle production workloads.

Most graph databases (such as Neo4j², AllegroGraph, Yarcdata, and InfiniteGraph) come with tools for facilitating and speeding up search – Neo4j comes with a simple query language (Cipher) for search, other graph databases support SPARQL. The Titan distributed graph database supports different storage engines (including HBase and Cassandra) and comes with tools for search and traversal (based on Lucene and Gremlin). Used by Twitter to store graph data, FlockDB targets operations involving adjacency lists.

Among Hadoop users HBase is a popular option for storing graph data. Hadapt’s analytic platform³ integrates Apache Hadoop and SQL, and now also supports graph analysis.

Graph-parallel frameworks: Pregel, PowerGraph, and GraphX
BSP is a parallel computing model that has inspired many graph analytics tools. Just like Hadoop’s map and reduce, Pregel⁴, Giraph and Pregelix, come with primitives that let neighboring nodes send/receive messages to one another, or change the state of a node (based on the state of its neighboring nodes). Efficient graph algorithms are a sequence of iterations built from such primitives. GraphLab uses similar primitives (called PowerGraph) but allows for asynchronous iterative computations, leading to an expanded set of (potentially) faster algorithms.

GraphX is a new, fault-tolerant, framework that runs within Spark. Its core data structure is an immutable graph⁵ (Resilient Distributed Graph – or RDG), and GraphX programs are a sequence of transformations on RDG’s (with each transformation yielding a new RDG). Transformations on RDG’s can affect nodes, edges, or both (depending on the state of neighboring edges and nodes). GraphX greatly enhances productivity by simplifying a range of tasks (graph loading, construction, transformation, and computations). But it does so at the expense of performance: early prototype algorithms written in GraphX were slower⁶ than those written in GraphLab/PowerGraph.

Machine-learning and analytics
Machine-learning tools that target graph data lead to familiar applications such as detecting influential users (PageRank) and communities, fraud detection, and recommendations (collaborative filtering is popular among GraphLab users). Moreover techniques developed in one domain are often reused in other settings. Besides GraphLab, distributed analytics have been implemented in Giraph, GraphX, Faunus, and Grappa. In addition, graph databases like Neo4j and Yarcdata come with some analytic capabilities. As I noted in a recent post, open source, single-node systems like Twitter’s Cassovary⁷ are being used for computations involving massive graphs.

Visualization
When you’re dealing with large graphs, being able to zoom in/out helps with clutter, but so do clever layout algorithms. Popular tools for visualizing nodes and edges include Gephi and GraphViz. Users who want to customize their graphs turn to packages like d3.

(1) I would love to see a version of GraphBuilder that’s built on top of Spark.
(2) Many of these systems are quite efficient. For example a single instance of Neo4j can handle very large graphs (“into the tens of billions of nodes/ relationships/ properties”).
(3) Note that using standard Hadoop for graph processing may not be the most efficient option. This talk by Hadapt co-founder Daniel Abadi describes an advanced approach to graph analysis using Hadoop.
(4) Related frameworks include GoldenOrb and Hama.
(5) Resilient Distributed Graphs (RDG) extend Spark’s Resilient Distributed Dataset (RDD).
(6) As the developers of GraphX note: “We emphasize that it is not our intention to beat PowerGraph in performance. … We believe that the loss in performance may, in many cases, be ameliorated by the gains in productivity achieved by the GraphX system. … It is our belief that we can shorten the gap in the near future, while providing a highly usable interactive system for graph data mining and computation”
(7) On the plus side, being single-node means Cassovary doesn’t have to deal with finding the optimal way to partition a graph. On the other hand, it is limited to graphs that fit in the memory of a server – a limitation it alleviates through the use of efficient data structures.

Google introduced a new version of Google maps at Google I/O this week that learns from each use to customize itself to individual users, adapting based on user clicks and searches. A post on the Google blog outlines the updates, which include recommendations for places you might enjoy (based upon your map activity), ratings and reviews, integrated Google Earth, and tours generated from user photos, to name a few.

Leo Mirani at The Atlantic says the update “fixes the one thing that has always been wrong with maps” — namely, neutrality. Though maps are typically viewed as neutral objects, Mirani argues, they’re “about as impartial as journalism.” He writes:

“Is the hot dog vendor who stands on a street corner as worthy of inclusion as the bank on the same corner? That decision still lay with mapmakers — in this case a giant internet company. Google’s solution to the problem was to remove itself from the equation.”

Mirani notes that “mental maps,” maps created based on how a person sees the world, is nothing new, but Google has managed to provide the mental map as a service to millions of users.

In a post at The Atlantic Cities, Emily Badger argues that this level of individual customization comes at a cost — “An algorithm that knows you too well,” she writes, “does a terrible job of telling you things you don’t already know.” Badger relates the new maps to Eli Pariser’s “filter bubble” concept: as the algorithms get to know you, you increasingly get content that leans toward your established point of view, preventing you from broadening your experiences. Badger also notes the increasing issue with “the inequality of information online…rendering some real-world people and places virtually invisible” and wonders if the new maps won’t exacerbate the problem. You can read her full report at The Atlantic Cities.

App development trends lean toward predictive, intelligent service offerings

MIT Technology’s Tom Simonite took a look at the growing app development trend to provide users with personalized information, service connections, and recommendations before even being prompted or searched. Noting the departure of this trend from typical “dumb” computers and software that waited for human operator interaction, Simonite looks at such apps as Google Now, which aims to predict a user’s actions in order to provide appropriate assistance, and the Osito iPhone app, which similarly predicts actions to offer helpful information but also provides actionable assistance, such as a button to call a cab when a user’s flight reminder pops up.

Bit.ly chief data scientist Hilary Mason told Simonite that Google Now is far from perfect in the usefulness of the information it provides, but she uses it anyway and finds the technology important “because it’s the first time Google has taken all they know about us to make a product that makes our lives better.” You can read Simonite’s full report at MIT Technology Review.

In related news, Google announced new services at its I/O conference this week that will help developers build apps that can track users as well as Google does — and without draining a user’s battery. Jessica Leber reports at MIT Technology Review that the service will allow developers to build apps that tap into the accelerometer of a user’s device instead of the “power-hungry” GPS sensor to determine whether a person is driving, walking or cycling; the apps would run Google algorithms, Leber notes, that would “learn over time whether a person is stuck in traffic or just out for an evening stroll.” The new services also include the ability for developers to create geofences to trigger actions based on a user’s location. You can read Leber’s full report at MIT Technology Review.

The New Yorker gets an anonymous inbox to protect sources

The New Yorker launched Strongbox this week, a new tool for people to share files, information, and messages with The New Yorker staff with an increased level of anonymity. In a post announcing the launch, Amy Davidson explains that with the way Strongbox is set up, they won’t be able to tell where a piece of information came from, and thus won’t be able to tell anyone where it came from, providing better protection for their sources.

The Strongbox site explains further that the system is only accessible through the Tor network and that when a user submits a message or file, no I.P. address is recorded and no information about a user’s browser, computer or operating system is gathered. Strongbox also doesn’t include any third-party content or use cookies. Users are given a randomly generated, unique code name so that New Yorker staff have a way to contact them if necessary via a message left in Strongbox.

Davidson notes that the Strongbox tool was developed by Aaron Swartz and Kevin Poulsen, and is based on their underlying code dubbed DeadDrop, which will be open source. You can read the development story and history in a post Poulsen wrote for the New Yorker.

Tip us off

News tips and suggestions are always welcome, so please send them along.

Related:

• Google unveils its Knowledge Graph
• Google’s Rich Snippets and the Semantic Web
• Predictive data analytics is saving lives and taxpayer dollars in New York City
• Why you can’t really anonymize your data
• More Strata Week coverage

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

What inspired you to become a code artist?

Scott Murray

Scott Murray: I had designed websites for a long time, but several years ago was frustrated by web browsers’ limitations. I went back to school for an MFA to force myself to explore interactive options beyond the browser. At MassArt, I was introduced to Processing, the free programming environment for artists. It opened up a whole new world of programmatic means of manipulating and interacting with data — and not just traditional data sets, but also live “data” such as from input devices or dynamic APIs, which can then be used to manipulate the output. Processing let me start prototyping ideas immediately; it is so enjoyable to be able to build something that really works, rather than designing static mockups first, and then hopefully, one day, invest the time to program it. Something about that shift in process is both empowering and liberating — being able to express your ideas quickly in code, and watch the system carry out your instructions, ultimately creating images and experiences that are beyond what you had originally envisioned.

Why did you decide to write Interactive Data Visualization for the Web?

Scott Murray: D3.js is the most powerful tool for creating visualizations on the web, hands down. Yet when I tried to use it for a project in late 2011, there wasn’t a lot of useful information available on how to use it. A few early members of the community had written tutorials, but they assumed a level of JavaScript familiarity that I didn’t have. The official D3 documentation is excellent, but not necessarily accessible to beginners. So I set about banging my head against the wall for a few weeks, struggling to learn the peculiarities of D3, and taking notes every time I encountered a challenging concept or had a revelation. Once the project was done, I revisited my notes and starting writing tutorials that would introduce each of those challenging concepts to beginners — those new to JavaScript, and even HTML and CSS. (I’m seeing more people drawn to JavaScript via D3, just as my first experiences with JavaScript were thanks to jQuery.) My goal was to spare others the frustrations I experienced. Within months, the tutorials on my site were getting a lot of traffic, and people wrote in to request more tutorials: on the basics, on interactivity, on mapping. So I expanded the tutorials into a full-length book.

Who do you envision reading this book? What will they learn after reading your book?

Scott Murray: This book is for anyone interested in learning how to use D3 to create and publish visualizations on the web: journalists, designers, data scientists, statisticians, students, researchers, and would-be mapmakers. The book includes an introductory chapter on web fundamentals — HTML, CSS, JavaScript — so it’s very accessible to people new to web development, and even programming generally. There are also more than 100 code examples that accompany the book, so it’s easy to follow along and tweak the examples as you learn. In the end, anyone who works through the book will have a grip on all the basic concepts of D3. You’ll be able to make charts and graphs, even highly customized geographic maps with data overlays. And hopefully you’ll be inspired to learn more, experiment, and share your creations with the datavis community.

Any words of advice for an aspiring code artist?

Scott Murray: Start making things now. I get to work with students, and I see them get stuck in their heads, trying to plan out every last detail before they start working on a project. This results in the project never starting at all, or being finished in a very rushed fashion. People (myself included) often have a tendency to over-think things, especially intimidating projects that will involve learning something new, or trying something we’ve never tried. While thinking is good, over-thinking prevents us from doing. And, in reality, doing is a critical part of thinking — you can’t really separate the two. So I suggest getting comfortable with not knowing what you’re doing before you do it. Just start making things now, today, even if you feel underprepared or like you don’t have all the answers you need yet. Guess what? No one has all the answers (even though we pretend to). We’re all just here figuring this stuff out as we go. So don’t over-think it, start producing projects, and get those projects out in the world. You’ll learn what you need to learn along the way.

This interview was edited and condensed.

Related: 

• Slides from Scott Murray’s D3.js tutorial at Strata Santa Clara 2013
• Engaging Audiences with Data Visualization – Webcast
• Interactive Data Visualization for the Web – Book
• Data Journalism Handbook
• Movers and shakers on the data journalism front

Click here for the full visualization.

On the Wikipedia Recent Changes Map, edits can be viewed on 11 Wikipedia language versions, including English, German, Russian and Japanese. Every time an unregistered user edits a Wikipedia entry, his or her IP address is recorded and translated into an approximate location. The about section of the visualization explains that registered users don’t have associated IP information, so registered user edits are not shown on this map.

In a blog post about the map visualization, LaPorte and Hashemi note that the map was built with several libraries and services, including d3, DataMaps, and freegeoip.net, and that the “map listens to live feeds of Wikipedia revisions, broadcast using wikimon.” The map’s code is open source and available on GitHub.

Hat tip to Megan Geuss at Ars Technica and to Emily Badger at The Atlantic for highlighting LaPorte’s and Hashemi’s work.

More visualizations:

• Building collapse rescue efforts
• Visualization of the Week: Every recorded U.S terror attack 1970-2011
• Commuting Paris
• Visualization of the Week: A day in the life of a bus line
• Block-level electricity use in Los Angeles

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

My data is bigger than yours.

These child prodigies of the data scene show great promise but spend a lot of time knocking each other around in the schoolyard. Their egos can sometimes be too big to accept that everybody has their place, and eyeball-seeking media certainly doesn’t help.

POPULAR KID: Look at me! Big data is the hotness!
HADOOP: My data’s bigger than yours!
SCIPY: Size isn’t everything, Hadoop! The bigger they come, the harder they fall. And aren’t you named after a toy elephant?
R: Backward sentences mine be, but great power contains large brain.
EVERYONE: Huh?
SQL: Oh, so you all want to be friends again now, eh?!
POPULAR KID: Yeah, what SQL said! Nobody really needs big data; it’s all about small data, dummy.

The fact is that we’re fumbling toward the adolescence of big data tools, and we’re at an early stage of understanding how data can be used to create value and increase the quality of service people receive from government, business and health care. Big data is trumpeted in mainstream media, but many businesses are better advised to take baby steps with small data.

Data skeptics are not without justification. Our use of “small data” hasn’t exactly worked out uniformly well so far, crude numbers often being misused either knowingly or otherwise. For example, over-reliance by bureaucrats on the results of testing in schools is shaping educational institutions toward a tragically homogeneous mediocrity.

The promise and the gamble of big data is this: that we can advance past the primitive quotas of today’s small data into both a sophisticated statistical understanding of an entire system and insight that focuses down to the level of an individual. Data gives us both telescope and microscope, in detail we’ve never had before.

Inside this tantalizing vision lies many of the debates in today’s data world: the need for highly skilled data scientists to effect this change, and the worry that we’ll inadvertently enslave ourselves to Big Brother, even with the best of intentions.

So, as the data revolution moves forward, it’s important to take the long view. The foment of tools and job titles and algorithms is significant, but ultimately it’s background to our larger purposes as people, businesses and government. That’s one reason why, at O’Reilly, we’ve taken the motto “Making Data Work” for Strata. Data, not technology, is the heartbeat of our world because it relates directly to ourselves and the problems we want to solve.

This is also the reason that the Strata and Hadoop World conferences take a broad view of the subject: ranging from the business topics to the tools and data science. If you talk to Hadoop’s most seasoned advocates, they don’t speak only about the tech; they talk about the problems they’re able to solve. The tools alone are never enough; the real enabler is the framework of people and understanding in which they’re used.

Our mission is to help people make sense of the state of the data world and use this knowledge to become both more competitive and more creative. We believe that’s best served by creating context in which we think about our use of data as well as serving the growing specialist communities in data.

Enjoy the noise and the energy from the growing data ecosystem, but keep your eyes on the problems you want to solve.

The Strata and Hadoop World Call for Proposals is open until midnight EDT, Thursday May 16.

This post was originally published on Radar.

It’s easy (if not eerie) to draw parallels between the Internet boom of the mid 1990s and the Big Data boom of the present day: in addition to the exuberance in the press and the new business models, a particular breed of technical skill became a competitive advantage and a household name. Back then, this was the software developer. Today, it’s the data scientist.

The time in the sun improved software development in some ways, but it also brought its share of problems. Some companies were short on the skill and discipline required to manage custom software projects, and they were equally ill-equipped to discern the true technical talent from the pretenders. That combination led to low-quality software projects that simply failed to deliver business value. (A number of these survive today as “repair-ware” that requires constant, expensive upkeep.)

How will the data science field avoid software development’s pitfalls? (As an aside, we shudder to think what would be the data science equivalent of “repair-ware.”) We started to explore some ideas but realized they were all rooted in education, business value, and openness:

Company leaders must educate themselves in order to understand how data analysis can improve their firm. That knowledge will guide them in building out a data science team and establishing its mission. Leaders otherwise risk trivializing the data scientist role or overindulging in analytics for the sake of analytics.

In turn, data scientists must understand how their work is meant to improve the business. That will serve as their compass when they explore new ideas so they can aim to deliver solid value. Without that guidance, it’s too easy to get stuck in rabbit-holes and yak-shaving.

Both parties must be vigilant of any needless barriers forming around the data science team, especially after the initial novelty fades. Open communication between the data science group and the rest of the business will ensure the former doesn’t land in a separate silo, marginalized out of the company mission.

These ideas may be a start, but are they enough? Probably not. What would you recommend to steer data science clear of pitfalls?

This post originally appeared on http://radar.oreilly.com/.

Related:

• Leading indicators
• Every leader has their “how I got here” story
• Bad Data Handbook – Book

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

But getting good results from machine learning still requires much art, persistence, and even luck. An engineer can’t yet treat ML as just another well-bahaved part of the technology stack. There are many underlying reasons for this, but for the moment I want to focus on how we measure or evaluate ML systems.

Reflecting their academic roots, machine learning methods have traditionally been evaluated in terms of narrow quantitative metrics: precision, recall, RMS error, and so on. The data-science-as-competitive-sport site Kaggle has adopted these metrics for many of its competitions. They are objective and reassuringly concrete.

But scaled, production systems have very different requirements than proof-of-concept academic implementations or prize-winning models. Adopting these metrics from the research world incentivizes one-off, specialized, brittle solutions rather than the reliable, reusable, composable subsystems that form the foundation of good software engineering.

So I’d like to propose some different evaluation criteria for ML systems, with the hope that we raise our collective expectations of what they should provide and, eventually, build them differently.

• Encapsulation & abstraction. An ML system should behave well as a component in a large software system. It should provide an elegant programming interface, use standard data formats, and hide as much complexity as possible from developer.
• Safety & conservatism. An ML system shouldn’t place the burden of avoiding overfitting on the user. It should be willing and able to communicate uncertainty about its results, including the possibility of “shrugging its shoulders” when the data is insufficient.
• Simple and transparent controls. An ML system should expose its configuration and parameters in a clear, transparent way. The user should not have to perform heuristic searches through the parameter space, and there should be no art or mystery involved. The system should require as little tuning as possible from the user, with sensible defaults that handle the common cases.
• Handling messy, real-world data. Real data has duplicates and missing values; is full of noise, errors, and surprises; and is composed of mixed of numerical, categorical, text, geospatial data, etc. ML systems should handle datasets as they are found in the wild, rather than forcing the user to perform significant cleanup and heuristic “feature engineering”.

ML methods and systems that are evaluated and perform well along these lines will help tame the complexity in smart software systems. As a result, more developers will be able to use them successfully and the resulting systems will be more resilient. ML will eventually transition from its current role as the the high-maintenance prima donna of the data stack to a workhorse component.

I would love to see more effort devoted to improvements on these fronts, even if that means less emphasis on capturing incremental accuracy improvements on specific problems. These criteria represent a significant change in focus, though, and they may disturb some ML experts because they embrace a more black-box approach in which the internals of the system are less accessible. I would argue that it is exactly this sort of encapsulation – wisely performed and tastefully engineered – that is needed for machine learning to have the wide impact that, say, relational databases have achieved in the past decades.

Machine Learning Related Resources:

• Machine Learning for Hackers Webcast
• Machine Learning for Hackers – Case Studies
• An Introduction to Machine Learning with Web Data Video

Requires little (to no) coding
The viz tools I currently use require programming skills. Coding means switching back-and-forth between a visual (chart) and text (code). It’s nice¹ to be able to customize charts via code, but when you’re in the exploratory phase not having to think about code syntax is ideal. Plus GUI-based tools allow you to collaborate with many more users.

Includes an expanded set of basic charts
Aside from statistical graphics (line, bar, scatter, histogram, bubble, boxplot,…), these days the ability to visualize hierarchies (treemap), financial (stock charts), longitudinal, geospatial (maps) and network data are essential.

Charts are easy to customize
It should be easy to tweak labels, colors, and other elements. There are times when default labels need to be resized or repositioned, to make them legible. You should also be able to adjust coloring schemes to your liking (colors are usually assigned based on category or, in the case of heat maps, value).

Templates can be created
Once you create a chart with your preferred color and labeling scheme, you should be able to templatize it for future projects. [Ideally templates support rule-based formatting ("if negative, color = red"), but this starts to involve some coding.]

Visual summaries are easy to generate (histograms, association matrix)
You’ll be exploring data sets that contain many observations (rows) and variables (columns). SAS Visual Analytics produces a quick summary (average, min/max, histogram) for each variable and displays the results in a compact, scrollable format. This is done entirely through a GUI and doesn’t require any coding.

Drill-down to source points: identify, isolate, and fix minor data errors
Visual summaries² alert you to potential problems with your data (outliers or errors). A few tools give you the ability to isolate outliers or fix simple data problems through a GUI. More generally, it’s nice to be able to drill-down from the chart to examine (via dynamic rollover or other method) the underlying data.

In-place filtering
While exploring data, you need to be able to quickly filter by value or category – using checkboxes, drop-downs, sliders, …

Support for visual pivoting
Many business analysts are heavy users of pivot tables – a tabular summarization technique found in spreadsheets and reporting tools. Visual pivoting replaces tabular presentation with charts. My first experience using this type of visual exploration was through the Trellis graphs introduced in S/S-Plus. Thanks to Tableau’s easy-to-use interface, this form of visual analysis has become a popular way to explore data.

Support for analytics
Many visualization tools lack analytic capabilities. From simple (error bar, quantiles) to advanced (clustering, forecasting, multidimensional scaling³), analytic tools expand what users can do. Case in point, SAS Visual Analytics has tools for conducting sensitivity analysis and forecasting (GUI-based, no coding required). An example is to take a given time-series (unit sales), plot a forecast of its behavior for the next six time periods, and study how the forecast varies when other key variables (customer satisfaction) change.

Tools for sharing, collaboration, and replication
Several tools let you publish⁴ your static or interactive charts, and some tools even let you subscribe⁵ to the work of other users. For sharing, collaboration, and documentation, it should be possible to annotate your work. Being able to collaborate with others would be nice, at a minimum one should at least be able to copy (and modify) the work of another user.

Big Data: Volume and Variety⁶
A tool should produce charts quickly even when it’s hitting massive data sets. Simply put, it should be truly interactive⁷. Several new tools target larger data sets, some are geared specifically for Hadoop users (a partial list includes Datameer, Platfora, SiSense, and SAS Visual Analytics). But there will be occasions when you’ll be working with small data sets (or be offline). To that end you should be able to visually explore small data (locally using your laptop) without having to connect to a more powerful environment (such as a cluster or a beefy server).

I haven’t come across great viz tools for exploring unstructured data, so I’ll interpret variety in a different way. Co-existence (usually of Hadoop & data warehouses) means data will continue to reside in different systems. Being able to connect to a variety of data sources is essential. (Among startups, Datameer does a good job of this.) Some tools include public data sets (e.g., US Census) and use them to generate examples.

(0) Thanks to Lynn Cherny for reviewing an early draft of this post and for suggesting a few features.
(1) Unless of course you have killer programming tools, a la Bret Victor. You can do some of the things described in the post using ScaleR from Revolution Analytics – but it’s a tool that requires coding in R.
(2) A good example: SAS Visual Analytics displays the number of distinct values of categorical variables. If the number of distinct values is unusually large, you likely have a data quality issue.
(3) Or other tools for handling high-dimensional data sets. Still waiting for a next-gen ggobi!
(4) Datameer takes this a step further: it has an app market.
(5) Some tools even send you realtime alerts when data for charts you’ve subscribed to have changed.
(6) I omitted Velocity – the ability to handle streaming data. I consider that a nice, but not a must-have feature for a visual exploration tool. Having said that, I do think the ability to handle realtime updates is essential when you share your work with others. See (5).
(7) When working with truly massive data sets it’s natural to have some latency. Rather than having users idle while waiting, visual analysis tools should support multiple tabs or workspaces. Most database query tools have this feature: you can work on other queries while a query is still running.

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England