While social media placed graph data on the radar of many companies, similar data sets can be found in many domains including the life and health sciences, security, and financial services. Graph data is different enough that it necessitates special tools and techniques. Because tools were a bit too complex for casual users, in the past this meant graph data analytics was the province of specialists. Fortunately graph data is an area that has attracted many enthusiastic entrepreneurs and developers. The tools have improved and I expect things to get much easier for users in the future. A great place to learn more about tools for graph data, is at the upcoming GraphLab Workshop (on July 1st in SF).

Data wrangling: creating graphs
Before you can take advantage of the other tools mentioned in this post, you’ll need to turn your data (e.g., web pages) into graphs. GraphBuilder is an open source project from Intel, that uses Hadoop MapReduce¹ to build graphs out of large data sets. Another option is the combination of GraphX/Spark described below. (A startup called Trifacta is building a general-purpose, data wrangling tool, that could help as well. )

Data management and search
Once you have a graph, there are many options for how to store it. The choice of database largely depends on amount of data (# of nodes, edges, along with the size of data associated with them), the types of tasks (pattern-matching and search, analytics), and workload. In the course of evaluating alternatives to MySQL (for storing social graph data), Facebook’s engineering team developed and released Linkbench – a data set that can be used to study how graph databases handle production workloads.

Most graph databases (such as Neo4j², AllegroGraph, Yarcdata, and InfiniteGraph) come with tools for facilitating and speeding up search – Neo4j comes with a simple query language (Cipher) for search, other graph databases support SPARQL. The Titan distributed graph database supports different storage engines (including HBase and Cassandra) and comes with tools for search and traversal (based on Lucene and Gremlin). Used by Twitter to store graph data, FlockDB targets operations involving adjacency lists.

Among Hadoop users HBase is a popular option for storing graph data. Hadapt’s analytic platform³ integrates Apache Hadoop and SQL, and now also supports graph analysis.

Graph-parallel frameworks: Pregel, PowerGraph, and GraphX
BSP is a parallel computing model that has inspired many graph analytics tools. Just like Hadoop’s map and reduce, Pregel⁴, Giraph and Pregelix, come with primitives that let neighboring nodes send/receive messages to one another, or change the state of a node (based on the state of its neighboring nodes). Efficient graph algorithms are a sequence of iterations built from such primitives. GraphLab uses similar primitives (called PowerGraph) but allows for asynchronous iterative computations, leading to an expanded set of (potentially) faster algorithms.

GraphX is a new, fault-tolerant, framework that runs within Spark. Its core data structure is an immutable graph⁵ (Resilient Distributed Graph – or RDG), and GraphX programs are a sequence of transformations on RDG’s (with each transformation yielding a new RDG). Transformations on RDG’s can affect nodes, edges, or both (depending on the state of neighboring edges and nodes). GraphX greatly enhances productivity by simplifying a range of tasks (graph loading, construction, transformation, and computations). But it does so at the expense of performance: early prototype algorithms written in GraphX were slower⁶ than those written in GraphLab/PowerGraph.

Machine-learning and analytics
Machine-learning tools that target graph data lead to familiar applications such as detecting influential users (PageRank) and communities, fraud detection, and recommendations (collaborative filtering is popular among GraphLab users). Moreover techniques developed in one domain are often reused in other settings. Besides GraphLab, distributed analytics have been implemented in Giraph, GraphX, Faunus, and Grappa. In addition, graph databases like Neo4j and Yarcdata come with some analytic capabilities. As I noted in a recent post, open source, single-node systems like Twitter’s Cassovary⁷ are being used for computations involving massive graphs.

Visualization
When you’re dealing with large graphs, being able to zoom in/out helps with clutter, but so do clever layout algorithms. Popular tools for visualizing nodes and edges include Gephi and GraphViz. Users who want to customize their graphs turn to packages like d3.

(1) I would love to see a version of GraphBuilder that’s built on top of Spark.
(2) Many of these systems are quite efficient. For example a single instance of Neo4j can handle very large graphs (“into the tens of billions of nodes/ relationships/ properties”).
(3) Note that using standard Hadoop for graph processing may not be the most efficient option. This talk by Hadapt co-founder Daniel Abadi describes an advanced approach to graph analysis using Hadoop.
(4) Related frameworks include GoldenOrb and Hama.
(5) Resilient Distributed Graphs (RDG) extend Spark’s Resilient Distributed Dataset (RDD).
(6) As the developers of GraphX note: “We emphasize that it is not our intention to beat PowerGraph in performance. … We believe that the loss in performance may, in many cases, be ameliorated by the gains in productivity achieved by the GraphX system. … It is our belief that we can shorten the gap in the near future, while providing a highly usable interactive system for graph data mining and computation”
(7) On the plus side, being single-node means Cassovary doesn’t have to deal with finding the optimal way to partition a graph. On the other hand, it is limited to graphs that fit in the memory of a server – a limitation it alleviates through the use of efficient data structures.

Requires little (to no) coding
The viz tools I currently use require programming skills. Coding means switching back-and-forth between a visual (chart) and text (code). It’s nice¹ to be able to customize charts via code, but when you’re in the exploratory phase not having to think about code syntax is ideal. Plus GUI-based tools allow you to collaborate with many more users.

Includes an expanded set of basic charts
Aside from statistical graphics (line, bar, scatter, histogram, bubble, boxplot,…), these days the ability to visualize hierarchies (treemap), financial (stock charts), longitudinal, geospatial (maps) and network data are essential.

Charts are easy to customize
It should be easy to tweak labels, colors, and other elements. There are times when default labels need to be resized or repositioned, to make them legible. You should also be able to adjust coloring schemes to your liking (colors are usually assigned based on category or, in the case of heat maps, value).

Templates can be created
Once you create a chart with your preferred color and labeling scheme, you should be able to templatize it for future projects. [Ideally templates support rule-based formatting ("if negative, color = red"), but this starts to involve some coding.]

Visual summaries are easy to generate (histograms, association matrix)
You’ll be exploring data sets that contain many observations (rows) and variables (columns). SAS Visual Analytics produces a quick summary (average, min/max, histogram) for each variable and displays the results in a compact, scrollable format. This is done entirely through a GUI and doesn’t require any coding.

Drill-down to source points: identify, isolate, and fix minor data errors
Visual summaries² alert you to potential problems with your data (outliers or errors). A few tools give you the ability to isolate outliers or fix simple data problems through a GUI. More generally, it’s nice to be able to drill-down from the chart to examine (via dynamic rollover or other method) the underlying data.

In-place filtering
While exploring data, you need to be able to quickly filter by value or category – using checkboxes, drop-downs, sliders, …

Support for visual pivoting
Many business analysts are heavy users of pivot tables – a tabular summarization technique found in spreadsheets and reporting tools. Visual pivoting replaces tabular presentation with charts. My first experience using this type of visual exploration was through the Trellis graphs introduced in S/S-Plus. Thanks to Tableau’s easy-to-use interface, this form of visual analysis has become a popular way to explore data.

Support for analytics
Many visualization tools lack analytic capabilities. From simple (error bar, quantiles) to advanced (clustering, forecasting, multidimensional scaling³), analytic tools expand what users can do. Case in point, SAS Visual Analytics has tools for conducting sensitivity analysis and forecasting (GUI-based, no coding required). An example is to take a given time-series (unit sales), plot a forecast of its behavior for the next six time periods, and study how the forecast varies when other key variables (customer satisfaction) change.

Tools for sharing, collaboration, and replication
Several tools let you publish⁴ your static or interactive charts, and some tools even let you subscribe⁵ to the work of other users. For sharing, collaboration, and documentation, it should be possible to annotate your work. Being able to collaborate with others would be nice, at a minimum one should at least be able to copy (and modify) the work of another user.

Big Data: Volume and Variety⁶
A tool should produce charts quickly even when it’s hitting massive data sets. Simply put, it should be truly interactive⁷. Several new tools target larger data sets, some are geared specifically for Hadoop users (a partial list includes Datameer, Platfora, SiSense, and SAS Visual Analytics). But there will be occasions when you’ll be working with small data sets (or be offline). To that end you should be able to visually explore small data (locally using your laptop) without having to connect to a more powerful environment (such as a cluster or a beefy server).

I haven’t come across great viz tools for exploring unstructured data, so I’ll interpret variety in a different way. Co-existence (usually of Hadoop & data warehouses) means data will continue to reside in different systems. Being able to connect to a variety of data sources is essential. (Among startups, Datameer does a good job of this.) Some tools include public data sets (e.g., US Census) and use them to generate examples.

Update (5/23/2012): A recent conversation with Sara Alspaugh inspired the following feature.
Recommend items worth investigating
When you first encounter a data set with lots of variables, it can be a bit overwhelming. Using simple pattern recognition techniques, tools should surface associations/patterns/anomalies worth investigating. Some tools in finance do this for time-series: trends, new highs/lows, and forecasts are drawn automatically. I’d love to have suggestions for what visual pivots (trellis charts) to draw.

(0) Thanks to Lynn Cherny for reviewing an early draft of this post and for suggesting a few features.
(1) Unless of course you have killer programming tools, a la Bret Victor. You can do some of the things described in the post using ScaleR from Revolution Analytics – but it’s a tool that requires coding in R.
(2) A good example: SAS Visual Analytics displays the number of distinct values of categorical variables. If the number of distinct values is unusually large, you likely have a data quality issue.
(3) Or other tools for handling high-dimensional data sets. Still waiting for a next-gen ggobi!
(4) Datameer takes this a step further: it has an app market.
(5) Some tools even send you realtime alerts when data for charts you’ve subscribed to have changed.
(6) I omitted Velocity – the ability to handle streaming data. I consider that a nice, but not a must-have feature for a visual exploration tool. Having said that, I do think the ability to handle realtime updates is essential when you share your work with others. See (5).
(7) When working with truly massive data sets it’s natural to have some latency. Rather than having users idle while waiting, visual analysis tools should support multiple tabs or workspaces. Most database query tools have this feature: you can work on other queries while a query is still running.

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

Scaling up machine-learning: Find efficient algorithms
Faced with having to crunch through a massive data set, the first thing a machine-learning expert will try to do is devise a more efficient algorithm. Some popular approaches involve sampling, online learning, and caching. Parallelizing an algorithm tends to be lower on the list of things to try. The key reason is that while there are algorithms that are embarrassingly parallel (e.g., naive bayes), many others are harder to decouple. But as I highlighted in a recent post, efficient tools that run on single servers can tackle large data sets. In the machine-learning context recent examples² of efficient algorithms that scale to large data sets, can be found in the products of startup SkyTree.

Use the same approach to scale real-time analytics: consider streamdrill
One can use the same strategy for real-time analytics: before opting for distributed systems that require many nodes before benefits accrue, evaluate simpler solutions that implement efficient algorithms for answering common questions. A new system called streamdrill fits this profile (coincidentally it was designed by machine-learning researchers). It’s a single-server system³ optimized to answer “top k questions” against massive amounts of structured and unstructured data (its capable of ingesting Twitter’s firehose). Specifically you can use streamdrill to count and identify the most active entities and events, over different time windows⁴. This leads to important stream mining applications such as identifying trends (discover “trending topics”), anomaly detection (“alerts”), correlations, clustering, and classification. Streamdrill also includes⁵ a feature (called traces) that lets users easily run queries over extended time windows.

Unlock your data sooner
The simplest and quickest way to mine your event data stream is to deploy efficient algorithms designed to answer key questions at scale. Streamdrill offers a quick path to real-time intelligence through a combination of ease-of-deployment (single server), streaming algorithms (heavy-hitters, anomaly detection, approximate percentiles, and more), and scale (intelligent resource management). As an added bonus streamdrill’s suite of efficient⁶ algorithms for real-time analysis can easily be accessed through a REST API.

Related posts:

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

In-memory storage system from UC Berkeley’s AMPLab
The team behind the BDAS stack recently released a developer preview of Tachyon – an in-memory, distributed, file system. The current version of Tachyon was written in Java and supports Spark, Shark, and Hadoop MapReduce. Working data sets can be loaded into Tachyon where they can be accessed at memory speed, by many concurrent users. Tachyon implements the HDFS FileSystem interface for standard file operations (such as create, open, read, write, close, and delete).

Workloads with working sets fitting into cluster memory can derive the most benefits from Tachyon. But as I pointed out in a recent post, in many companies working data sets are in the gigabytes or terabytes. Such data sizes are well within the range of a system like Tachyon.

High-throughput writes and fault-tolerance: Bounded recovery times using asynchronous checkpointing and lineage
A release slated for the summer will include features² that enable data sharing (users will be able to do memory-speed writes to Tachyon). With Tachyon, Spark users will have for the first time, a high throughput way of reliably sharing files with other users. Moreover, despite being an external storage system Tachyon is comparable to Spark’s internal cache. Throughput tests on a cluster showed that Tachyon can read 200x and write 300x faster than HDFS. (Tachyon can read and write 30x faster than FDS’ reported throughput.)

Similar to the resilient distributed datasets (RDD) fundamental within Spark, fault-tolerance in Tachyon also relies³ on the concept of lineage – logging the transformations used to build a dataset, and using those logs to rebuild datasets when needed. Additionally as an external storage system Tachyon also keeps tracks of binary programs used to generate datasets, and the input datasets required by those programs.

Tachyon achieves higher throughput because it stores a copy of each “write” to the memory of a single node, without waiting for it to be written to disk or replicated. (Replicating across a network is much slower than writing to memory.) Checkpointing is done asynchronously, with the latest generated dataset checkpointed, each time a checkpoint is done being saved.

High-performance data sharing between different data science frameworks
Tachyon will let users share data across frameworks and perform read/write operations at memory-speed. In particular a system like Tachyon will appeal to data scientists who rely on workflows that use a variety of tools: their resulting data analytic pipelines will run much faster. To that end, its creators simulated a real-world⁴ data pipeline comprised of 400 steps, and found that Tachyon resulted in “17x end-to-end latency improvements”.

Tachyon uses memory (instead of disk) and recomputation (instead of replication) to produce a distributed, fault-tolerant, and high-throughput file system. While it initially targets data warehouse and analytics (Shark, Spark, Hadoop MapReduce), I’m looking forward to seeing other popular data science tools support this interesting new file system.

Related posts:

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

A workflow tool for data analysts: Chronos from airbnb
A raw bash scheduler written in Scala, Chronos is flexible, fault-tolerant², and distributed (it’s built on top of Mesos). What’s most interesting is that it makes the creation and maintenance of complex workflows more accessible: at least within airbnb, it’s heavily used by analysts.

Job orchestration and scheduling tools contain features that data scientists would appreciate. They make it easy for users to express dependencies (start a job upon the completion of another job), and retries (particularly in cloud computing settings, jobs can fail for a variety of reasons). Chronos comes with a web UI designed to let business analysts³ define, execute, and monitor workflows: a zoomable DAG highlights failed jobs and displays stats that can be used to identify bottlenecks. Chronos lets you include asynchronous jobs – a nice feature for data science pipelines that involve long-running calculations. It also lets you easily define repeating jobs over a finite time interval, something that comes in handy for short-lived⁴ experiments (e.g. A/B tests or multi-armed bandits).

The unreasonable effectiveness of data: model selection & deployment
By enabling airbnb analysts to take prototype workflows and easily deploy them to production, Chronos taps into a need that other⁵ tools are beginning to address. Startup Alpine Data Labs provides a GUI tool that lets business analysts define and manage, multi-step analytic workflows.

The landmark paper by Banko and Brill hinted that with massive amounts of data, the choice of models become less important. Thus tools that let you easily deploy analytic models at scale, become just as important as specific algorithms. A noteworthy project out of UW-Madison – Hazy – seeks to simplify the deployment and maintenance of analytic models.

“The next breakthrough in data analysis may not be in individual algorithms, but in the ability to rapidly combine, deploy, and maintain existing algorithms.”
Hazy: Making it Easier to Build and Maintain Big-data Analytics

Related posts:

• Data Science tools: Are you “all in” or do you “mix and match”?
• Data Science tools: Fast, easy to use, and scalable

(1) Data scientists may build prototypes, but repeatable pipelines tend to be the domain of data engineers.
(2) As with other workflow tools, Chronos includes alerts (for job deletes and failure after specified # of retries).
(3) Chronos jobs are defined via a web GUI, other tools require the creation/maintenance of “configuration” files. Chronos also comes with a simple REST API.
(4) Chronos uses ISO8601 which makes it easy to define repeating intervals and configure jobs that repeat over a time period, after which they get deleted.
(5) Other companies include Trifacta, Ufora, and BI tools Datameer and Platfora.

Around the same time a team of researchers from Microsoft went a step further. They released a study that concluded that for many data processing tasks, scaling by using single machines with very large memories is more efficient than using clusters. They found two clusters devoted to analytics (one at Yahoo and another at Microsoft) had median job input sizes under 14 GB, while 90% of jobs on a Facebook cluster had input sizes under 100 GB. In addition, the researchers noted that

… for workloads that are processing multi-gigabytes rather than terabyte+ scale, a big-memory server may well provide better performance per dollar than a cluster.

One year later: some single server systems that tackle big data
BI company SiSense won the Strata Startup Showcase audience award with Prism – a 64-bit software system that can handle a terabyte of data on a machine with only 8GB of RAM. Prism² relies on disk for storage, moves data to memory when needed, and also takes advantage of the CPU (L1/L2/L3 cache). It also comes with a column store and visualization tools that lets it easily scale to a hundred terabytes.

Late last year I wrote about GraphChi, a graph processing system that can process graphs with billions of edges with a laptop. It uses a technique called parallel sliding windows to process edges efficiently from disk. GraphChi is part of GraphLab, an open source project that comes with toolkits for collaborative filtering³, topic models, and graph processing.

Cassovary is an open source, graph processing system from Twitter. It’s designed to tackle graphs that fit in the memory of a single machine – nevertheless its creators believe that the use of space efficient data structures makes it a viable system for “most practical graphs”. In fact it already powers a system familiar to most Twitter users: WTF (who to follow) is a recommendation service that suggests users with shared interests and common connections.

Next-gen SSD’s: narrowing the gap between main memory and storage
GraphChi and SiSense scale to large data sets by using disk as primary storage. They speed up performance using techniques that rely on hardware optimization (SiSense) or sliding windows (GraphChi). As part of our investigation into in-memory data management systems, the potential of next-generation SSD’s has come to our attention. If they live up to the promise of having speeds close to main memory, many more single-server systems for processing and analyzing big data will emerge.

(1) SAP blog post: “Even with rapid growth of data, 95% of enterprises use between 0.5TB – 40 TB of data today.”
(2) Prism uses a hierarchy: Accessing data from CPU is faster compared to main memory, which in turn is faster than accessing it from disk.
(3) GraphLab’s well-regarded collaborative filtering library has been ported to GraphChi.

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

Time-series and big data:
Over the last six months I’ve been encountering more data scientists (outside of finance) who work with massive amounts of time-series data. The rise of unstructured data has been widely reported, the growing importance of time-series much less so. Sources include data from consumer devices (gesture recognition & user interface design), sensors (apps for “self-tracking”), machines (systems in data centers), and health care. In fact some research hospitals have troves of EEG and ECG readings that translate to time-series data collections with billions (even trillions) of points.

Search and machine-learning at scale:
Before doing anything else, one has to be able to run queries at scale. Last year I wrote about a team of researchers at UC Riverside who took an existing search algorithm (dynamic time-warping²) and got it to scale to time-series with trillions of points. There are many potential applications of their research, one I highlighted is from health care:

… a doctor who needs to search through EEG data (with hundreds of billions of points), for a “prototypical epileptic spike”, where the input query is a time-series snippet with thousands of points.

As the size of data grows, the UCR dynamic time-warping algorithm takes time to finish (it takes a few hours for time-series with trillions of points). In general (academic) researchers who’ve spent weeks or months collecting data are fine waiting a few hours for a pattern recognition algorithm to finish. But users who come from different backgrounds (e.g. web companies) may not be as patient. Fortunately “search” is an active research area and faster (distributed) pattern recognition systems will likely emerge soon.

Once you scale up search, other interesting problems can be tackled. The UCR team is using their dynamic time-warping algorithm in tasks like classification, clustering, and motif³ discovery. Other teams are investigating techniques from signal-processing, pattern recognition, and trajectory tracking.

Some data management tools that target time-series:
One of the more popular sessions at last year’s HBase Conference was on OpenTSDB, a distributed, time series database built on top of HBase. It’s used to store and serve time series metrics, and comes with tools (based on GNUPlot) for charting. Originally named OpenTSDB2, KairosDB was written primarily for Cassandra (but also works with HBase). OpenTSDB emphasizes tools for readying data for charts (interpolating to fill in missing values), KairosDB distinguishes between data and the presentation of data.

Startup TempoDB offers a reasonably priced, cloud-based service for storing, retrieving, and visualizing time-series data. Still a work in progress SciDB is an open source database project, designed specifically for data intensive science problems. The designers of the system plan to make time-series analysis easy to express within SciDB.

(1) I worked on trading strategies for derivatives, portfolio & risk management, and option pricing.
(2) From my earlier post: In a recent paper, the UCR team noted that “… after an exhaustive literature search of more than 800 papers, we are not aware of any distance measure that has been shown to outperform DTW by a statistically significant amount on reproducible experiments”.
(3) Motifs are similar subsequences of a long time series; shapelets are time series primitives that can be used to speed up automatic classification (by reducing the number of “features”).

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

Multiple tools and languages can impede reproducibility and flow
On the other end of the spectrum are data scientists who mix and match tools, and use packages and frameworks from several languages. Depending on the task, data scientists can avail of tools that are scalable, performant, require less² code, and contain a lot of features. On the other hand this approach requires a lot more context-switching, and extra effort is needed to annotate long workflows. Failure to document things properly makes it tough to reproduce³ analysis projects, and impedes knowledge transfer⁴ within a team of data scientists. Frequent context-switching also makes it more difficult to be in a state of flow, as one has to think about implementation/package details instead of exploring data. It can be harder to discover interesting stories with your data, if you’re constantly having to think about what you’re doing. (It’s still possible, you just have to concentrate a bit harder.)

Some tools that cover a range of data science tasks
More tools that integrate different data science tasks are starting to appear. SAS has long provided tools for data management and wrangling, business intelligence, visualization, statistics, and machine-learning. For massive⁵ data sets, a new alternative to SAS is ScaleR from Revolution Analytics. Within ScaleR programmers use R for data wrangling (rxDataStep), data visualization (basic viz functions for big data), and statistical analysis (it comes with a variety of scalable statistical algorithms).

Startup Alpine Data Labs lets users connect to a variety of data sources, manage their data science workflows, and access a limited set of advanced algorithms. Upstart BI vendors Datameer and Platfora provide data wrangling and visualization tools. Datameer also provides easy data integration to a variety of structured/unstructured data sources, analytic functions and PMML to execute predictive analytics. The release of MLbase this summer adds machine-learning to the BDAS/Spark stack – which currently covers data processing, interactive (SQL) and streaming analysis.

What does your data science toolkit look like? Do you mainly use one stack or do you tend to “mix and match”?

(1) This usually includes matplotlib or Bokeh, Scikit-learn, Pandas, SciPy, and NumPy. But as a general purpose language, you can even use it for data acquisition (e.g. web crawlers or web services).
(2) An example would be using R for viz or stats.
(3) This pertains to all data scientists, but is particularly important to those among us who use a wide variety of tools. Unless you document things properly, when you’re using many different tools the results of very recent analysis projects can be hard to redo.
(4) Regardless of the tools you use, everything starts with knowing something about the lineage and provenance of your data set – something Loom attempts to address.
(5) A quick and fun tool for exploring smaller data sets is the just released SkyTree Adviser. After users perform data processing and wrangling in another tool, SkyTree Adviser exposes machine-learning, statistics, and statistical graphics through an interface that is accessible to business analysts.

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

The Python data community is well-organized:
Besides conferences (PyData, SciPy, EuroSciPy), there is a new non-profit (NumFOCUS) dedicated to supporting scientific computing and data analytics projects. The list of supported projects are currently Python-based, but in principle NumFOCUS is an entity that can be used to support related efforts from other communities.

It’s getting easier to use the Python data stack:
There are tools that facilitate the dissemination and sharing of code and programming environments. IPython² notebooks allow Python code and markup in the same document. Notebooks are used to record and share complex workflows and are used heavily for (conference) tutorials. As the data stack grows, one of the major pain points is getting all the packages to work properly together (version compatibility is a common issue). In particular setting up environments were all the pieces work together can be a pain. There are now a few solutions that address this issue: Anaconda and cloud-based Wakari from Continuum Analytics, and cloud computing platform PiCloud.

There are many more visualization tools to choose from:
The 2D plotting tool matplotlib is the first tool enthusiasts turn to, but as I learned at the conference, there are a number of other options available. Continuum Analytics recently introduced companion packages Bokeh and Bokeh.js that simplify the creation of static and interactive visualizations using Python. In particular Bokeh is the equivalent of ggplot (it even has an interface that mimics ggplot). With Nodebox, programmers use Python code to create sketches and interactive visualizations that are similar to those produced by Processing.

Large-scale data processing and wrangling tools have improved:
Pandas and PyTables are already popular, and there was very strong interest in the forthcoming Blaze project at the conference. Other options include the Disco Project, a data processing platform that includes an implementation of Map/Reduce, and PySpark, the Python API for the Spark data analytics framework.

There are viable tools for large-scale data analytics:
Scikit-learn (machine-learning library) and scikit-image (image processing) are used by many academic research groups and companies. Both have extensive libraries of algorithms, and come with lots of examples to help users get started³. Another tool written in Python focuses on deployment⁴: Augustus is an open source system for building and scoring, scalable data mining and statistical algorithms. Augustus produces and consumes PMML, and includes components for simple data wrangling (users can embed Python code for data processing in their PMML files).

In addition, new tools like H20 and wise.io plan to make their massively scalable algorithms accessible via Python. Frameworks that expose distributed algorithms to Python programmers include GraphLab (Python/Jython interface) and Spark (algorithms⁵ in Scala that are accessed via PySpark). Finally, there are also tools that let Python programmers target GPU’s for parallel programming: NumbaPro and PyCUDA

(1) The event drew about 300 attendees and is one of three PyData conferences scheduled this year (Boston in the summer, NYC in the fall).
(2) The new language Julia, and IPython are starting to work well together.
(3) In practice these tools let Python programmers efficiently develop prototypes that are later re-implemented (in another language) and optimized before being deployed to production.
(4) Data scientists tend not to focus on the deployment and maintenance of “models”. The Hazy project may change this mindset.
(5) A suite of distributed algorithms will be available upon the release of MLbase on Spark.

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

Spark is attracting attention
I’ve written numerous times about components of the Berkeley Data Analytics Stack (Spark, Shark, MLbase). Two Spark-related sessions at Strata were packed (slides here and here) and I talked to many people who were itching to try the BDAS stack. Being able to combine batch, real-time, and interactive analytics in a framework that uses a simple programming model is very attractive. The release of version 0.7 adds a Python API to Spark’s native Scala interface and Java API.

SQL is alive and well
Impala’s well-received launch at Strata NYC last fall confirmed the strong interest in interactive analytics (adhoc query-and-response) within the Hadoop ecosystem. The list of solutions for querying big data (stored in HDFS) continues to grow with CitusDB and Pivotal HD, joining Impala, Shark, Hadapt, and cloud-based alternatives BigQuery, Redshift, and Qubole. I use Shark and have been impressed by its speed and ease of use. (I’ve heard similar things about Impala’s speed recently.)

Business Intelligence reboot (again)
QlikTech and Tableau had combined 2012 revenues of more than $450M. They are easy-to-use analysis tools, that let users visually explore data, and share charts and dashboards. Both use in-memory technologies to speed up query response and visualization rendering times. Both run only on MS-Windows.

Startups that draw inspiration from these two successful companies are targeting much larger data sets – in the case of Datameer and Platfora, and Karmasphere, massive data sets stored in HDFS. Platfora has been generating buzz with its fast in-memory, columnar data store, custom HTML5 visualization package, and emphasis on tools that let users interact with massive data. Datameer continues to quietly rack up sales – it closed 2012 with more than $10M in revenues. Strata Startup showcase (audience choice) winner SiSense, offers a hardware optimized business analytics platform that delivers fast processing times by efficiently utilizing disk, RAM, and CPU.

Scalable machine-learning and analytics are going to get simpler¹
H20 is a new, open source, machine-learning platform from 0xdata. It can use data stored in HDFS or flat files and comes with a few distributed algorithms (random forests, GLM, and a few others). H20 also has tools for rudimentary exploratory data analysis and wrangling. Users can navigate the system using a web browser or a command-line interface. Just like Revolution Analytics’ ScaleR, users can interact with H20 using R code (limited to the subset of models and algorithms available). H20 is also available via REST/JSON interfaces.

What I found intriguing² was SkyTree’s acquisition of AdviseAnalytics – a desktop software product designed to make statistical data analysis accessible. (AdviseAnalytics was founded by Leland Wilkinson, creator of the popular Systat software package and author of the Grammar of Graphics.) The system now called SkyTree Adviser, provides a GUI that emphasizes tasks (cluster, classify, compare, etc.) over algorithms. In addition it produces results that include short explanations of the underlying statistical methods (power users can opt for concise results similar to those produced by standard statistical packages). Finally SkyTree Adviser users benefit from the vast number of algorithms available – the system uses ensembles, or finds optimal algorithms. (The MLbase optimizer will perform the same type of automatic “model selection” for distributed algorithms.)

SkyTree now offers users an easy-to-use tool for analytic explorations over medium sized data sets (SkyTree Adviser), and a server product for building and deploying algorithms against massive amounts of data. Throw in MLbase and Hazy, and I can see the emergence of several large-scale machine-learning tools³ for non-technical users.

Reproducibility of Data Science Workflows

Source: “Computational Information Design” by Ben Fry

Data scientists tend to use many tools and the frequent context-switching is a drag on their productivity. An important side-effect is that it’s often challenging to document and reproduce analysis projects that involve many steps and tools.

Data scientists who rely on the Python data stack (Numpy, SciPy, Pandas, nltk, etc.) should check out Wakari from Continuum Analytics. It’s a cloud-based service that takes care of many details including data management, package and version management, while insulating the user from the intricacies of Amazon Web Services.

Loom is a just-released, data management system that initially targets users of Hadoop (and R). By letting users track lineage and data provenance, Loom makes it easier to recreate multi-step data analysis projects.

(1) In previous posts I detailed why I like GraphChi/GraphLab and why I’m excited about MLbase. Two other open source projects are worth highlighting: Mahout has many more algorithms but VW generates more enthusiastic endorsements from users I’ve spoken with. However the sparse documentation and the many command-line options makes it tough to get going in VW. (A forthcoming O’Reilly book should make VW more accessible.) For users who want to roll their own, I’ve written a few simple distributed, machine-learning algorithms in Spark, and found it quite fast for batch training and scoring.
(2) Update (3/18/2013): I removed this from the original version of this post, and re-inserted it following the official launch of SkyTree Adviser.
(3) BI tools like Datameer already come with simple analytic functions available through a GUI.