ENTRIES TAGGED "big data trends"

Strata Week: The complexities of forecasting the big data market

IDC forecast underestimates big data growth, EU report sounds an alarm over FISA Amendments Act, and big data's growing role in daily life.

Here are a few stories from the data space that caught my attention this week.

Big data needs a bigger forecast

The International Data Corporation (IDC) released a forecast this week, projecting “the worldwide big data technology and services market will grow at a 31.7% compound annual growth rate (CAGR) — about seven times the rate of the overall information and communication technology (ICT) market — with revenues reaching $23.8 billion in 2016.”

According to the press release, findings from IDC’s research also forecasted specific segment growth, including 21.1% CAGR for services and 53.4% for storage. GigaOm’s Derrick Harris says IDC’s research “only tells part of the story” and that the market will actually be much bigger. For instance, Harris notes that the report doesn’t include analytics software, a critical component of the big data market that the IDC predicts will hit $51 billion by 2016. And what of the outliers? Harris writes:

” .. .where does one include the rash of Software-as-a-Service applications targeting fields from marketing to publishing? They’re all about big data at their core, but the companies selling them certainly don’t fit into the mold of ‘big data’ vendors.”

Harris highlights potential problems the IDC might have in maintaining their report segments — servers, storage, networking, software and services — with more and more cloud providers hosting big data applications and startups offering cloud-based big data services; calculating these revenues will be no easy feat, he writes. You can read Harris’ piece in full at GigaOm.

Read more…

Comment |

Strata Week: Big data’s daily influence

Big data's broad effect on our world, myriad uses for traffic data, and Obama's big data practice vs. policy.

Here are a few stories from the data space that caught my attention this week.

How big data is transforming just about everything

Professor John Naughton took a look this week at how big data is transforming various industries that affect our daily lives.

He highlights finance, of course, which he says has been “pathologically mathematised;” marketing, for which there is more data about human behavior than we’ve ever had; and the very broad category of science. Naughton notes that researchers used to conjure up theories and look to data to support or refute; now, researchers turn to data to find patterns and connections that might inspire new theories. Naughton also looks at medicine, which is just on the brink of delving into the big data realm. He writes:

“Last week’s news about how Cambridge researchers stopped an MRSA outbreak affecting 12 babies in the Rosie Hospital by rapidly sequencing the genome of the bacteria illustrates how medicine has become a data-intensive field. Even a few years ago, the resources required to achieve this would have involved a roomful of computers and upwards of a week.”

Naughton addresses the use of big data in sports as well, speculating that baseball has been the sport most transformed by data. He’ll likely find agreement there. Barry Eggers goes into depth on the dramatic effect big data is having on baseball over at TechCrunch. He notes that simple data analysis of statistics, which baseball has embraced since its beginnings, has evolved into gathering mountains of unstructured data and employing Hadoop to gain new and better insights from data that isn’t part of the structured game information. Eggers writes:

“By having his data scientist run a Hadoop job before every game, [San Francisco Giants manager] Bruce Bochy can not only make an informed decision about where to locate a 3-1 Matt Cain pitch to Prince Fielder, but he can also predict how and where the ball might be hit, how much ground his infielders and outfielders can cover on such a hit, and thus determine where to shift his defense. Taken one step further, it’s not hard to imagine a day where managers like Bochy have their locker room data scientist run real-time, in-game analytics using technologies like Cassandra, Hbase, Drill, and Impala.”

Read more…

Comment |

Strata Week: Data-driven politics

Big data's role in the US presidential election, trends shaping the future of data, and extenuating consequences of the Megaupload case.

Here are a few stories from the data space that caught my attention this week.

Big data, big politics

In the aftermath of the US presidential election, much attention has been focused on Nate Silver’s art of predicting the election results with data. Some looked at it from a coverage angle and how Silver’s work in the spotlight will affect the process of covering elections in the future. John McDermott reports at AdAge that Silver’s work will help shift the “nebulous aspects” of reporting that focus on “feel” and “momentum” to reporting that is anchored in facts and statistics. ComScore analyst Andrew Lipsman said to McDermott, “Now that people have seen [statistics-driven political analysis] proven over a couple of cycles, people will be more grounded in the numbers.”

Which also shows the attention Silver attracted may serve to help democratize big data as well. Tarun Wadhwa reports at Forbes that the power of big data has finally been realized in the US political process:

“Beyond just personal vindication, Silver has proven to the public the power of Big Data in transforming our electoral process. We already rely on statistical models to do everything from flying our airplanes to predicting the weather. This serves as yet another example of computers showing their ability to be better at handling the unknown than loud-talking experts. By winning ‘the nerdiest election in the history of the American Republic,’ Barack Obama has cemented the role of Big Data in every aspect of the campaigning process. His ultimate success came from the work of historic get-out-the-vote efforts dominated by targeted messaging and digital behavioral tracking.”

Michael Scherer at Time has an in-depth look at the role big data and data mining played in Obama’s campaign as well. Campaign manager Jim Messina, Scherer writes, “promised a totally different, metric-driven kind of campaign in which politics was the goal but political instincts might not be the means” and hired dozens of data crunchers to establish an analytics department. The team put together a massive database that merged information from all areas of the campaign — social media, pollsters, consumer databases, fundraisers, etc. — into one central location. Scherer reports: “The new megafile didn’t just tell the campaign how to find voters and get their attention; it also allowed the number crunchers to run tests predicting which types of people would be persuaded by certain kinds of appeals.”

Scherer’s piece is a fascinating look at how data was put to use in a successful presidential campaign. It is this week’s recommended read.

Read more…

Comment |