ENTRIES TAGGED "finance"
"Modelers have a bigger responsibility now than ever before."
People come to data science in all sorts of ways. I happen to be someone who came via finance. Trained as a mathematician, I worked first at a hedge fund and then a financial risk software company, each for about two years, starting in June 2007 and ending in February 2011. If you look at those dates again, you’ll realize I had a front row seat for the financial crisis.
I worked on a few projects in algorithmic trading with Larry Summers at the hedge fund and was invited, along with the other quants at Shaw, to see him discuss the impending doom one evening with Alan Greenspan and Robert Rubin. It honestly kind of surprised and shocked me to see how little they seemed to know, or at least admitted to knowing, about the true situation in the markets. These guys were supposed to be the experts, after all.
A math band-aid will distract us from fixing the problems that so desperately need fixing.
This piece originally appeared on Mathbabe. We’re also including Jordan Ellenberg’s counter-point to Cathy’s original post as well as Cathy’s response to Jordan. All of these pieces are republished with permission.
I just finished reading Nate Silver’s newish book, The Signal and the Noise: Why so many predictions fail – but some don’t.
The good news
First off, let me say this: I’m very happy that people are reading a book on modeling in such huge numbers – it’s currently eighth on the New York Times best seller list and it’s been on the list for nine weeks. This means people are starting to really care about modeling, both how it can help us remove biases to clarify reality and how it can institutionalize those same biases and go bad.
As a modeler myself, I am extremely concerned about how models affect the public, so the book’s success is wonderful news. The first step to get people to think critically about something is to get them to think about it at all.
Moreover, the book serves as a soft introduction to some of the issues surrounding modeling. Silver has a knack for explaining things in plain English. While he only goes so far, this is reasonable considering his audience. And he doesn’t dumb the math down.
In particular, Silver does a nice job of explaining Bayes’ Theorem. (If you don’t know what Bayes’ Theorem is, just focus on how Silver uses it in his version of Bayesian modeling: namely, as a way of adjusting your estimate of the probability of an event as you collect more information. You might think infidelity is rare, for example, but after a quick poll of your friends and a quick Google search you might have collected enough information to reexamine and revise your estimates.)
The bad news
Having said all that, I have major problems with this book and what it claims to explain. In fact, I’m angry.
It would be reasonable for Silver to tell us about his baseball models, which he does. It would be reasonable for him to tell us about political polling and how he uses weights on different polls to combine them to get a better overall poll. He does this as well. He also interviews a bunch of people who model in other fields, like meteorology and earthquake prediction, which is fine, albeit superficial.
What is not reasonable, however, is for Silver to claim to understand how the financial crisis was a result of a few inaccurate models, and how medical research need only switch from being frequentist to being Bayesian to become more accurate. Read more…
How Facebook stacks up against other tech IPOs.
This week's visualization comes from The New York Times and compares the last 30 years of tech IPOs (hint: watch for the big blue dot).
From healthcare to finance to emergency response, data holds immense potential to help citizens and government.
The explosion of big data, open data and social data offers new opportunities to address humanity's biggest challenges. The open question is no longer if data can be used for the public good, but how.
Megaupload's demise raises data questions and Bloomberg opens up its market data interface.
In this week's data news, Megaupload users face data deletion, Bloomberg opens its market data interface and Pentaho changes its licensing for Kettle.
Randall Munroe's new visualization puts money (almost all of it) in perspective.
In an audacious new visualization, Randall Munroe of xkcd takes on money — where it comes from, where it goes and what it buys.
Facebook says we're closer than we thought, Gnip targets finance, and eBay grabs Hunch.
Facebook research questions the "six degrees of separation" rule, Gnip gets into the real-time financial data business, and eBay looks to put Hunch's recommendation engine to use.
Roger Magoulas on data's potential to improve finance systems and create new businesses.
O'Reilly director of market research Roger Magoulas discusses the intersection of big data and finance, and the opportunities this pairing creates for financial experts.
Data could disrupt the stock world, how stolen data is sold, and geography data's predictive power
Will big data kill the stock exchange? That question was recently explored by Andy Kessler. Plus: How recent security breaches could shape the black market and a look at how "island biogeography" predicted Osama Bin Laden's location.
Sentiment analysis gives algorithmic trading an edge
Sorting through thousands of news stories and categorizing information based on mood and tone creates useful data points for financial systems.