Per head, per crossing, over 170 years, is a heck of a lot of resource wasted. But to change it would entail changing the rail stock of the entire country and realigning about 225,000 km (140,000 mi) of track.

Talk about technical debt.

Data suffers from a similar disconnect. It really wasn’t until the advent of XML 15 years ago that we had an agreed (but not entirely satisfactory) mechanism for storing arbitrary data structures outside the application layer. This is as much a commentary on our technical priorities as it is a social indictment. We are simply not good at playing with others when it comes to data.

Think otherwise? Cast back to last year’s tit-for-tat shortly after Facebook bought Instagram: Twitter responded by disabling Instagram’s ability to import their friends from Twitter, followed by Instagram’s disabling their Twitter cards integration, not as retaliation, of course, but rather because viewing images on Instagram itself is “a better experience” for consumers. A poor excuse really; this was a battle over users-as-data. Most commonly in these battles, the consumer pays the price.

Corporate attitudes towards data are still very focused on this idea of data as proprietary value.  My data is mine  and yours is yours (even if it was created by our users). The Promethean gift of big data tools has not lessened this attitude.  Companies have now the means to handle all the data they gather and keep it even closer to their bosom, and keep the rail gauges disconnected.

This attitude remains prominent but is fundamentally atavistic. The fact is, data is not natively a zero sum game. At Factual we manage data for 65 million small businesses and landmarks in 50 countries. The size and fluidity of this dataset demands the combined attentions of multiple organizations, so we’ve designed technical connectors to help with contributions, diff streams to broadcast edits, and a legal framework that ensures that if you improve our data, we share ownership. At Strata next week, I will be going into detail about why we think a shared approach is the only approach here, and why it is bad principle to keep the passengers waiting.

At the time, most Internet businesses were keen to perpetuate extant geo-local user experiences, rather than seeing them as crude, if profitable, interim steps to something better. Nowhere was this more evident than at my former employer, Yahoo: as late as 2006, investment cases were still being pitched for either maps or local; the bulk of Yahoo’s geo engineering team resided in an entirely different country; and the mobile group had its own building with few, if any, connections to the rest of the company.

Today, entrepreneurs and execs know that local, geo, and mobile are not distinct product groups and technology stacks, but rather essential components of a unified toolset that better connects people with the world around them.

Nothing better reflects the diversity of this geo toolkit than the products on display at the recent Where 2.0 conference. The three-day event embraced the full spectrum of all things spatial: coding, licensing, marketing, daily deals, 3D visualization, and a bit of mapping thrown in for the traditionalists in the audience.

In this context I gave a short presentation on “Big Data, Big Local.” I’ve embedded the slides below. Although I anticipate receiving an award for including Borat, Homer Simpson, Bender, Winston Churchill, porn shops, kittens, surfing, and unicorns coherently in a single presentation on geo — the deck does not stand well on its own. Go figure.

I wanted therefore to provide a brief overview here, and perhaps tackle it more fully in subsequent posts. This is not intended to be a detailed argument, simply an accompanying narrative to the deck.

The general tenor of the discussion focuses upon how we have traditionally employed coordinates as the final word in how we position things and people in the physical world. The problem with this approach is that a coordinate pair lacks context — they are regular and orderly, which is great for machines, but are entirely ambiguous when used to represent more conceptual places like states, cities, stores and neighborhoods. That’s not so good for people. In fact, a single coordinate pair can be used to represent all these places, and their different associations, concurrently.

Social location has moved us toward a new form of positioning: by business or points of interest (POI). These geo-referenced entities exist across the world in an irregular and poorly typed graph, but they are rich with context. People exist and events happen at places, not coordinates — social location applications “snap” us to these nodes, and their context becomes ours.

We are stretching the traditional use of business listings beyond their limits. Looking back, however, you can see how they’ve been continuously extended to fit new models of bringing consumers and businesses closer together. You’ll soon see listings data act as hooks in the mobile payments space, and they will become increasingly pivotal in creating rich topological graphs among people, brands, and physical places.

We can do great things with business listings and POI as geo-commercial beacons, but they come with their own problems, two of which I cover in the deck. The first is a more pedestrian example of name canonicalization: stores in the same retail / restaurant chain are represented today in too many forms. The evidence presented by variants on “Subway Sandwiches” in slide 12 (below) would suggest that current data providers expend comparatively little effort normalizing this content. The problem may exist because this information is currently intended to serve an extant, limited use case — make the name discoverable using free text search and present to users — rather than drive new, data-driven markets.

The second, more important issue is that we are faced on the Internet with a multitude of electronic incarnations of one physical entity. While intended to enhance discoverability across properties, the current situation effects a near-complete inability to understand how users engage electronically with a physical place outside a single caisson such as Facebook, Twitter, or Yelp. This need is increasingly important, because URLs are now often employed as surrogates for a place — they can be resolved to machine-readable attributes and in fact act exactly like nodes in a graph of linked data.

This is a problem that requires solving, and we are tackling it at Factual. I include some topical commentaries from Chris Dixon and Albert Wenger as artillery support in slide 18:

So the problem is that we are collectively going to need this normalization, very soon. I conclude noting that Factual’s duty does not end with the creation of tab-delineated files. Instead, we are looking to create data-centric platforms that instill order on an increasingly chaotic Internet, and ensure that the result is meeting the future business needs of our contemporaries in the local space.

Related:

• Video: Tyler Bell on how big data will shape location
• Why location data is a mess, and what can be done about it
• Where the semantic web stumbled, linked data will succeed
• Toward a local syzygy: aligning deals, check-ins and places

1. Those attending to learn more about data, having recently discovered its potential.
2. Long-time data enthusiasts watching with mixed emotions as their interest is legitimized, experiencing a feeling not unlike when a band that you’ve been following for years suddenly becomes popular.

A data-oriented event like this, outside a specific vertical, could not have drawn a large crowd with this level of interest, even two years ago. Until recently, data was mainly an artifact of business processes. It now takes center stage; organizationally, data has left the IT department and become the responsibility of the product team.

Of course “data,” in its abstract sense, has not changed. But our ability to obtain, manipulate, and comprehend data certainly has. Today, data merits top billing due to a number of confluent factors, not least its increased accessibility via on-demand platforms and tools.   Server logs are the new cash-for-gold: act now to realize the neglected riches within your upper drive bay.

But the idea of “big data” as a discipline, as a conference subject, or as a business, remains in its formative years and has yet to be satisfactorily defined.  This immaturity is perhaps best illustrated by the array of language employed to define big data’s merits and its associated challenges. Commentators are employing very distinct wording to make the ill-defined idea of “big data” more familiar; their metaphors fall cleanly into three categories:

• Natural resources (“the new oil,” “goldrush” and of course “data mining”): Highlights the singular value inherent in data, tempered by the effort required to realize its potential.
• Natural disasters (“data tornado,” “data deluge,” data tidal wave”): Frames data as a problem of near-biblical scale, with subtle undertones of assured disaster if proper and timely preparations are not considered.
• Industrial devices (“data exhaust,” “firehose,” “Industrial Revolution”): A convenient grab-bag of terminologies that usually portrays data as a mechanism created and controlled by us, but one that will prove harmful if used incorrectly.

If Strata’s Birds-of-a-Feather conference sessions are anything to go by, the idea of “big data” requires the definition and scope these metaphors attempt to provide. Over lunch you could have met with like-minded delegates to discuss big data analysis, cloud computing, Wikipedia, peer-to-peer collaboration, real-time location sharing, visualization, data philanthropy, Hadoop (natch’), data mining competitions, dev ops, data tools (but “not trivial visualizations”), Cassandra, NLP, GPU computing, or health care data. There are two takeaways here: the first is that we are still figuring out what big data is and how to think about it; the second is that any alternative is probably an improvement on “big data.”

Strata is about “making data work” — the tenor of the conference was less of a “how-to” guide, and more about defining the problem and shaping the discussion. Big data is a massive opportunity; we are searching for its identity and the language to define it.

But this is not important. While OpenGraph avoids, or outright ignores, many of the problematic issues surrounding semantic annotation (see Alex Iskold’s excellent commentary on OpenGraph here on Radar), criticism focusing only on its technical purity is missing half of the equation. Facebook gets it right where other initiatives have failed. While OpenGraph is incomplete and imperfect, it is immediately usable and sympathetic with extant approaches. Most importantly, OpenGraph is one component in a wider ecosystem. Its deployment benefits are apparent to the consumer and the developer: add the metatags, get the “likes,” know your customers.

Such consumer causality is critical to the adoption of any semantic mark-up. We’ve seen it before with microformats, whose eventual popularity was driven by their ability to improve how a page is represented in search engine listings, and not by an abstract desire to structure the unstructured. Successful adoption will often entail sacrificing standardization and semantic purity for pragmatic ease-of-use; this is where the semantic web appears to have stumbled, and where linked data will most likely succeed.

Linked data intends to make the Web more interconnected and data-oriented. Beyond this outcome, the term is less rigidly defined. I would argue that linked data is more of an ethos than a standard, focused on providing context, assisting in disambiguation, and increasing serendipity within the user experience. This idea of linked data can be delivered by a number of existing components that work together on the data, platform, and application levels:

• Entity provision: Defining the who, what, where and when of the Internet, entities encapsulate meaning and provide context by type. In its most basic sense, an entity is one row in a list of things organized by type — such as people, places, or products — each with a unique identifier. Organizations that realize the benefits of linked data are releasing entities like never before, including the publication of 10,000 subject headings by the New York Times, admin regions and postcodes from the UK’s Ordnance Survey, placenames from Yahoo GeoPlanet, and the data infrastructures being created by Factual [disclosure: I've just signed on with Factual].
• Entity annotation: There are numerous formats for annotating entities when they exist in unstructured content, such as a web page or blog post. Facebook’s OpenGraph is a form of entity annotation, as are HTML5 microdata, RDFa, and microformats such as hcard. Microdata is the shiny, new player in the game, but see Evan Prodromou’s great post on RDFa v. microformats for a breakdown of these two more established approaches.
• Endpoints and Introspection: Entities contribute best to a linked data ecosystem when each is associated with a Uniform Resource Identifier (URI), an Internet-accessible, machine readable endpoint. These endpoints should provide introspection, the means to obtain the properties of that entity, including its relationship to others. For example, the Ordnance Survey URI for the “City of Southampton” is http://data.ordnancesurvey.co.uk/id/7000000000037256. Its properties can be retrieved in machine-readable format (RDF/XML,Turtle and JSON) by appending an “rdf,” “ttl,” or “json” extension to the above. To be properly open, URIs must be accessible outside a formal API and authentication mechanism, exposed to semantically-aware web crawlers and search tools such as Yahoo BOSS. Under this definition, local business URLs, for example, can serve in-part as URIs — ‘view source’ to see the semi-structured data in these listings from Yelp (using hcard and OpenGraph), and Foursquare (using microdata and OpenGraph).
• Entity extraction: Some linked data enthusiasts long for the day when all content is annotated so that it can be understood equally well by machines and humans. Until we get to that happy place, we will continue to rely on entity extraction technologies that parse unstructured content for recognizable entities, and make contextually intelligent identifications of their type and identifier. Named entity recognition (NER) is one approach that employs the above entity lists, which may also be combined with heuristic approaches designed to recognize entities that lie outside of a known entity list. Yahoo, Google and Microsoft are all hugely interested in this area, and we’ll see an increasing number of startups like Semantinet emerge with ever-improving precision and recall. If you want to see how entity extraction works first-hand, check out Reuters-owned Open Calais and experiment with their form-based tool.
• Entity concordance and crosswalking: The multitude of place namespaces illustrates how a single entity, such as a local business, will reside in multiple lists. Because the “unique” (U) in a URI is unique only to a given namespace, a world driven by linked data requires systems that explicitly match a single entity across namespaces. Examples of crosswalking services include: Placecast’s Match API, which returns the Placecast IDs of any place when supplied with an hcard equivalent; Yahoo’s Concordance, which returns the Where on Earth Identifier (WOEID) of a place using as input the place ID of one of fourteen external resources, including OpenStreetMap and Geonames; and the Guardian Content API, which allows users to search Guardian content using non-Guardian identifiers. These systems are the unsung heroes of the linked data world, facilitating interoperability by establishing links between identical entities across namespaces. Huge, unrealized value exists within these applications, and we need more of them.
• Relationships: Entities are only part of the story. The real power of the semantic web is realized in knowing how entities of different types relate to each other: actors to movies, employees to companies, politicians to donors, restaurants to neighborhoods, or brands to stores. The power of all graphs — these networks of entities — is not in the entities themselves (the nodes), but how they relate together (the edges). However, I may be alone in believing that we need to nail the problem of multiple instances of the same entity, via concordance and crosswalking, before we can tap properly into the rich vein that entity relationships offer.

The approaches outlined above combine to help publishers and application developers provide intelligent, deep and serendipitous consumer experiences. Examples include the semantic handset from Aro Mobile, the BBC’s World Cup experience, and aggregating references on your Facebook news feed.

Linked data will triumph in this space because efforts to date focus less on the how and more on the why. RDF, SPARQL, OWL, and triple stores are onerous. URIs, micro-formats, RDFa, and JSON, less so. Why invest in difficult technologies if consumer outcomes can be realized with extant tools and knowledge? We have the means to realize linked data now — the pieces of the puzzle are there and we (just) need to put them together.

Linked data is, at last, bringing the discussion around to the user. The consumer “end” trumps the semantic “means.”

Related:

• Facebook Open Graph: A new take on semantic web
• Linked data is opening 800 years of UK legal info
• Google’s Rich Snippets and the Semantic Web