Big Data vs SDI? It's not an either/or.
There's a debate in the geo blogosphere about "Big Data" versus spatial data infrastructure (SDI), that is, deriving information from searches of unstructured data versus deriving information from structured data.
Thierry G's blog post last week, "From Lego to Play-Doh: I plead guilty at the altar of Big Data" provides a fine and amusing summary of the argument for big data. The opposing argument is one that spatial data coordinators, data managers and standards organizations have been making for twenty years -- we should all be working with others to develop and use data standards, metadata standards, encoding standards and geospatial software interface standards, including catalog interfaces.
One reason the debate is important is that geospatial semantics is emerging as an important part of the Semantic Web, and the Semantic Web, along with linked data, is going to make search engines even more powerful and useful.
Anyone interested in this debate would have been torn apart trying to decide which sessions to attend at the recent OGC Technical Committee meeting in Boulder. There were discussions about architectures that use both OGC Web Services and linked data; discussions about where data sharing communities should draw the line between sufficiently and insufficiently harmonized ontologies; and discussions about GeoJSON, REST, GeoSPARQL, and whether to develop a specification for writing RESTful specifications. Big data vs. structured data figured in discussions about augmented reality, urban modeling, sensor discovery, and crowdsourcing.
As Carl Reed, the OGC's CTO, says, "It's not an either-or. There are requirements for both, whether they're used independently or in blended approaches. Scientists, researchers, military analysts and others will continue needing to assess the resolution, provenance, accuracy, and other measures of spatial/temporal data quality and fitness for use. At the same time, they, along with many others (business intelligence, GEOINT, social networking, etc.), are grateful for the gooey Big Data tar ball and innovative tools to make inferences and discover trends."