Many sources of new information are causing exponential growth in data.  The data is primarily a by-product of mobile computing and the plethora of embedded devices e.g. vehicles and TVs. However, there is more than just the data that originates from the devices; there is also exponential growth in data produced by the IT Infrastructure to support new applications targeted at the connected new world.

Meanwhile a diversity of applications results in numerous different data contexts and structures.  For example, many companies are moving to next generation NoSQL stores. Netflix uses Apache Cassandra for their storage. Craigslist uses MongoDB. Companies that host in the Amazon cloud often use DynamoDB. When thinking of Social networking, companies rely on Graph based stores. Meanwhile the majority of the world is still running their ERP and CRM systems in traditional, transactional Relational databases like MS SQL Server and Oracle RDBMs.  But the key here is, is that different types of data are ideally suited for different types of data stores. And generally, when building applications, you optimize for the application’s purpose. It is often when companies bridge into the world of analysis that they run into their Big Data challenges; likewise, when they have accumulated years of data over time within a Data warehouse.

The sheer volume of Data and numerous data structures places a dramatic demand and complexity on analytics.

Can the Enterprise choose to ignore Big Data? No. Companies that choose to ignore this wave will be crushed by innovators that learn to ride it well.

The fact is… expectations of the consumer and business partners have changed. We all choose brands that use predictive analytics to understand what we want and to optimize for our experiences.

Jim Blomo of Yelp does an excellent job describing Big Data challenges. First off, he claims, “If you can’t process your data on one node, then you have a big data problem.” He talks about the concepts behind a Yelply Insight. My version of his story telling…  As a consumer, when I type in Organic on Yelp, it knows that I am not looking for fertilizers but rather restaurants or grocery stores. It knows because I live in King County, WA and our population on the whole here more often than not tends to eat “Farm to Table” and it is not gardening season. On Google, when I type in “Do churros have egg in them” it knows before I finish typing that I am trying to address an allergy question.  We are used to systems predicting our needs, and as a result, our expectations have grown.  We expect companies to answer their most crucial Bid Data questions.

Big Data isn’t just a problem for the pure play internet technology player. Big Data is a problem for click and mortar as well. And in fact, they probably have years and years of data to review. Here are some scenarios I have discussed with UC4 customers:

  • Movie theater company: What candy to stock for a given movie.
  • Large pet store: How to keep millions of fish alive.
  • Large retailer: How to keep their Supply Chain aligned with demand.
  • Media and advertising company: How to not annoy the people they advertise to.
  • Oil and Gas companies: Seismic analysis to prevent disasters and disaster recovery analysis.
  • Food manufacturer: Wants to determine what food is more commonly purchased in a recession.

In each of these scenarios, pieces of data may sit in different silos, a different database, and a different data structure type. Big Data is about pulling together data and handling enormous amounts of unstructured data from different silos for the purpose of analysis.

My team at UC4 is already helping our customers to orchestrate their data for analytics. We help manage the resources to make sure the right data is in the right place at the right time. We are currently The Enterprise Scheduler for Hadoop Map Reduce and numerous large Enterprise companies in both click and mortar.  I am on a mission to learn more so we can help our customers be positioned to catch the wave rather than be caught in the churn.