[Update: check Steve Job’s video at the end of this article]
To most of the discussions happening at the time, big data is another fad. Serious players know that world is now changing, and will be driven completely by data in near future. With consumers (and organizations) going from Mobile Present to Mobile First to Mobile Only, Google Search will not really be the biggest thing happened ever over internet as it is today and the context driven actions done inside the apps will be the future. Human interaction ( Going to Google and searching for How to tame a dragon ) will not be required and behavior will change.
42% of organizations have more than 50TB of data with 6% of them having > 10PB (credits: IOUG Big Data Strategy Survey 2012). This is the data of whats happening with physical and online entities as well as human behavior. This is the data which will tell how much time you spent over apps / websites doing what, how many products were sold in your shop at what time and what else was happening in the world which triggered the sale, how much time you spent doing sex and how vigorous was it (check this ), is human race getting dumber or smarter and with what pace, how many cars crossed a particular spot on road in last n years etc.. Data will be infinite out there. 32% of companies use this data by pre-processing it and then loading data warehouse for integrated analysis and 14% conduct big data analysis separately from traditional enterprise analysis (credits: IOUG Big Data Strategy Survey 2012). The current setup, thus is exactly like what retail commerce was in 1990s. And world is changing so fast that analysis done by Big Data as a separate process might be stale to use it for next day.
BI requires integration of real time big data analytics results integrated into BI systems directly, without any file migrations and use this real time analytical results to take real time actions on user behaviors.
So, if you are planning to integrate big data analysis in your organization or already have a process in action and evaluating integration options, please consider this very seriously and don’t end up in masses for whom big data is a fad. Real time analysis and integration directly to your BI systems is the mantra. Start logging everything and treat data very carefully because it is going to be much more important than all your managers.
PS: I am not against hadoop as that is conventional big data tool which most companies use with data migrations to their BI system. Checkout Slideshare presentation on how facebook used hadoop in real time. Checkout storm too.