Balboa Bay Club, Newport Beach, USA
January 28-31, 2013
Despite the hype, Big Data is a business reality. Large scale data-analytics is nothing new, but the exponential rate of data growth and data-processing technologies since the 1980’s has led to a new data frontier. Information is power, and we stand at a time when 90% of the data in the world today was generated in the last two years alone.
But data is only as useful as your ability to access, process and analyse it within a real-time business application. Despite the sheer enormity of the task, off the shelf hardware, open source frameworks, and the processing capacity of the Cloud, mean that Big Data processing is within the cost-effective grasp of the average business. Organizations can now initiate Big Data projects without significant investment in IT infrastructure.
Data has become the competitive weapon of choice for industry and government in today’s “smart” world. To get ahead or even to survive, companies need to embrace data analytics and become data-driven rather than simply using databases to run reports after the fact. A McKinsey report in 2011 predicted a 60% margin increase for retail companies able to harvest the power of Big Data.
Big Data envisages the live streaming of petabytes or zettabytes of information, gleaned from such sources as web logs, social networks, RFID, sensor networks and transaction details, with the intention of gaining a better understanding of customer drivers to achieve competitive advantage. However, simply accessing mass data (volume), from multiple sources (variety) at speed (velocity), does not constitute business advantage. Big Data can equal Big Headache, unless you deploy processes to distil value from the “Data Lake”.
Big Data means that the enterprise architect will have to take a fresh look at the architecture for data acquisition, organization and interpretation. As an enterprise resource, big data can also be a liability if it is interpreted differently by different parts of the enterprise, and, if it results in the proliferation of a new generation of "information silos." New data handling techniques are emerging almost daily to deal with the volume, velocity and variety of the data, but these need to be evaluated, understood and either thrown out or integrated with existing systems. The transformation to the enterprise is significant, but these are exactly the sorts of challenges we need to address today.
Cloud Computing , in particular, is transforming the possibilities for collecting, storing, and processing big data. Cloud services can be the source of massive amounts of information. Cloud storage is a readily expandable means of holding that data. Use of the Cloud infrastructure  to run analysis programs means that the required processing power is always available, as and when it is needed.
Knowing which elements of a huge, heterogeneous collection of data are related to each other is a crucial aspect of the analysis. Semantic  technology has a role to play in enabling these relationships to be discovered and exploited. This is one of the most exciting of the many technical challenges presented by big data.