Companies that have successfully invested in big data analytics are unlocking its business potential, and new technologies are storing and analysing data ever more affordably.
Recent astronomic increases in the rate of data creation have left companies around the world grappling with the same conundrum: how to make what is immense, slow and unwieldy into what is small, nimble and usable. In 2011, it was said that an estimated 90% of the world’s data (from the WWW and machine generated data from network nodes and applications) was created over the past two years. The world’s data is reportedly doubling every two years and global annual data creation is set to leap from 1.2 zettabytes in 2012 to 35 zettabytes in 2020, according to IDC’s 2011 Digital Universe Report. This data explosion, dubbed Big Data, is proving both a stimulating technical challenge and exciting business opportunity. Companies that have successfully invested in big data analytics are unlocking its business potential, and new technologies are storing and analysing data ever more affordably.
Database vendors have exploited these advances by developing new high speed platforms. 2011 saw big data technologies hit the mainstream, with Oracle, IBM, SAP, Teradata and others starting to integrate Hadoop into their enterprise data solutions. Hadoop is a software framework which supports data intensive, distributed applications under a free licence. It enables these applications to work with thousands of nodes and petabytes of data. Originating in the mid-2000s through technologies from Google (MapReduce), Yahoo and others, Hadoop is now central to the big data initiatives of enterprises, service providers and other organizations.
Decision makers across many industries, in particular telecoms, utilities, retailers, governments, healthcare and manufacturing, are beginning to apply new big data strategies and technologies to derive immediate and detailed data-driven insights. Enhanced data visualization; more sophisticated sales, marketing and other CRM applications analytics; rapid detailed, interactive, multidimensional statistical analysis; and improved forecasting are starting to transform the business world.
Data Driven Business recently asked three big data experts with different perspectives to share their insights on the current big data technology landscape.
JAMES KOBELIUS, SENIOR ANALYST, FORRESTER RESEARCH: Adoption of big data technology has been staggeringly rapid. Batch loading is still typical, but less dominant. Big data technologies are enabling a shift away from batch-loading to data-streaming and real-time continuous analytics. Pilots, testing and development, and an expanding range of applications have recently progressed at a fever pace. Focus on big data technology has intensified over the past two years, with substantial interest from large enterprises and IT companies. In 2010, companies were asking about big data technology definitions. In 2011, they were asking about technology decisions. Over the past six to nine months they have a general idea of who or what, but want input on best practices, optimizing and insight on what’s not quite working with Hadoop, and how they can learn from mistakes that have already been made. Hadoop is evolving rapidly in new directions. It is being widely commercialized and adopted in enterprises and its open-source nature will be the heart of the new frontier in big data. What’s more, Hadoop and other approaches will merge and mash, which will be very exciting.
KRISH KRISHNAN, CEO, SIXTH SENSE ADVISORS: There are two things that will revolutionize big data technology in 2012. The first is in-memory processing for Hadoop, which I expect will happen sooner or later this year. The second is a strong emergence of powerful mobility based business intelligence (BI) solutions on mobile platforms. This is being driven by the arrival of LTE (4G) and will be a huge adoption driver for people to start utilising big data reports on mobile platforms. It will offer major benefits in terms of enabling easier big data visualization and personalizing big data. Tracking business performance from big data on social media and non-traditional solutions will radically change how people view and consume big data. An example of this is smart meters: energy and cost conscious end-users will be able to control domestic energy use from an app on their smartphones when out and about, depending on their needs that day. In terms of the technology solutions marketplace, this is an exciting phase. The key Hadoop start-up technology distributors (Cloudera, Hortonworks, Hadapt, Skytree etc.) are marketing themselves as a partnership point for the established enterprise data players. Lots of tie-ups have already happened in the first quarter of 2012 and there will be more over the rest of the year. There may well be significant mergers and acquisitions activity, especially towards the last quarter of 2012 with data leaders such as IBM, Oracle, SAP, Microsoft and Teradata looking to purchase the new start-up leaders. Or we could see solutions based consolidation with significant acquisitions of front-end enablers, leaving Hadoop alone for a while. The third thing to watch for is the emergence of cloud based development and deployment, especially for big data solutions. Now, with examples like Amazon Dynamo DP, you have a much more scalable database from the back-end that supports everything that you can throw at it.
KEITH MANTHEY, VP INTEGRATED DATA SOLUTIONS, EQUIFAX: The past year has been characterised by the maturing of the reporting and BI tools that work on top of the existing big data tool sets. Now in 2012, tools such as Data Meer and JasperSoft provide a more robust reporting and BI capability for the big data ecosystem. The availability of these tools allows for reporting direct from the big data environment instead of having to ‘spin a data mart’ for every report. While this may seem minor, a group that generates 200 different types of reports may in fact find themselves spinning upwards of 50 or more marts to feed those reporting requirements. The development of new tools and the big data ecosystem build-out are providing a compelling reason for even companies like Equifax, who have traditionally ‘rolled their own’ big data platforms, to look towards the vendor offerings that are available on the market. The ecosystems surrounding the big data tools will continue to evolve over 2012/13. My expectation is that the level of investment by vendors and consolidation of software providers will help grow the total ecosystem supporting tools like Hadoop. As the big data ecosystem grows, enhancements around data security, reporting, scalability, and auditing will help even heavily regulated industries to get a better comfort level with Hadoop and other big data tools. I see less of a focus on ground breaking technologies in the next year and a stronger focus on hardening the tools that already exist. Continued investment in tools like Cloudera and Mapr that are working on hardening the entire big data ecosystem will continue to help adoption at companies wanting that level of security and hardening instead of them having to put investment into building those capabilities themselves.
If you'd like to hear what these experts have to say about enterprise big data strategy, adoption, application and outlook, you can download our full Big Data for Enterprise report here.
Krish and Keith will be also be speaking at Data Driven Business’s Big Data for Enterprise conference, taking place on June 27-28 in San Francisco. Big Data for Enterprise USA 2012 brings together innovative companies which are now recognizing the power of big data analytics to drive business growth and deliver sustainable advantage.