Big Data Emerges As A Challenge In A World Of Unstructured Data Proliferation
Data volumes continue to explode with a proliferation of devices, social media tools, video usage, and emerging forms of both structured and unstructured data. The rate of data explosion may be occurring faster than Moore’s Law. Organizations now face a significant challenge in dealing with this data deluge. Across the 5 pillars of consumer tech effecting enterprise software, organizations must deal with:
- Managing unstructured user generated social interaction data. Massive smartphone adoption and social network usage will converge to create massive data volumes. Twitter now has 106 million users generating over 3 billion requests per day. Most analysts firms forecast at least 300 million smart phones in use among the 1.6 billion mobile devices sold in 2010. Sensing data, call detail records, location based information, digital media, and other sources will lead the individual data explosion.
- Coping with explosion in transactional data volumes. A collusion of compliance, regulatory, and digitalization leads to exponential increase in transactional data. Audit and compliance requirements lead to increase in security log files, network and system event logs, emails, and searchable messaging communciations. Add significant automation of business processes and Constellation Research estimates that annual growth in online transactional data and repositories will grow 66%. Most data centers now commit 25% of their infrastructure spend to support storage for data growth.
Informatica 9.1 Focuses On Big Data
Announced June 6th, 2011, Informatica 9.1 is generally available. The new release focuses on four key themes that address the Big Data issue:
- Delivering a near open data integration platform. The new release supports Hadoop, big transactional data, and big interaction data. Hadoop support includes connectivity to the file system, HDFS and MapReduce for big data processing. Big transactional data features support EMC Greenplum and other DW appliance vendors soon, in addition to existing Oracle, IBM DB2, IBM Netezza and Teradata connectivity. Big interaction data connectivity support for the Big 3: Facebook, Twitter and LinkedIn.
Point of View (POV): Hadoop provides low cost processing and storage platforms required to address the big data issue. While Informatica 9.1 is designed for a mind boggling petabyte connectivity to OLAP and OLTP data stores today, power users will push for exabyte scale in 12 to 18 months. The new release also delivers a complementary relational/data warehouse appliance package. For social data, organizations will improve their ability to correlate social media signals with transactional data to deliver new insights across the organization. Expect Hadoop and social media connectors to be delivered later in June 2011.
- Incorporating master data management technologies with Big Data. The new release incorporates key assets from the Siperian Master Data Management (MDM) acquisition. Users gain new multi-style and multi-domain MDM approaches. Data governance is addressed via resusable data quality policies while proactive data quality builds on Informatica’s complex event processing technology to identify and alert users on data quality exceptions.
POV: Informatica’s MDM offering remains among the top in shortlists at Constellation Research. The solution delivers true multi-style, multi-domain, multi-deployment, and multi-use capabilities on one technology platform. Users gain the ability to manage data quality rules in source applications that not only propagate downstream, but also take advantage of complex event processing (CEP) to provide proactive alerting (see Figure 1). Informatica’s Rule Point CEP engine also provides key geo-aware processing capabilities for advanced scenarios.
Figure 1. Informatica’s Self Service Proactive Monitoring
Will you make the move to address Big Data with Informatica 9.1? Will you consider other options? What will drive you to go with one platform? Add your comments to the blog or send us a comment at r (at) softwareinsider (dot) org or info (at) ConstellationRG (dot) com.
Although we work closely with many mega software vendors, we want you to trust us. For the full disclosure policy, stay tuned for the full client list on the Constellation Research website.
Copyright © 2011 R Wang and Insider Associates, LLC All rights reserved.
(Cross-posted @ A Software Insider's Point of View)