Managing Data Overload

There is excessive data all around us, boring down like giant tsunamis. Data overload has taken on epidemic proportions as an unintended consequence of advances in IT and connectivity. Business intelligence dashboards are churning out live data by the minute, Web analytic tools are generating endless data, and several other tools are producing data that needs to be processed into usable information. Newer technologies such as RFID and RSS feeds are adding to the overwhelming onslaught of information. Many companies have untapped treasure troves of information that can prove useful in business decisions. But the overload of data makes data mining a daunting task for any organization of a considerable size. According to industry experts, technologies that help reduce and manage data overload will be the demand of the future.

Researchers in the area of data analytics are building new tools to help make sense of the ever growing amount of data being created. The focus is on being able to unlock information that is buried in unstructured data. For instance, for any retail organization or a producer of consumer goods, customer data collected can be used to identify purchase trends and preferences. Retailers often collect unstructured text in emails, online chat sessions, social media and networking sites, and customer feedback comments. They can use data analytics tools to better respond to input that their customers are giving them. Google, for example, is avid user of data mining and text analytics. They use it to try to offer better and more competitive services to their advertisers. The more that Google can identify a web page and match it to an advertiser’s product, the more likely that a click-through to the advertiser’s web page will result.

How data is collected, analyzed and used is changing rapidly as real-time, sensor-based monitoring applications grow in popularity. Business organizations will need to adapt to these emerging technologies by unlearning what they have been taught about business intelligence. Data mining professionals will still need to design and implement data warehouses and integrate the client-side environment. But what won’t work are long-established methods, as the type of data and its vehicles have changed. For instance, streaming data is different from traditional business intelligence since it’s loading constantly and sources are widely distributed. Also streaming data is dynamic and traditional data analytics tools are not equipped to process such real time data.

There is a critical need for technology to support this data overload. This means better information visualization, a result-oriented presentation of relevant data, and better overall management of increasing volume of historical data, test findings, customer/partner/vendor interactions and so on. Businesses today need data management solutions that will research and visualize patterns and trends, and will analyze data to put all that information in context and translate into decisions. Moreover, regulations demand that the data collected cannot be discarded. And that’s why data management and that includes the storage technology behind it, and knowledge management is going to be key technology to remain operationally compliant and commercially competitive.

Technology will become a key player for successful enterprise feedback, media analysis and customer satisfaction initiatives, and long-standing data analytics users will find themselves relying on technology more than ever. The technology is being built into e-discovery and listening platforms, law-enforcement IT, BI suites, data-mining workbenches, search interfaces, and the Web. These tools include filtering mechanisms for groupware and E-mail, software agents that scour databases, and search engines that help tame the data overload beast.


Leave a Reply