Big data: Are our CIOs there, yet?

BANGALORE, INDIA: Large amount of data is not new in an enterprise scenario, however, the 'velocity' and 'volume' at which it is being produced by a 'variety' of sources and the bid to make sense out of it is definitely a new paradigm that is setting in gradually. It is this paradigm shift that is paving way for big data and its analysis.

Big data, as the term denotes, is a large set of data, both structured and unstructured; but mostly unstructured. Structured data is the usual business related data that gets accumulated in database, whereas, unstructured data comes in the form of social network, such as Twitter, Facebook, updates, e-mails, call centre conversations, RFID tag, CCTV footage, video, digital photo, sensor, etc.

Data is termed structured and unstructured based on the manner in which it is produced. Structured data is a database where data is stored based on a methodology of columns and rows, a language that computers recogonise. On the contrary, unstructured data does not fall in either of the categories and so is incomprehensible for computers.

Enterprise data has always been structured, or so as to say, at least the comprehensible ones were always those that were in the form of columns and rows because traditional data analytical tools were not meant for unstructured set of data. So, although a large chunk of data was being produced at the backend, most of it was being left behind due to lack of desired set of tools.

"Large enterprises, having widespread operations since years, have retained information and data sets so large and complex that it is beyond the ability of commonly used software tools to capture, manage and process the data within defined timeline. Data (both structured and unstructured) gets accrued over months and years, in different data sets, which remains unutilized because of absence of data retention policies and also proper analytical tools to derive value from them. We use only 10 per cent of the data which is structured whereas the other 90 per cent of the unstructured data remains unutilized. Now, the challenge is to derive meaning from 100 per cent of this data," notes Prasenjit Mukherjee, GM-Information Technology, BSES.

N Raveendran, group CIO/sr. GM, Enterprise wide Solutions, Sakthi Finance Ltd./ABT Industries, says: "Tons of transactions are created by ERP and other applications in most of the organizations. They are generally in high volume and cannot be handled effectively by routine queries or reports. ‘Big data' is an approach or a strategy to interpret the same using proper methods and tools, mainly for the decision makers of the organization. The top two layers of the general Information Pyramid - top and middle management teams, need supportive IT environment towards getting the desired information with minimal key strokes and the manner which are comfortable with - in the form of dash boards, comparative charts, trend/what-if analysis etc."

Pertisth Mankotia, head, IT, Sheela Foam, also agrees that the volume of both unstructured and structured data is so large that it is difficult to process them using traditional database. However, at the same time, he adds that traditional tools support big data to some extent.

"We are already using certain applications for business decision purpose for structured data and that is very helpful. The only place where we are stuck is with unstructured data , which we can analyse, but need to spend too much time on it," Pertisth adds.

However, others like to differ when it comes to their preferred choice of tools to analyse big data.

"Traditional tools are good, however, they may not be aligned in the ERP environment as handling the same has to be in a structured manner. I strongly advocate tools which are associated with the concerned ERP application, under its family, for seamless and effective data warehousing and mining" opines Raveendran.

Speaking on similar lines, Prasenjit adds: "I would definitely consider the newer big data analytical tools as they come with high power computing and analysis. For example, traditional tools can work on predefined datasets, data analysis on predominantly historic data and data velocity restricted to batch processing. However, the new tools work on all data sets encompassing and iterative, data velocity is proactive and dynamic (real time) and further can do predictive, forecasting and optimized data analysis."

Big data has been one of the most discussed topics among technology trends of 2012. While many were, and are still, trying to understand what it means, some others adopted the usual wait and watch strategy so as to see how it would develop into once the disillusionment subsides. Currently, it is emerging as one of the prominent business strategies for 2013, where some have already implemented it, while others are planning to do the same soon.

"Definitely there is a need for enterprises to plan and decide towards this because of many reasons. As per the latest IT Act data retention in an enterprise is very critical. Hence, it is highly pertinent for enterprises to plan and decide towards garnering value from this existing Big Data. We started planning on this two years ago and have already implemented technology and statistical models to derive value from these historical data for our business benefits," adds Prasenjit.

Raveendran agrees that big data has been prominently placed in his company's IT agenda for 2013. He also adds that big data approach is going to be the core expectation in the IT environment, hereafter.

"Just implementing ‘Data producing' applications will not suffice in fulfilling the expectations of management. CIOs have to work on strategizing and executing this requirement with priority. Though, we are migrating to the new application-software landscape, in order to manage the data generated by the existing and new setups, ‘Big Data' solution is warranted to be in place. Due to the non-availability of this solution, many a time, some reports to the management are being generated in Excel worksheets with meticulous efforts after extracting the data from the legacy data sources," Raveendran adds.

Pertisth says that they have already implemented such analytics for structured data, whereas, for unstructured, they are still evaluating tools.

"I firmly believe that big data can improve upon all the existing process and more. It can dramatically improve decision-making process across the business, right from product design to supply chain management. To remain competitive, one needs to analyse both internal and external data as quickly and cost effectively as possible. The world is becomes more instrumented, with RFID tags, sensors and other sources, as more and more data is created. When paired with external data, generated by social media sites, there is incredible opportunities that are largely untapped and unanalysed. The only challenge is datasets that grow so large that it is difficult to capture, store, manage, share, analyze and visualize," Pertisth adds.

However, this shift is not easy as it sounds. It has its own set of challenges ranging from which tool to select, how and where to begin, what could be the return of such investment and most of all, convincing management that it is a relevant step.

While, Pertisth believes that big data requires a transformation both in business and IT, Prasenjit feels that the real challenge lies in deciding the right tool and ROI on financials.

"Vendors definitely communicate with us for such boxes and tools, however we would definitely look for solutions which have already been implemented for our business vertical or would go forward for a POC. The POC itself would bring out the power of the tools or their expertise. We have followed the same process when we had implemented our pilot project," Prasenjit add.

On the other hand, Raveendran points out that though there are several tools avaialble in the market a lot of thought process has to go into before giving a purchasing order or implementing it.

"Big Data approach has to be provided by the vendors after understanding the environment of the customers like current data nature and volume, the software-application landscape, expectations etc. Without freezing this approach, applying certain tools will not yield the desired results. Finding the expectations of the big data approach from the management is the primary and critical challenge being faced by all the CIOs," Raveendran adds.