The rise of the data engineer brings AI to the forefront within the enterprise

The hype around AI is justified by its transformative potential for organizations of all sizes right from SMEs to large enterprises to governments. While its ability to automate and hence reduce costs is well understood, AI’s real potential comes from being able to increase the topline through enabling innovation and improving employee efficiency. According to Gartner, 70 per cent of organizations will integrate AI to assist employees’ productivity by 2021. And more and more companies are using AI to shorten the innovation cycle, for instance in drug discovery.

And yet, according to Gartner, 85 per cent of AI projects won’t deliver for CIOs. That means, out of 20 artificial intelligence projects, only three will succeed, and 17 of them will fall short. In order to understand what it takes to be among the three that succeed, let’s understand why AI projects often fail and also understand what unifies the AI projects that succeed.

The most common reason for the failure of that AI projects are often poorly conceived. Their adoption may have driven by FOMO or the Fear Of Missing Out and lead to companies simply following the herd. In such cases, the necessary investment of intellectual bandwidth to clearly defining the objectives and desired outcomes of the AI project is not made. What then happens is that there is misalignment between what is being asked of the AI and the actual business problem.

The other common reasons for failure are not having the right data to train the AI system, poor communication between business teams and the technical teams, setting incorrect expectations, building the wrong training model and using the wrong algorithms.

All of these reasons can be attributed to not having a good data engineering team in place. Enterprises need to heavily focus on hiring and empowering data scientists to create advanced analytics to ensure that they are one of the three that succeed.

A good data engineering team is one that can bridge the gap between business and data. A common refrain from companies whose AI projects have failed is that the data engineers speak geek and not the business. They talk using jargon and terminology that is intimidating to the business folks leading to communication gaps and misaligned expectations.

Goals need to be specified in terms of deliverables that will be done in a phase-wise manner so that course corrections that will be inevitably needed can be done early.

Data collection is often a huge pitfall. The required data for the AI project might not be digitized and even if available in a digital format might require a lot of pre-processing before it is fit to be used for training by an AI system. A good data engineer will be able to identify what data is required, where exists, how to collect it and how to prepare it for training.

The choice of AI training models and algorithms can also have a huge impact on the success of AI projects. This is where an experienced data engineer who has seen several projects before comes in handy. It is much better to rely on someone who knows the common pitfalls and what is likely to work for a particular problem.

The huge potential of AI has meant that AI projects are on the radar of CIOs of organizations of all sizes. But for these AI projects to actually succeed it needs the right data engineering team to be in place to deliver on the promise of AI.

By Dr Ranjit Nair (PhD AI), CEO and Founder, Germin8