Transformative Power of Data Streaming: Rubal Sahni, Area VP and Country Manager, India, Confluent

Confluent has released its 2024 Data Streaming Report, revealing that 95% of Indian businesses are achieving substantial returns on their data streaming investments. This groundbreaking survey of over 500 IT leaders from mid to large-sized companies highlights the transformative impact of real-time data on innovation, AI adoption, and market agility within India's digital economy. As businesses increasingly prioritize data streaming platforms for enhancing operational efficiency and driving customer-centric strategies, Confluent underscores the pivotal role of continuous data flow in empowering organizations to seize new opportunities and accelerate growth.

In an exclusive interview with Rubal Sahni, Area VP and Country Manager for India at Confluent, he discussed the role of data streaming platforms in driving innovation in product and service offerings, especially within AI and ML development in Indian businesses. He also highlighted the significant contributions of data streaming to enhancing cybersecurity measures across various industries in India, shedding light on its multifaceted impact.

Can you elaborate on how data streaming platforms are facilitating product and service innovation, particularly in AI and ML development, among Indian businesses?

Data streaming platforms are revolutionizing product and service innovation among Indian businesses, particularly in AI and ML development. As GenAI takes center stage, these platforms address the critical need for trustworthy, properly formatted, and real-time data, which is essential for generating accurate and relevant results. The emergence of Retrieval Augmented Generation (RAG) as a common pattern for GenAI-enabled applications has further heightened the importance of data streaming. By tapping into continuous streams of real-time data from core business systems, data streaming enriches RAG-enabled workloads with trustworthy and contextual information.

This approach allows businesses to connect general-purpose LLM models to domain-specific data while maintaining granular access controls and minimizing hallucinations – a common challenge in AI. Moreover, data streaming platforms efficiently transform data into formats suitable for vector databases, streamlining the data preparation process for AI applications.

This enables Indian businesses to focus more on model development and innovation, accelerating their AI and ML initiatives. As a result, companies can now develop AI-powered applications that respond in real-time to changing business conditions, offer personalized experiences, and make data-driven decisions with greater accuracy and speed.

A prime example of this innovation is Observe.ai, which leverages real-time data streaming to provide live conversation intelligence for contact centers. Their platform utilizes LLMs to automatically search knowledge bases and provide answers to end-user questions, freeing up agents to focus on more complex customer interactions. The system also generates real-time transcripts of customer calls, further enhancing agent efficiency. Importantly, Observe.ai's platform demonstrates the scalability of these AI-powered solutions, capable of processing thousands of calls simultaneously.

How has data streaming contributed to improvements in cybersecurity measures across industries in India?

Data streaming improves cybersecurity measures by allowing for real-time threat detection and response. Continuous monitoring of millions of data points per second enables enterprises to discover and eliminate threats almost instantly, thereby reducing the effect of possible breaches. This proactive strategy fills gaps left by traditional security measures, transitioning from a reactive to a proactive security culture and allowing organizations to predict and address threats before they do harm.

Furthermore, data streaming complements machine learning to improve threat detection. ML systems, trained on large datasets of historical security incidents, may detect subtle trends and abnormalities that indicate prospective dangers. This combination enables for more accurate and quick threat identification, which improves overall cybersecurity. The advantages extend beyond IT systems, with applications for monitoring physical security, detecting fraudulent activity in financial transactions, and spotting internal dangers. Data streaming is critical for preserving the trust of customers and stakeholders in different industries by offering strong security measures and addressing data privacy and compliance concerns.

How critical is real-time data in enhancing decision-making and operational efficiency in the Indian market?

The growing internet penetration throughout India and a stable government are creating numerous opportunities for businesses, and also contributing to the growth of the startup ecosystem in India. This conducive business environment is resulting in increasing competition and a highly dynamic market. Consumer preferences, competitor strategies, and economic conditions can all fluctuate quickly. With real-time data, Indian businesses can stay on top of these sporadic changes and make informed decisions that adapt to the competitive market landscape where businesses need to be quick to adapt and capitalize on new trends.

Real-time data also allows for continuous monitoring of operational processes. Indian businesses can identify bottlenecks, optimize resource allocation, and address inefficiencies as they arise. This leads to smoother workflows, reduced costs, and improved overall operational efficiency. For example, a retail chain can use real-time sales data to identify popular products and adjust inventory levels accordingly, preventing stockouts and maximizing profits.

What are the main challenges Indian businesses face with data accessibility, and how do data streaming platforms help overcome these?

Indian businesses face significant challenges with data accessibility and management, primarily centered around data quality and siloed data issues. Our survey reveals that 82% of India’s IT leaders with data streaming experience report encountering five or more data-related challenges impacting their organizations. More specifically, the top data-related challenges reported are:

●    Inconsistency of data sources (73%); 

●    Outdated data (71%); 

●    Data silos (69%) within the organization. 

●    Uncertain data lineage (67%); and

●    Uncertain timeliness or quality of data (67%).

Data streaming platforms offer powerful solutions to these challenges. An overwhelming 92% of businesses report that data streaming helps break down data silos in most or many situations. Furthermore, 90% note improved access to existing data and 84% cite enhanced data discovery capabilities.

Data streaming platforms like Confluent act as a central nervous system for businesses, connecting applications and systems to ensure data is available in real-time, everywhere it’s needed. This breaks down departmental silos and makes information readily accessible across the organization. It also allows companies to focus on accelerating innovation instead of navigating the maze of point-to-point connections.

How is Confluent contributing to the growth of the startup ecosystem in India, particularly through support in real-time data solutions?

Making the right tradeoffs is essential for startups. At such an early stage, how a young company allocates its limited resources – be it capital, talent, or time – can determine its fate in the competitive business landscape.

Through our Confluent for Startups Program, we work to help startups make informed tradeoffs that set them up for success. This program goes beyond simply providing access to Confluent's platform. It offers startups valuable insights from Confluent's deep pool of expertise in Kafka and data streaming. This support is instrumental in helping startups navigate challenges and make sound architectural decisions, significantly reducing the time it takes to bring products to market. With Confluent Cloud, startups can rapidly scale their Kafka implementations to keep pace with their growth and adapt to fluctuating demand.

One of the key benefits for startups is the ability to focus their engineering resources on delivering product value and driving innovation. By offloading the complexities of managing data infrastructure to Confluent, startups can dedicate more time and energy to experimenting with new ideas and refining their core offerings. This approach not only accelerates development but also enhances the overall quality and competitiveness of their products in the fast-paced Indian market.

Above this, our consumption model allows customers to pay according to how much they use. It provides the scalability and flexibility startups require, enabling efficient real-time data processing as the company grows. 

What are the key trends in data streaming that you believe will shape the future of business agility and innovation in India?

Based on our survey findings, emerging Generative AI technologies are at the forefront, with 85% of IT leaders expecting the technology to trend up over the next two years. Data streaming and continuous flowing of data is close behind, with 80% of businesses acknowledging its importance. This indicates a strong trend towards integrating AI-driven solutions for enhanced decision-making and innovative product development.

We are especially excited with our introduction of Tableflow, which will better unify the operational and analytical estates. This is because while the same data is required in both the operational and analytical estates, they are often accessed in different ways. Tableflow makes it easier to convert streaming data to Apache Iceberg tables to feed data warehouses, data lakes, and analytics engines. 

While data warehouses and data lakes are still vitally important in the data architecture, it comes with data management challenges. We see a paradigm shift where data processing and governance can be done closer at the source, allowing clean, reliable and trustworthy real-time data to be delivered to the data warehouses and data lakes. For businesses, this means reduced cost, increased engineering agility and a future-proofed data architecture.

As organizations look to broaden and simplify data access and reusability, data streaming can also simplify the management of data into data products. Data products are live, refined, fully governed, and ready-to-use data assets that are instantly discoverable, contextualized, trustworthy, and reusable for many use cases. Data products allow organizations to reuse data across a variety of use cases to save costs and time. In India, 80% cite significant benefits from embracing a data product approach. 

Can you provide specific scenarios where data streaming directly improved customer satisfaction or engagement?

In the gaming industry, Mobile Premier League (MPL) faced challenges with their batch processing systems, which hindered real-time decision-making and user engagement. By collaborating with Confluent, MPL gained access to real-time data streams, enabling them to generate tailored offers for their gamers and also identify fraudulent users instantly. This resulted in improved user retention and engagement, while also achieving significant cost savings. The real-time capabilities not only enhanced security but also allowed MPL to offer a level of personalization that sets them apart in the competitive eSports market.

In e-commerce, Meesho needed a scalable platform to handle the immense growth in site traffic and demand. Adopting Confluent allowed them to manage throughput bursts efficiently and scale their operations without increasing the burden on their engineering teams. This solution enabled Meesho to handle triple the load during peak sale periods, using elastically scalable Confluent Cloud clusters. The improved scalability and performance directly translated to a more responsive and personalized online buying experience for customers, enhancing their satisfaction and engagement with the platform.