Optus streams 400 Oracle tables into Databricks to speed regulatory reporting

Optus has sped up the extraction of information from legacy customer-facing applications that sit on an Oracle database, as it works to satisfy regulatory and compliance obligations. Its pipeline combines Confluent's Oracle CDC and LogMiner components with the Kafka streaming platform, moving 400 tables into the Databricks analytics platform.

Anupam Das, Optus's associate director of design and delivery, described the setup at a Databricks customer event. He said the data feeds analytical and reporting uses and, most importantly right now, regulatory ones. Das said the data is "very crucial in terms of accuracy".

The method has cut the time taken to prepare data for Optus leadership and other parts of the business from days to hours, according to Das. The aim is to give leadership an accurate view of how the business performed over the previous day. Das said that "This was not possible two years back" and that Optus previously waited two or three days.

The same technology lets Optus track customer behaviour in its stores in near real-time, which Das said opens marketing opportunities such as product offers based on purchase behaviour. He said it would also allow more machine learning and AI to be applied to the data, rather than leaving it locked in legacy silos.

The work sits alongside a unified data platform that Optus has recently completed with a consortium of Microsoft, Databricks and Tech Mahindra. The platform brings data from across the business into a data lakehouse on Azure, with Databricks providing analytics and governance. Optus chief information officer Mark Potter earlier said the transformation would run in two phases: a lift-and-shift from a data warehouse nearing end-of-life, then the consolidation of two other substantial data marts. Potter said some data would stay outside the platform and be added as needed, and that Optus was not pursuing "one data mart for the entire organisation".

Das said Confluent's streaming technology is already playing a "key role" in the transformation, which will continue for now. He said the tech stack may change, but Confluent is expected to play a significant role in the layer that moves data into the platform.