Foxtel has set up a data platform primarily based on Google cloud platform services as it looks to better understand its audience and how to serve them content and advertising.
Head of data engineering Cameron Joannidis told Google’s Cloud OnAir summit that Foxtel needed a more consistent view of data, as well as a “clearer picture” of how its products are consumed.
“Like many enterprises, Foxtel faced a series of data-related challenges in its early days,” Joannidis said.
“Since it's been around for over 20 years, many data silos have been built out of necessity, and over time this has made it harder and harder to produce a consistent view of the business.
“Beyond this basic need for data reconciliation and consistency, it was identified at Foxtel that in order to remain competitive and enhance and grow the product offerings, we needed to have a clearer picture of our customers, their expectation and their usage of our products.”
Joannidis said the company saw a need to build advanced analytics capabilities and a strong data science presence internally - backed by a new data platform.
“Foxtel needed to find a scalable way to store, transform and leverage this data as the existing landscape was not fit to handle the data volumes associated with our key data assets, such as our viewing data event streams,” Joannidis said.
“We had an initial scope of work that included ingesting the various data sources across the organisation; processing, enriching and modelling into an enterprise data warehouse; data science, engineering, BI and analytics use cases that underpin each of these initiatives; self-serve analytics for business users; and data products for internal and external consumption.”
The company identified five initial use cases for the data platform, which it believed would deliver the “highest value”.
One of the use cases is to “optimise spend on content, since Foxtel spends so much money on premium content,” Joannidis said.
“It's important to ensure that this content is both relevant and interesting to our customers and adds to the breadth and depth of our overall offering.”
A second use case is around “data-driven promotion targeting, which involves leveraging and automating a more fine-grained understanding of our customers and their usage and preferences to ensure that we can direct the right promotion to the right person at the right time more effectively.”
A third use case is to improve communications with customers and better understand how they interact with Foxtel’s products and brand.
“We [also] need smarter advertising to ensure that we leverage our knowledge of our customer base and their preferences to serve them the most relevant ads, and also to measure the effectiveness of this for our advertising partners,” Joannidis said.
“And finally, we need to be able to get timely and consistent answers to business questions to ensure Foxtel can understand its key metrics.”
To accomplish all of this, Foxtel has stood up an “initial data platform” of primarily GCP cloud-based services, with BigQuery acting as both a data lake and data warehouse.
An architecture diagram does show other components including Alteryx, Tableau, DBT and Kubeflow for various functions of the data platform.
“GCP was selected for a long list of reasons,” Joannidis said.
“However, some of the key ones included the fact that it was a PaaS, which enables us to focus on business value and creating and managing our data assets rather than worrying about the underlying infrastructure.
“It has some of the best in class AI capabilities on the market at the moment, including AI Platform which gives us notebooks, training and scoring as completely managed services.
“It [also] has a particularly strong data processing suite in BigQuery, Dataflow and various other data transformation tools, and the roadmap of GCP is very much aligned to Foxtel's strategic direction.”
Joannidis said Foxtel is taking an ELT - extract, load, transform - rather than ETL approach to data ingestion.
“As opposed to ETL - extract transform load - which focuses on transforming data as it's being brought into the platform, we focus on ensuring our data is kept as raw as possible by replicating it from source, and then transforming it once it's inside our data platform,” he said.
“This has many benefits, the most obvious being that we have the full raw historical data to leverage and reinterpret as necessary without the need for costly ETL projects for every new requirement. It also allows us to leverage Google's wide array of data transformation options.”
Foxtel is also baking governance and security into the platform by design.
“Governance is one of those things that is really hard to retrofit once you have a data platform in place already, and as such, we've put a lot of effort in to ensure that our GCP platform is well governed and automated from day one,” Joannidis said.
“We also highly value our customers’ privacy and security, and as such we're focused on ensuring a lot of effort has gone into securing and automating our platform at the various layers.”
The company is also tapping into managed services, where possible, to meet its various use cases for data.
“Since there are so many managed services for data transformation, quality, machine learning, and visualisation that natively connect into BigQuery, we lean towards these services where appropriate for our use cases,” Joannidis said.
Joannidis said Foxtel had mapped out “a lot of exciting use cases” to pursue over the next year that would help the company build upon the work it has already done using the data platform.
“Some of the exciting stuff is to have self-serve analytics in the hands of business users, and to extend our data commercialisation opportunities, in particular with our viewing data,” he said.
“We also are looking at democratising ingestion and scaling our SRE [site reliability engineering] capabilities to ensure that it's as much of a self-serve platform as possible, to reduce the friction from our business users from leveraging the platform.”
The data platform appears to solve a long-running challenge at Foxtel; that its viewing data was simply too voluminous to be handled by regular data warehouse infrastructure.
It has previously used audience data to improve customer retention rates.
