AGL transforms advanced analytics to run on central Azure platform

AGL has rebuilt its analytics capabilities to run off a central platform powered by Azure services instead of running projects from standalone virtual machines and laptops.

The new platform went live in mid-2020, according to a new case study, and forms the technological foundation for AGL’s data and analytics centre of excellence (CoE), which was created last year.

“We needed a platform that allowed us to manage models, code, and data as a coherent whole, rather than as individual bits and pieces,” senior manager of machine learning engineering Joey Chua said.

“That was the motivation for reaching out to Microsoft to see if they had a tool we could build on.”

AGL said in June this year that it would move “almost all computing” to the cloud by 2022, with most systems destined to run on Azure.

At the time, it said data and AI services were among the key ones it would consume, but the company declined to elaborate on its use cases at the time.

It’s now clear that a package of Azure services have been used to create a single platform that combines and centralises “analytics tools, data science resources, and machine learning” for AGL-wide use.

AGL said it piloted the architecture first before progressing to a production deployment of the platform in “mid-2020”.

“Azure Machine Learning sits at the center of the new solution,” the case study notes.

“It provides integration across a variety of Azure services, including Azure Kubernetes Service (AKS) and Azure Databricks, giving AGL a consistent, preconfigured environment with all artifacts code controlled, managed, and documented. 

“Azure Databricks provides big data analytics and powerful data engineering tools in a standardised workspace so that AGL’s machine learning teams can fully capitalize on the model building and training capabilities of Azure Machine Learning. 

“Production environments also take advantage of Azure Key Vault, the Application Insights and Azure Monitor Logs features in Azure Monitor, Azure Storage, and more. 

“Using these services together, AGL has a highly secure and efficient way to train, deploy, and manage thousands of models in parallel.”

AGL said an early production use case for the platform feeds “site-level energy forecasts” to customers.

From “a single, large forecasting model”, the utility is able to “create thousands of variations based on the parameters of each customer.”

“Customers can use these forecasts to understand their energy usage and change behaviours - with the models adjusting alongside them,” the case study says.

Chua said there are more data projects in the pipeline.

“The CoE already has initiatives planned across AGL’s lines of business,” Chua said.

“We have projects in the works from our wholesale market to renewable power generation. 

“This one platform will meet the needs of all those different applications.”