Outage for Ubuntu users on Azure

A systemd security update has brought unexpected grief for Azure users running Ubuntu 18.04: it’s broken their DNS queries.

The ongoing outage began at around 6am UTC (4pm AEST) on August 30.

Microsoft Azure’s outage notice reveals a bonfire of dependencies: users of Bionic Beaver in Azure virtual machines, with unattended upgrades enabled, would have had been pushed systemd version 237-3ubuntu10.54.

“A bug in this version will lead to DNS resolution errors,” Microsoft explained.

Those errors will affect any application that needs to retrieve DNS information.

“This bug and a potential fix have been highlighted on the Canonical / Ubuntu website, which we encourage impacted customers to read.

“An additional potential workaround customers can consider is to reboot impacted VM instances so that they receive a fresh DHCP lease and new DNS resolver(s). 

“If you are running a VM with Ubuntu 18.04 image, and you are experiencing connectivity issues, you can evaluate the above mitigation options,” Microsoft said.

Other Azure services are caught up in the downstream effects: “A large portion of impact has been to Azure Kubernetes Service (AKS) in multiple regions, and other Azure services reliant on AKS.”

Microsoft said it is testing “automatic mitigation steps” to apply to AKS resources once validated, and added that the offending Ubuntu updates have been removed for the time being.

Ironically, the offending systemd package was a security patch to protect the daemon against CVE-2022-2526, a medium-severity DNS vulnerability.