WooliesX has revealed for the first time the real impact that scaling up Woolworths’ digital operations over the past two years had on systems, engineering practices, and the people that did the work.
Head of technology Souvik Mukherjee told the recent Apidays Australia conference that the rapid expansion of digital to meet pandemic requirements and changing customer behaviour led to “internal engineering inefficiencies” and impacted “team health”.
“Growth comes at a cost,” Mukherjee said.
“People who have worked in a startup would tell you this - that when you transition from a startup to a scaleup, it is a very, very painful process.
“When you think about WooliesX, which is a massive business when it comes to servicing customers [of Woolworths], scaling at that rate was even more painful.
“Our growth trajectory was very, very steep, and our maturity level to manage such growth was not coping.”
The behind-the-scenes story of what Woolworths - and indeed most large customer-facing businesses - went through in the past couple of years to make digital run at scale is largely an untold one.
Financial results presentations consistently highlight the benefits of the scale-up effort: growing digital sales, both as a share of total sales and in their own right, without mention of the mechanics of the efforts.
Mukherjee is candid about the amount of effort and resourcing that sits behind these results.
WooliesX felt the pressure almost immediately as customers compared the digital experience of Woolworths to what was on offer elsewhere, particularly from digital-only players like Amazon.
“We were not ready for Amazon Prime style delivery at that point when the pandemic hit us,” Mukherjee said.
“We were not ready for those kinds of features.
“We were not ready to even service our products well on our websites, to be honest with you.
“When you have those hyper recommendations that Amazon does, we were nowhere close to those. So our customer needs were skyrocketing very, very quickly.”
To bring new features to market faster, WooliesX onboarded a sizable - though unspecified - number of new staff.
“We increased the number of people we had in the organisation and we increased the number of features we delivered, just to cope with all the needs that the customer put on us,” Mukherjee said.
Squad sizes doubled, but “velocity started falling because the squad became inefficient.”
“To manage that, we split squads into two just to make sure that we continue that velocity going forward, but then what it also means is a lot of IP is now split across two different teams,” Mukherjee said.
“There are interdependencies between teams because they were working off a single repository which was not ready to be serviced in tandem across multiple squads. So it created internal engineering inefficiencies.”
Still, a lot of new code was created: in the space of one-to-two years, the number of microservices under WooliesX’s gaze quadrupled.
Mukherjee wouldn’t put an exact number on the current microservices footprint. “I can't give you the exact number, but [it was] - let's say - in upper three-digit figures when it came to the number of microservices we had, and we four-folded that in a year.”
The end result wasn’t just an impact on engineering processes and practices, but on the people doing the work.
“We were trying to deliver a lot of stuff in a very short period of time and one after the other in tandem, and that started having an impact on the team’s health,” Mukherjee said.
“People started talking about feeling burnt out. They were dedicated so they kept on doing what we needed, but that was not a sustainable model from a long-term perspective.”
The solution to these challenges is a four-pillared scheme that WooliesX has termed the ‘guardianship model’.
Its four pillars are customer-centricity, a service maturity model, ‘innersourcing’, and setting “healthy foundations” for individuals and teams employed by WooliesX.
Re-engineering customer problems
Engineering teams now review customer complaints on a weekly basis and align their work to what customers want.
“There's a quick sense of purpose from that perspective to say, ‘If I'm working on this feature, I'm directly impacting that problem statement that our customers are complaining [about],” Mukherjee said.
“That resonated really well. All our engineers got very very motivated to say, ‘These are the 2000 customers who complained about this and today I'm working on this feature to fix that’.
“That correlation - by bringing those customers and events directly to the development community, I saw the motivation of the engineers accelerate.”
This pillar of the guardianship model appears at least partially related to stemming the microservices sprawl of the past couple of years.
The retailer’s engineers group customer responses by topic - for example, product availability or checkout.
“We [then] applied domain-driven design on our microservices design principles and mapped our domains back to the components of voice-of-customer, so that there was a direct correlation between what our customer had to say linked back to what our service domains were,” Mukherjee said.
WooliesX also started using Backstage.io to catalog its microservices.
“When we started scaling from 10 to 20 to hundreds of microservices, we lost track of how many microservices there were, what they did and how they worked,” he said.
“Backstage.io … is a very simple tool to manage our application catalogs, linking back to our repositories.”
In addition to voice-of-customer, WooliesX is also comparing itself to industry benchmarks - such as the performance of other retailers’ homepages - on a weekly cadence as well.
“This is basically putting our ‘finger on the pulse’ on steroids,” Mukherjee said, adding that WooliesX had “set an ambitious target to be rank one in that space” but is “not there yet”.
Service maturity model
Mukherjee said the second part of the guardianship model is the establishment of a “service maturity model” that ranks the “criticality” of services, sets performance benchmarks for each, and tracks the “health of that service against that benchmark”.
He did not spend much time on this aspect of the guardianship model, though he said the benchmarking was “well accepted” internally.
Work is continuing in this area.
“It works really well with services that have recently [been] developed, but if it was a legacy piece of software or software that was built way before anyone's time, I'm seeing service maturity models [are] not very effective for us right now,” he said.
“That's something which we are working on. How do we mature our service maturity model to tackle legacy as well?”
Innersource adoption
The third part of guardianship is an embrace of innersource, a set of software engineering practices that are used to create an open source-like culture inside of an organisation.
Innersource has found favour in at least three of Australia’s big four banks. WooliesX’s use of innersource wasn’t previously known.
“When we split the squads, when we split the teams, what happened was the IP got distributed, so people had dependencies between teams and that increased the complexity,” Mukherjee said.
“We moved to an innersourcing model where we opened up repositories to our internal developers so anyone can actually contribute to any repository if they need a change, but it comes with some governance and guardrails.”
Those guardrails define the type of access to the code that is permissible - whether that covers read or write access, or both - and who has ultimate sign-off on suggested changes.
Mukherjee said some of the rules around innersource are still being defined, particularly relating to incident response.
“The toughest one - and we haven't cracked it yet - is inter-squad warranty support,” he said.
“It's great that someone can come to my repository, make changes and release it to production, but what if something goes wrong? Who is accountable for it immediately after it has gone wrong?”
‘Healthy foundations’
Mukherjee said he is “very close” to the fourth pillar of the guardianship model, “working on healthy foundations”.
This aspect impacts both people working on digital programs, and the ‘health’ of WooliesX’s code repositories.
On the people side, he said the focus is on wellbeing - “how do we ensure that we are creating good career paths for our people, that they are growing within the organisation, [and] that they have a good state of good mental health and team bonding going on in the organisation?”
He said it is now clearer to staff what kind of capability and accountability is required to move to different roles in the organisation, and that had helped staff decide how to hone their skills.
Mukherjee cited cultural initiatives as supporting this, which include “chapter time”, where - in a nod to Google’s 20 percent rule - employees are able to spend 20 percent of their time every week “to focus on themselves”.
They can “spend that on learning, training, developing standards, developing guidelines to enable the business”, but it’s left to them to decide, he said.
On the coding side, the ‘healthy’ focus is applied to tech debt reduction and to not prioritising delivery speed over everything else.
Mukherjee said there is top-down support at Woolworths to “ensure that the level of tech debt within the organisation that impacts our operational stability is kept under control.”
“Just like a bank, you don't want your loans, your debts to go out of your control, so we manage that very honestly,” he said.
“We build a lot of new features, we build new capabilities, but we move on to the next one very very quickly - we don't really sync it and we don't really grow our IP, we don't really embed those changes.
“That's something we are focusing on right now.
“If we have built a new feature, let it settle down. Let's learn about it, let's think through it a bit more before we move on to the next one, so that the IP sinks in.”
Overall, Mukherjee indicated that WooliesX would “double down” on implementing the guardianship model.
“There is commitment right from the top to invest in this guardianship model that we built as an engineering community,” he said.
