Ticketek parent TEG has created an automated process for its business users to schedule and provision resources for high-demand ticketed events, partially to stop IT over-provisioning capacity for every ticket sale.
Head of enterprise architecture Tane Oakes told AWS’ re:Invent 2018 in Las Vegas that the company periodically sold “hot shows”, where demand for tickets would swamp its web properties.
The phenomenon of ticketing and web transaction sites buckling under excessive load is a well trodden path for online vendors and government in Australia with performance issues hitting everything from the Melbourne Cup to My Health Record.
The challenge is when people come, they come hard and fast.
“We go from a small amount of people on the site to 300,000 people within a second,” Oakes said.
Oakes likened the surge in demand to Black Friday sales in the United States - and, closer to home, to Aldi’s ‘special buys’ in Australia.
But the so-called FOMO (fear of missing out) of herd behaviour that can clog transaction pipes rather than just the mob itself.
“As soon as Aldi’s doors open everyone comes charging in, puts everything into their basket, sort of wanders around and then decides if they actually want it or not, but they need to get it in there to make a decision because if they’re standing around looking, someone else is going to grab it and they miss out,” Oakes explained.
“Hot shows are the exact same thing. You get a lot of cases where everyone’s coming in, getting tickets, putting them into their baskets, and then going to their friends, ‘I’ve got seats here, have you got seats there, should we go with yours?’ You have to be manage and control this flow.”
Scheduling these big ticketed events - and provisioning enough IT resources to meet the expected surge in demand - was previously a largely manual process inside of TEG.
“Previously, the whole way things would work is maybe a Friday the week before, we’d have a big meeting and the business would go through what they were looking at scheduling,” Oakes said.
“The IT team would then be told, ‘Can you spin this up, can you have it ready?’
“The IT team would agree, but the easiest thing for them was to just spin it up and have it sit there forever until it was shut down.”
This was easy, in part, because the business might then change the times of the sale or other plans after the meeting, and this was communicated “back and forth over emails and by passing around Excel spreadsheets”.
The IT shop was also in the practice of over-provisioning.
“[The hot show request] would go to the IT team and they would always put it as a full-sized hot show as if it was always expecting someone who would get 300,000 ticket requests over an hour,” Oakes said.
“That may not be the case all the time. They were overprovisioning.”
Because of this, Oakes said it took TEG “a number of days” to spin up each hot show.
Reforming 'hot show' scheduling
TEG has changed this by creating an interface through which the business can schedule hot shows, and by automating all the background steps to spin up and spin down the infrastructure needed to back these sales.
The new interface is powered by tools including AWS Service Catalog and Atlassian’s Jira.
“We needed to allow our business users to submit a request to schedule a hot show,” Oakes said.
“So [the first step is] they come and schedule it; they say, ‘I want to have a hot show spun up and available, provisioned for me, from this time to that time, and have it available’.”
Business users are prompted to provision for events “like a T-shirt size” - guessing whether the site load will be small, medium or large, though Oakes noted there is some flexibility in the system.
“That goes into Jira’s service desk and it starts to kick off a track of [the required] pieces of work.”
A “hot show approver” is required to sign off on the request - or to send it back if the details don’t look right.
If it gets waved through, “what happens then is it goes into Jira and it gets Jira-integrated into an [AWS] API Gateway stack to essentially schedule this hot show in DynamoDB.
“That scheduler allows you to parallelize hot shows so you can do a Phil Collins one at 9am Monday and that’s going to come in under a URL that represents Phil Collins and take the traffic that way, and we can do another one for the Red Hot Chilli Peppers at the same time to another URL, and it coordinates and directs it all.”
“So you’ve got this schedule - then we use [AWS] CloudWatch scheduled events to trigger a Lambda that’s going and checking whether anything needs to be provisioned now.
“So our site is running as normal, and then Lambda says ‘I need to go and schedule a hot show’, and it goes to Service Catalog to trigger this.
“Service Catalog have a set of profiles that represent the hot show with all of the products that together build up the hot show - websites, enterprise load balancers S3 buckets, streaming architecture and so on.
“So it goes and pulls that all together with Service Catalog and then using our CloudFormation templates, it puts the show out and makes it available, and then it also triggers itself to pull that back down again when [the sale event is] completed.”
Oakes noted the importance of allowing the business to have input and control over the process.
He flagged more opportunities to use a similar structure and workflow for more than just hot shows.
“Service Catalog has allowed us to productise some of our internal systems like what we’re doing now with hot shots to drive or empower the business,” Oakes said.
“Where we’re seeing things with Service Catalog is we’re going to start productising a lot more of our parts of our ticketing system in there and turn it more into a ticketing-as-a-service offering, giving a lot more to our clients for them to pick and choose what they want to do.”
That, he said, could allow promoters and venues to “get data through real-time streaming, pull product data onto their websites” or to customise the way customers bought tickets.
“We can look at using AWS Marketplace to put those products out there, and put some more commercial monetisation around it,” Oakes added.
The hot shot scheduling overhaul is part of a broader digital transformation at TEG that has involved both internal staff as well as Deloitte Australia.
.jpg&h=140&w=231&c=1&s=0)
