Cloud data warehousing has revolutionised our ability to handle and analyse big data more quickly and efficiently. Last year, Gartner predicted that by 2022, the global public cloud service market will grow from $182.4B in 2018 to $331.2B. Entire data strategies are built around it, but like with most tools, costs can quickly add up as usage patterns change. AutoML can save costs. Today, this is a clear priority for most businesses.
While reviewing our operational costs, we noticed our cloud data warehousing costs were significantly increasing month-on-month. It was becoming unsustainable for the business. Using our own solution we were quickly able to act.
We dropped our costs by 30%, overnight.
Our own AutoML solution allowed us to avoid impacting customers and still understand not only the computational load of queries on our cloud data warehouse, but also the cost of each query and where it came from. Not an easy task – we see millions of queries per day across many warehouses, clusters, databases and users. Not to mention all the different types: inserts, selects, updates, truncates – just to name a few.
Using a traditional approach would have taken us weeks, but we needed to act quickly. It’s widely accepted that AutoML can save costs and also time, and our AutoML solution allowed us to perform deep-dive analytics in minutes and to take useful actions within hours.
How we demonstrated the extent to which AutoML can save costs.
First off, the prep work.
We needed to assign costs to individual queries. Then we used Avora’s virtual data model to join the costs attributed to these queries with other useful attributes for each query. We looked at the results in our dashboards using the most suitable visualisation options to sense-check our work so far. Then we were ready to really get to grips with the data.
We jumped into the Metric Explorer where we ran Machine Learning – specifically the anomaly detection and forecasting algorithms – against our history of warehouse costs. Starting with a weekly view, the system automatically showed us some unusual behaviour – a big uptick in costs during the second half of March.
Digging deeper, we switched to a daily view (easy, because Avora builds queries on the fly) and found the few days where the increase began. For example, 18 March showed us costs at 60% above projections.
Naturally, the next question we asked was “why are costs well above projections?”.
If we’d taken a traditional approach to answer this question, pulling in large amounts of data, building a lot of charts in a dashboard to test theories, it would have taken a long time to answer.
But we know AutoML can save costs – so using our Root Cause Analysis, which is automatically triggered when clicking into an anomaly, we let the algorithms look for all the potential reasons.
Sure enough, within seconds the system showed us the most relevant reasons. Then we were able to go deeper using a business flow to help us understand what had happened in each case:
- Users from certain customers were associated with cost increases. Some had sent more queries through the Avora application. Our Customer Success team were able to help them ensure they were using Avora optimally. Others saw a spike in queries taking longer than average – our analysts were quickly able to make suggestions to improve their newly implemented data models which were causing this.
- A number of queries had failed caused by deployment of a new beta functionality. We came up with a fix right away and stopped it from recurring.
- There was new behaviour from an internal system user sending queries to a warehouse. Some new calls were being made against a high-powered warehouse for relatively simple queries. The computational equivalent of using a sledgehammer to crack a nut! We redirected these to the correctly sized assets.
It took us just 60 seconds to run the analytical process.
- 10 seconds to find out when the cost was unusually high using anomaly detection.
- 20 seconds for the automated Root Cause Analysis to tell us the most relevant reasons why it was high.
- 30 seconds to assess the reasons and go a level deeper with a business flow.
60 seconds, saving costs with AutoML to the tune of 30%, overnight. All we needed to do then was engage with our teams across the business to achieve these savings, which was easy because the evidence and likely benefit was so clear.
So what next?
At Avora, we don’t sit on our laurels – we live and breathe continuous improvement. Continuous monitoring is key – our Customer Success, Data Engineering and Operations teams are constantly using our AutoML solution to keep optimising our costs while better serving customers.
Read further research from Juniper about how AI analytics will boost operator’s revenue.