Why will 2021 bring augmented analytics to life for businesses?
Why will 2021 bring augmented analytics to life for businesses?
Augmented analytics will fundamentally change the way businesses interact with, and use, data. Yet despite all the benefits and promise, augmented analytics tools still haven’t been widely adopted, particularly by small and medium sized businesses.
Why is this? And how do we know 2021 will be different? In this article, I’ll review the main challenges to adopting augmented analytics and share the top five reasons why 2021 will unlock augmented analytics for many businesses.
Augmented Analytics Meaning
Augmented analytics (an umbrella term for technologies that automate the process of preparing and analysing data) offers businesses a fundamentally different approach to traditional BI by using AI and machine learning to simplify and enhance the process of preparing data, identifying patterns, flagging anomalies, finding the root causes and answering questions. It also empowers business users to be more self-sufficient, leaving analytical teams with less manual work and more time to spend on other important projects.
But the challenges for businesses to adopt augmented analytics tools have been significant. Let’s take a look at some of these.
They are more valuable if you have a lot of data. Augmented analytics tools can analyse vast amounts of data very quickly to look for anomalies and insights, as well as make connections between disparate data that might initially seem unrelated. If a company isn’t becoming overwhelmed by the amount of data it has, then the value-add might not be as obvious.
They require a lot of processing power. A lot of computer resource is needed to unleash the power to analyse millions of rows of data in a matter of seconds, particularly when doing this 24/7 so that issues are identified as soon as they happen. This can be expensive and challenging to manage, particularly with on-prem data warehouses.
They require data to be centralised. If data is kept in silos, then machine learning algorithms won’t be able to consider all the possibilities when running analyses.
They require data to be clean. If a company has ‘dirty’ data (incomplete, inconsistent, duplicate, sparse), then the machine learning algorithms won’t be able to do their job as effectively.
They require a change in culture. For the last two decades, dashboards and custom reports based on pre-defined KPIs have been the standard for tracking and communicating performance across the business. This approach is limited in scope and mostly reactive. Augmented analytics, on the other hand, is proactive, and makes it possible to consider every possibility, pointing to problems and opportunities businesses didn’t even know existed. However, old habits die hard and human factors such as resistance to change can be a barrier to adoption of augmented analytics, like with any new technology.
They require a change in process. Historically, there has been a clear division between the role of IT (those who generate insights for the business) and the role of business (those who request insights to make decisions). According to BI-Survey, one of the main barriers to success is that business users consider data to be the domain of IT and IT doesn’t want to give away control of data either. Successfully implementing an augmented analytics solution requires a change where IT supports business users in becoming more self-sufficient and data literate.
With all these challenges, it might seem bullish to suggest that businesses can be ready for augmented analytics now, in 2021. But surprising though it may seem, the conditions are ripe this year.
With the explosion of data and the increasing move towards SaaS, most companies are generating more data than ever before, and it would take an army of analysts to crunch, slice and dice all the data available. Without using machine learning to help comb through, there would certainly be critical insights left on the table. Furthermore, this rate of data collection is accelerating at a speed beyond what humans can handle, and this is likely going to compound in the years to come.
The previous challenges to implementing augmented analytics (namely barriers to preparing, storing and processing data) are disappearing thanks to powerful, scalable and cheap data modelling and cloud warehousing solutions, and Covid has brought about a growing need for businesses to be more stringent and cautious in how they invest budgets, requiring their users to be more self-sufficient. The solution can’t be to just keep hiring more data people. This, in turn, will shift the role of BI teams to focus on higher value problems including process improvements, creating new models, and upgrading systems and software.
Let’s dig into the details a bit more, looking at five significant reasons why businesses are ready for augmented analytics in 2021.
1. Traditional BI is struggling to keep up with the amount of data companies are generating
Over the last few years, the surge in the amount of data being produced has been mind boggling. A 2016 report from IBM stated that 90% of the world’s data had been created in the previous two years, and the amount of data in the world at that time already stood at 2.7 trillion gigabytes. Just three years later, in 2020, we reached approximately 40 trillion gigabytes and it’s predicted that this will increase to 175 trillion gigabytes by 2025.
As a result, more than half of all data captured by companies is never touched. This is the paradox – data should enable businesses to make better decisions, but instead they’re being overwhelmed with the amount of data accumulated.
This is the paradox – data should enable businesses to make better decisions, but instead they’re being overwhelmed with the amount of data accumulated.
Augmented analytics tools are unparalleled in their scope to analyse vast amounts of data very quickly. They can uncover anomalies, identify root causes and make connections between disparate data that might seem unrelated at the surface and could be missed by humans (or take humans a lot longer to find). In other words, they can enable companies to make full use of all the data available to them and make better decisions faster. In a time where we are generating more data than we can handle, augmented analytics is essential for helping make sense of it.
2. Companies are moving to cloud data warehouses.
Increasingly, companies are moving away from managing their own data storage needs towards using public cloud data warehouses such as Snowflake, Redshift and Big Query. These offer several benefits:
They don’t need upfront investment or maintenance costs.
They tend to be cheaper compared to on-premise solutions that require expensive hardware, upgrades, maintenance, and outage management. In a survey of 166 IT leaders, 61% cited cost as the main reason for switching to cloud data warehouses.
They are elastic, which means they can easily deal with increasing volumes of data coming in every year, and they offer flexibility with the changing requirements of the business so that businesses only pay for what they use.
Covid-19 has also spurred cloud data warehouse adoption. A survey of 187 respondents found that more than thought that cloud usage will be higher than initially planned, citing reasons such as difficulties in accessing data center facilities and delays in hardware supply chains as a result of Covid. The uncertainty caused by Covid has shown that the cloud is a more reliable option for business continuity.
This shift to more powerful, reliable, flexible and affordable cloud-based data warehouses opens the door to augmented analytics that tools have high processing demands, something that would have been more costly and prohibitive in the past with on-prem or private cloud warehouses.
3. Data prep is much easier than before
An important prerequisite for augmented analytics is having clean and consistent data, and this is particularly important in the era where data is collected from multiple sources and in multiple formats. If data is not transformed and prepared correctly, it will affect the accuracy of machine learning models.
But doing this isn’t easy and takes time – between 70-80% of a data engineer’s time. This in itself causes a bottleneck and limits the amount of data that can viably be used for analysis. And as the number of data sources increases, this problem gets increasingly bigger. This has typically been a challenge for many companies, particularly in the era of SaaS.
This challenge has created space for a proliferation of data modelling and transformation tools that make it much simpler to transform, clean, aggregate and join data in a warehouse, all within one platform. These are simple to use, so they can be used by data analysts and other non-data engineers. Tools such as DBT and Dataform (the latter recently acquired by Google) are designed to work with the most common public data warehouses, serve as a centralised hub for transformations and model management, and act as a central point of truth so that all data is consistent across the organisation.
4. The impact of Covid
Covid has affected businesses of every size in every industry. Working through damage control and reevaluating priorities and ways of working has magnified the need for augmented analytics as part of the BI stack. Furloughs and Layoffs
According to a TDWI survey of over 200 companies, 13% said that they were either furloughing or laying off data and analytics professionals, and this was regardless of industry or size. Furthermore, 40% said that there was an active hiring freeze for data and analytics professionals, with only 5% actively looking to hire more.
At the same time, data and analytics teams said they were being asked more questions by the business and asked to run analyses more frequently. This makes sense given that now more than ever, businesses need to retain customers and understand what is happening in their business, but it places a huge amount of pressure on analysts to deliver insights quickly.
Whether we like it or not, Covid-19 has accelerated a new way of working where organisations operate remotely with teams that are dispersed. According to a 2020 report by Forbes, 97% of enterprises said that all their employees were working from home and so they have an increased need for cloud-based self-service BI applications and tools, and this is expected to continue even after we have seen the worst of the pandemic. As a result, business users would benefit from being more self-sufficient in getting the answers they need to ad-hoc questions about the business and being able to derive insights from data themselves. Gone are the days where they can quickly walk over to ask the BI team to ask a question or quickly generate a report.
Budgets and Spending
IT budgets have also been affected by the pandemic. In the middle of 2020, Gartner predicted that IT budgets would fall by 8%, and although spending is expected to increase in 2021, it is predicted that CIOs will spend more on subscription products and cloud services to lower upfront costs and mitigate the impact of any further downturns.
5. Data culture in organisations is maturing
There is an increasing shift in the role of data analysts and BI teams from spending their time collecting and analysing data for the business, to empowering business users themselves with the skills, tools and support needed to be more self-sufficient, thus decentralising the process of generating insights and ensuring everyone in the organisation is data literate. This in turn frees up analytical teams to work on other projects and custom developments, and allows for scalability as the amount of data a business collects increases. According to Forbes, in 2020, 62% of businesses said that self-service BI and the ability for business users to perform their own analytics was extremely important as workloads for analyst teams continues to grow.