Ready to learn Data Science? Browse Data Science Training and Certification courses developed by industry thought leaders and Experfy in Harvard Innovation Lab.
The four “Vs” of data are well known – volume, velocity, variety and veracity. However, Data Warehousing infrastructure in many organizations is no longer equipped to handle these. The fifth elusive “V” – value – is even more evasive. Meeting these challenges at the scale of data that modern organizations have requires a new approach – and automation is the bedrock.
For CDOs, defining the Data and Analytics Strategy for their organization is increasingly the most important aspect of their job, with 86 percent ranking it their top responsibility in 2017 (Gartner). It’s all about finding methods of using data for value creation and revenue generation, which occupies 45 percent of a CDO’s time.
This means harnessing the growing beast that is data in a way that is practical, manageable and useful. That’s where the Data Warehouse comes in, providing a centralized space for enterprise data that business users, including the CDO, can use to derive insights.
Creating a successful Data Warehouse, then, is critical for CDOs to succeed in monetizing data within their organization. However, the traditional waterfall approach to Data Warehousing, first introduced in the 1970s, delivers only a fraction of the value that it could potentially offer. Instead, the approach needs to evolve to become more responsive as organizational needs change, addressing new data sources and adapting to business demand.
Using automation software to design, develop, deploy and operate Data Warehouses is providing far-reaching value to business leaders. This change is positioning IT to incorporate timely technologies and new data sources more easily, as well as flex when business needs demand it.
The CDO’s Role in Re-invigorating the Data Warehouse
As the central storage point for enterprise data, the Data Warehouse is invaluable for providing business users with information. However, as users become more aware of the potential benefits of data-driven decision making, the gap between user expectations and the Data Warehouse’s ability to provide current, consumable data in a timely manner has grown. Businesses want insights from data far faster than previously.
This is exacerbated with the growth of new forms of data, particularly semi-structured or un-structured information, including social media data, sensor data, real-time messages and client communications, and video/audio files.
Traditionally, Data Warehouse development and evolution required long-cycle IT projects, incongruent with the needs of a more agile project design and build environment. As the change agents for digital transformation, CDOs must lead in the re-architecting of Data Warehouses, from artistry to acceleration and automation in order to improve business time-to-value.
Practical Steps for the Successful Implementation of an Automated Data Warehouse
As IT departments are expected to do much more with much less, processes need to change. Data Warehouses can no longer be created “artisinally” – rather than crafting a complex infrastructure with unique configurations and a long lifespan, IT teams need to focus on producing an adaptable decision support infrastructure. This means creating a Data Warehouse that can transform, with ease, as business needs change. Here are five steps for CDOs to help their company achieve this:
-
Understand the desired outcomes
Before making any decisions as to the future of your Data Warehouse infrastructure, CDOs need to ensure they understand the specific challenges the business teams are facing where data could help. In essence, the Data Warehouse automation and modernization program need to be built around enabling decision-making that will lead to differentiation in the market place.
According to a recent survey, re-alignment to business objects is the top reason for Data Warehouse modernization, selected by 39 percent of respondents. By enabling collaboration between business teams and IT teams, the CDO helps chart the course for how business goals and technology meet. In turn, this will lead to overall business transformation, accelerated through the new Data Warehouse’s approach to data-driven decisions.
-
Understand what you have already
Most organizations already have sophisticated Data Management tools deployed as part of their infrastructure – however these may not be working to the fullest of their abilities. Organizations already using SQL Server, Oracle or Teradata, for example, have a range of data management and data movement tools, already within their IT real estate, which can be automated and leveraged more effectively as part of a Data Warehouse automation push.
However, in that inventorying process, CDOs should be ensuring they have considered the capacity requirements of their Data Warehouse. Data is expected to continue growing exponentially, so while the Data Warehouse may be fit for purpose today, it’s important that the automation processes, storage requirements and general infrastructure is of a speed and standard capable of handling this in the future too.
As part of this, Data Warehouse automation needs to integrate with the business as it currently is and will realistically be in the future, rather than the business as the IT teams wish it might be in an ideal world. CDOs need to encourage their teams to understand the data that is available, and the automated analytics and evaluation processes which can be used to meet specific business priorities. To support this, the Data Warehouse automation strategy needs to be designed not just for an ideal set up of data, expertly managed and curated, but for the realistic “messiness” of the business data landscape.
-
Automate efficiently
Data Warehouse automation, as with any other large-scale transformation project, requires resources – and these are often scarce due to strict budgets and competing priorities. This means that CDOs need to think long and hard about what actually should be automated in order to free up man hours in the future.
Some good examples of where automation is a cost-effective step could be in hand-coding SQL, writing scripts or manually managing metadata. All of these are systematic processes, where Data Warehouse automation can either eliminate the need for human involvement or dramatically accelerate the process.
-
Embrace change
CDOs should look at Data Warehouse modernization and automation as an avenue of constant, on-going development. As business needs change and new data sources emerge, CDOs need to be able to re-strategize different parts of the infrastructure to match.
Similarly, to minimize disruption and ease the transition for business users, CDOs should look to take a staged approach to the initial automation and modernization process, with a set schedule of when different requirements will be met. Post-production change is inevitable due to evolving business needs, new technologies used and continuous improvement desired. Change needs to be planned for.
At the same time CDOs need to prepare for the human change that automation will create. In business teams, users can be re-deployed to double down on analyzing business intelligence and translating insight into business value. In the IT teams, automation provides new capacity to plan for the future – looking at new analytics tools, or planning for smarter, better ways to deliver on business priorities further down the line.
A Data Warehouse Automation Mentality
Data Warehouse automation is not solely software you buy. It’s a philosophy and culture you implement. Tools and technologies form the bedrock of the processes, but a Data Warehouse strategy requires strong leadership, a transparent process, and an unrelenting focus on the business’s end goals in order to succeed.
Without robust Data Warehouse automation, businesses will struggle to capitalize on the potential of data and its associated technologies. As the strategic lead for data-driven transformation, and the change agents across both business and IT teams, the responsibility falls to CDOs.
Professionals in this role need to understand, strategize and execute on the way that large-scale data usage will influence future business decisions. The adaptability of the supporting data infrastructure can either be a CDO’s greatest weakness or greatest asset. Use the four steps covered in this guide to ensure it is the latter, and to achieve the ultimate goal of any business investment – value.
