Ready to learn Big Data? Browse Big Data Training and Certification Courses developed by industry thought leaders and Experfy in Harvard Innovation Lab.
In any organization, it’s plain to see there are distinct differences in the way business executives and IT professionals think about their company’s data. The disconnect is a result of perspective, and manifests itself in a lack of communication that produces further confusion as IT systems evolve to meet the needs of a rapidly forward-charging business. The fruit of this misunderstanding often includes friction between the business and IT.
Both groups want to help and see their organization succeed, but require a decoder ring to help them understand each other. The decoder ring exists. It’s not magic or some lofty ideal – it’s an application created by the right people with the right experience to confront and address this known problem.
More Data, More Misunderstanding
In any organization, data sits in multiple systems, separated by technical and functional boundaries. These isolated systems, referred to as data silos, exist for good purpose and reason – and do their job well. The problem is that to run the business successfully and make informed, business-driving decisions, business stakeholders require a cohesive and unified view of data sourced from all those systems.
Business thinks in terms of entities and relationships, but more as high-level artifacts, like any recognizable object in the real world.
Entities and relationships to a business user are simple. They have customers. A customer places orders. Orders can be placed online or in-store. Orders have products. Products are composed of parts. Parts have suppliers, etc. There are rules for how customers can interact with orders and products, and rules for how products are sourced. And these rules are reflected in the relationships between these entities.
The problem is that these simple entities and relationships get mapped to very complex underlying storage systems – CRM, ERP and POS systems just to name a few. The minute this occurs, business is disconnected from their own data and becomes reliant on IT to provide them with any view of their own data, especially if the view requires data from multiple systems.
IT thinks in terms of entities and relationships, too, but as mapped to complex, underlying storage systems in a form suitable for upstream and downstream applications and consumers. IT understands the complex, underlying storage systems and the applications on top of them. It understands the data processing flow and governance of the individual systems. But IT’s view of data can be fragmented and disconnected as well, adding to the disconnect.
Above the Silos of Data are Silos of People
Every organization is like a beach ball. Where business wants to see the entire ball and all its stripes (e.g. how does data from all the systems and functional areas fit together to answer the questions about the entities and relationships we care about?), IT departments often just see one particular stripe of the ball; the stripe associated with the system(s) they support for a particular functional area of the business. Any IT department might not see all the entities and relationships across the business clearly, but they’ll have considerable expertise with a subset.
To bridge the gap, logical models are created. These, while well intentioned, are often created by technical folk. The hope is that by understanding a conceptual abstraction layer, business and IT will be able to bridge the gaps in the technologies and functional areas to better communicate together better. This does help a bit, but it’s not enough.
Some Not-So-Secret Secrets
- Every organization requires some unified view of data from disparate systems to successfully run their company.
- Every organization thinks they’re unique, no one else is like them and that their Data Management pain is unique.
- This just isn’t so as every organization has had access to the same technologies over the last 30-40 years and built similar variations of the same data processing beast. As a result, common patterns have emerged. It doesn’t matter if you’re in retail, healthcare, financial services, manufacturing, etc. There are common Data Management problems across all industries.
- With common problems identified, common solutions can be created.
- While better shovels and wheelbarrows for moving data around have been built (NoSQL databases, yourFavoriteJavaScriptFramework.js, etc.) they are technical solutions and therefore require technical expertise to begin working with the data.
- For a successful unified view of data to be implemented, business absolutely MUST have a voice at the data unification table with IT. It’s hard to bring requirements to the table when you’re not able to communicate the rules of data as you understand it to the group that is the gatekeeper for your data.
- If implementation of any data unification solution begins by talking about REST APIs, programming languages, or schema modeling: business will be automatically disconnected from the data unification discussion.
The good news and a bit of a secret that starting to emerge out of the shadows: some of us out here working with your data long enough have seen the familiar patterns in the people, processes and technology, and are now building the applications that enable business and IT to work collaboratively together to unify data successfully.
The Problem: Data Silos
Traditional Data Integration begins by first surveying the IT landscape and inventorying all the data supporting systems and attributes that define the data (column names, metadata, mappings, etc.). The idea is then you build the uber-schema for all the data you want to integrate, in which you think will support answering all the questions business could possibly ask at the point. This never works, as it takes months to develop a model, then you get to write ETL jobs to load the data, but business has moved on and requirements changed as they have new questions, requiring updates to the target model. Source systems change as well or are acquired, requiring updates to the ETL. Essentially there’s a lot of churn, and organizations find themselves three years into a two-year project with nowhere near the data integrated that they had hoped to.
To get to results that allow business to get their hands on their own data, Data Marts and their supporting ETL pipelines are setup for subsets of silos. These result in more complex storage and more conceptual schematics as well. And they also have proved not to be enough. Something else entirely is needed.
The Solution: A Business Data Model
With the right application, you can turn this all on its head and rapidly get to unified data and real results. Assuming you have this application, you can start to create a business data model.
A business model of data captures data in a coherent structure similar to how normal human beings, not machines, think about data. Business rules are defined and applied within the structure of the model. This model is easily understood and validated by executives and subject matter experts together.
This business data model is created with input from both business stakeholders and IT leadership. With the right application to create this model, it effectively acts as the decoder ring between IT and business to help them understand each other’s point of view.
This isn’t a theory or lofty goal. I’ve seen this in action. And it isn’t complex. Anyone who can use a whiteboard can use an application interface to create this model.
To create the model, start with the assumption your org has the data you need. Don’t look at it. Create the model decoupled from the underlying sources. This can be done drawing circles and lines for entities and relationships in a Whiteboarding application.
As business and IT communicate with each other via the whiteboard model they are creating, they gain empathy and a clearer view into the challenges each group has in managing their company’s data and getting the answers they need out of it to support the business.
After the model is created, another application component can then provide for mapping the required sources to the conceptual whiteboard model, and make it physical. As a result, the conceptual model of data and physical storage of data are closely aligned, making it simple for non-technical users to understand, communicate, and interact with the data for their business.
The Big Picture (aka Digital Transformation)
With TOGAF, improving business capability begins by defining the business architecture. The right application allows business and IT to model their data together in the best interest of the business, and persist the data in a way that respects the rules of business as manifested in relationships while aligning the conceptual and physical models of data. This provides for more coherent information systems architectures (which include the data and application architectures) to be defined, which ultimately informs the supporting technology architectures. The baseline is respected without disrupting any existing silos, while providing a clear path to a target architecture based on data.
A truly data-driven enterprise.
