Ready to learn Blockchain? Browse courses like Blockchain for Finance Professionals developed by industry thought leaders and Experfy in Harvard Innovation Lab.
This article is the fourth of five articles that discusses a new paradigm for enterprise software development. Specifically, these articles are focused on how we can design systems from the bottom up to increase our efficiency in creating systems and to allow us to easily exchange data between systems.
In the first article, I talked about a new type of primary key that is essential in order to allow us to exchange data between systems (https://www.linkedin.com/pulse/primary-keys-paramount-our-future-enterprise-systems-blair-kjenner). The new primary key will be assigned to a record when it is created and it will follow that record from system to system.
The next key concept I introduced was core data models. They represent the common data that we find in every system. Examples of core data include contacts, finances, assets, contracts, and activities. https://www.linkedin.com/pulse/core-data-models-critical-future-enterprise-systems-blair-kjenner. All systems in the new paradigm will start with the core data structures which will then be augmented to capture the specific data for any system.
In the third article, I discussed the need for record governance. Record governance ensures that no two systems can update the same record at the same time. (https://www.linkedin.com/pulse/data-governance-key-new-paradigm-enterprise-software-blair-kjenner)
When you combine these three concepts, systems will be designed based on the core data models. The core data models will then be augmented to manage data to meet the specific needs of any organization. Each record in the system will be assigned a primary key that will stay with that record, no matter which system it is transferred to.
By default, when a system creates a record, it will have full governance to update it until it transfers that governance to another system.
The fourth cornerstone deals with how data is exchanged between systems. Exchanging data will be primarily accommodated by a Blockchain broker that will ensure the systems are following the protocols of the new paradigm. It will be used for requesting data updates from providers and for broadcasting updates.
A secondary method for exchanging data between systems will be QR Codes. QR codes will be printed on contracts, invoices, purchase orders, and assets, to mention a few. The QR codes will contain the primary key and core data for a particular record type. This will allow external systems to read data, setup records, and then communicate with other systems based on known records.
The key advantages of the new paradigm for exchanging data are:
>
All systems that are designed based on the protocols of the new paradigm will inherently have the ability to exchange data with any other system that is designed according to the new paradigm protocols.
Organizations will be see a dramatic improvement in efficiency as a direct result of not having to key and re-key data between systems.
Management will have access to data across all functional, geographic and time dimensions.
While the new paradigm might seem challenging to implement, in reality it is no less challenging than what we are doing now. It is simply adding in the disciplines for creating systems according to standards that are designed from the bottom up to address specific needs.
It is really no different than when the accountant’s profession recognized the need for Generally Accepted Accounting Practices (GAAP). Prior to that there were no debits, credits, income statements, or balance sheets. Every accountant devised their own method for tracking finances which resulted in a tremendous amount of duplication of effort. Furthermore, it was impossible to compare the finances between two organizations.
The new paradigm protocol that is described in this article is intended to start the conversation about creating generally accepted practices for the creation of software.
I appreciate your time and hope you will have time to read the article in full. I look forward to your comments. Hope you like the article!
INTRODUCTION
>
The fourth cornerstone in the new paradigm is how data actually gets exchanged between systems. There are four key components to this discussion – what triggers the transfer, how it actually gets transferred, what gets transferred, and how is it received.
To put these questions into perspective, I will be using the following example which discusses how data will be exchanged between entities in the Oil and Gas industry.
The Oil and Gas industry is continually exchanging data between parties. These parties include the operator of the well, primary owner, partners, suppliers, labs, and the government (to mention a few).
In the new paradigm, each oil and gas well (well) will have a primary key that is unique to that well. The oil and gas company that first created the well will assign the key. It will have governance to the core data about that well. The core data will represent the core attributes about a well that you will see in almost every system that deals with oil and gas data.
When the core data about the well is communicated to the government, it will include the primary key plus core data. The government then will append data they require to the core record. Other partners of the well will also inherit the well record and its core data. They too will be able to append data that they require.
Even suppliers can leverage the primary key and core data for the oil well. When a company performs certain services on a well, they will use a smartphone or tablet to read the QR code on the well. This will provide them with the core data about the well and the unique primary key. They will then submit their service records to the Oil and Gas Company. The service records will include activity and financial data, which will be linked to the appropriate well and contract in the Oil and Gas company’s database.
All data about the well, including ownership data, contractual data, volumetric data, lab data, activity data, and financial data will be shared using the primary key concepts, core data model concepts, and record governance concepts.
Eventually the Oil and Gas Company will sell the well to another provider. When it happens, they will transfer the record governance for the main well record and subsidiary records such as ownership records to the new company.
TRANSFERRING DATA
>
What triggers a transfer?
Data about a record can be transferred on a push or pull basis. In the case of a push transfer, there will be a number of subscribers to the well. The primary subscribers interested in changes to the core data and subsidiary data about the well will be other owners in the well as well as the government. Having said that, any organization that is interested in the state of that particular well can be a subscriber. When a change is made to the well, all subscribers will be sent the updated record.
From a pull perspective, any system can request updated information about the well. The Blockchain Broker will provide information to requesters from the system that is currently governing a particular record. Another example of pulling data is when a supplier uses a smart device to read QR Codes that contain the primary key and core data.
How does it actually get exchanged?
There will be four key mechanisms for exchanging data between systems.
Blockchain Broker
The Blockchain Broker will be a critical service for exchanging data between the new paradigm systems. It will ensure that all data exchanges follow the rules of the protocol. They include things such as ensuring that a system is using its appropriate system ID or ensuring that a system is not pushing an update for a record that it doesn’t govern.
The Blockchain Broker will take care of broadcasting updates and fulfilling requests from systems. It will also store data updates for systems that are offline and then deliver the updates to the systems in the appropriate order.
The Blockchain Broker will also provide valuable information to all systems such as who is governing a particular record or which system does a particular system ID refer to (of which some will be private).
The Blockchain broker will send and receive data from all systems using web services.
QR Code
QR codes will be an effective method of communicating data between systems. Based the example above, QR codes can be printed on Wells, Assets, Contracts, Purchase Orders, and Invoices. Every QR Code will contain the primary key and core data for the record. Any system that is developed in the new paradigm will be automatically capable of reading in this data, connecting to its own data, and then sharing it back to outside parties.
Database to Database Communications
The only time database-to-database communication will occur is when all databases are under the control of a single organization. For example, a large organization that is regionalized will be able to exchange data directly between regional systems and the head office system. Another example would be departmental systems within government.
What gets transferred in a data exchange?
The contents of the transfer record will include the following:
Primary Key
The primary key of the record which will be guaranteed to be unique across all systems.
Data Dictionary Record Identifier
The data dictionary record identifier will identify the specific record in the data dictionary that the primary key was related to. For core data, any system will have full details on the record. Examples of core data include contacts, contracts, assets, activities, etc. Industry specific core records will also be available.
Key Value Pairs
The key value pairs will contain all values on the core data core that was updated. Each key value pair will include the primary key for the attribute within the dictionary, descriptive details about the attribute, and the value of the attribute.
How it is received?
When records are received into a system, actions will be programmed for each system. The actions can be something as simple as updating a record or it can be as complex as initiating a series of workflow processes within the organization. For example, if a well was shut-in (i.e. closed), it can trigger many manual processes to occur throughout an organization.
TECHNICAL CHALLENGES
>
Certainly there are many technical challenges that will need to be resolved. Here are a few of them and the respective possible solutions.
How are deletion of records tracked?
All records in the new paradigm will have a deletion flag on them. When a record is flagged for deletion it will be communicated just like any other attribute update for a record. This will cause the record to be flagged for deletion in the destination systems as well. There will be maintenance processes in systems to physically delete records that are flagged for deletion. The considerations for physically deleting a record, will be that the records have no child records and that no other parties are interested in future updates of the record. The change history will keep information on records that are physically deleted.
What happens if a foreign key is referenced that doesn’t exist in the destination system?
It is possible that a record being communicated to another system may reference a foreign key that does not exist in that system. In such a case, a subsequent call will be made to the source system to request core data about the master record which the foreign key was referring to. Of course, the master record can have foreign key references that are unresolved as well. It will need to cascade until all references are resolved. The destination system will either use the foreign key reference in its current form with its primary key as is, map to an existing foreign key reference if it exists, or create its own reference for that value. The new paradigm integration specialists will work through these scenarios and code the appropriate business rules based on the needs of the organization.
What if an attribute is referenced that doesn’t exist in the destination system?
It is possible that a destination system may not have an attribute that is specified by a source system. This can happen if the destination system is operating at a higher version level for the core data record than the source system. In this case the destination system, may keep a record of these attributes and then update the attributes when a new version of the core data models is applied.
A key rule for core data is that new versions of it will be backward compatible. As a result, a system on a newer version of a core data record will be able to communicate changes to a system with an older version but not the other way around.
How do we know which system currently governs the record?
The system can call the Blockchain Broker with the primary key and the primary key of the record from the data dictionary. If record governance has never been transferred for a record then the governor of the record will be the system that created it.
How do we queue updates for systems that have been offline for a period of time?
It is possible for systems to be offline for a period of time. In such a situation, any broadcast updates will be missed. To solve this issue, a brokerage service that will hold updates and then communicate them to the destination system when the offline systems are connected will need to be established.
The brokerage service will also make sure that updates to the systems are communicated in the order they were broadcasted.
How do we communicate with systems that aren’t developed in the new paradigm?
It isn’t realistic to believe that all systems will be developed using the new paradigm. In order to accommodate old paradigm systems, interfaces can be built with the brokerage service. Once an old paradigm system has an interface to the brokerage system, it will be able to accept updates from any new paradigm system. The interface to the old paradigm systems will be one way–that is, old paradigm systems can accept updates from the broker but not push them, unless they build interface modules that creates data using the primary key and core data model protocols and observes the rules of data governance rules of the new paradigm.
HOW IS IT DIFFERENT THAN WHAT WE DO NOW?
>
There are vast differences between how the new paradigm will exchange data and how it is done now. Here are a couple of key differences.
Consistent Primary Keys
In the old paradigm, every system has its own method for defining primary keys for records. In our Oil and Gas example, if you have a dozen systems perform tasks related to wells, then every system will have its own primary key for a given well. It means you cannot easily connect data from one system to the next for a particular well.
In the new paradigm, a primary key is assigned to a record when it is created and lives with that record as it is transferred from one system to the next. This allows us to connect all common data from one systems to the next.
Standardized Database Designs
In the old paradigm, every development team designs databases in their own unique way. That makes integration of all data challenging, if not impossible. (https://www.linkedin.com/pulse/what-definition-word-integration-blair-kjenner)
In the new paradigm, systems at their core, will be designed on standardized core data models. Each team will then augment the core data to address the unique needs of their customer. The core data will establish patterns for naming conventions and structural patterns for complex structures such as recursive structures and temporal data.
Databases designed based on core data models will allow common functionalities to be created once and shared across many systems. It is also one of the critical cornerstones to being able to share data between systems.
BENEFITS OUR CUSTOMERS WILL SEE
>
Organizations that standardize their systems with the new paradigm will see dramatic improvements in efficiency and have opportunities for information management well beyond what is possible now.
Efficiency Improvements
The efficiency improvements will come primarily from the fact that organizations not have to key, re-key, and reconcile data between systems. Data will be entered once and then exchanged between systems as required. For example, suppliers will be able to create data which can be shared with their customers. Also, data can be captures on smart devices and then communicated to the main systems without changing primary keys.
Create small systems that can communicate
One of the greatest benefit of the new paradigm is that organizations will be able to implement smaller systems that can communicate. This not only reduces risk, it also simplifies data security. As an example, an organization that is regionalized can create a system for each region that can communicate with between themselves and with the head office system. This eliminates the need for implementing complex row level security features and allows the head office system to have access to detailed information for gaining an organizational perspective on what is happening.
Management Information
In the old paradigm, management information is often created on a system by system basis with little ability to compare data across all business systems. In the new paradigm, management will have access to reports and dash boards that cross all functional, geographic, and time dimensions. For the same reason, artificial intelligence will be able to take advantage of well-organized and consistent data across all business systems and extract and merge data from regional databases to gain a holistic picture of what is happening in the organization as a whole.
This will be possible as data from its origin to end will be assigned a primary key only once. It will live with the primary key regardless where the data travels to. That means all data in all systems can be connected for reporting purposes.
SUMMARY
>
While creating the new paradigm will be a challenging task, we have seen those transformations in other industries. For example, imagine accounting practices before there was Generally Accepted Accounting Principles (GAAP). Back then, there were no debits, credits, income statements, or balance sheets. Every accountant devised their own unique method for tracking finances. There was an amazing amount of duplication of effort. It was almost impossible to compare finances between organizations.
Once GAAP was implemented, the efficiency of accounting improved and analysts could then compare finances between organizations.
The software development industry is currently operating with no generally accepted practices. In fact, you hear statements like “Data Models are like underwear. No one wants to wear anyone else’s”. This sad but true statement is exactly how we approach systems. Until we change this approach, we will always struggle to create application software efficiently for our customers.
It will be a challenge to create a new paradigm for application software development but no less challenging than rebuilding application frameworks over and over, rebuilding the same functionality in every system, or trying to integrate systems that were never designed to work together.
Now is the time to seriously consider a new approach to application software. I look forward to your comments and hope that you like this article.
Originally published at: LinkedIn
