Increasing complexity is part of the natural order. Instead of fighting it, we should learn to take advantage of it.
Thinkstock
It has become cliché to say the world is getting more complex with every passing day. Increased access to vast amounts of information, social media, and ubiquitous computing makes it seem as if the complexity of our world is increasing faster than we can comprehend it. But technology is not driving complexity, it is only making complexity more visible.
The reality is the world has always been a complex place, and human organizations have been complex since humanity first started forming social groups millennia ago. No amount of technology will reduce the complexity your organization is facing. But understanding complexity and learning to use it to your advantage will help you harness it and make it work for you.
Understanding complexity
When I talk about organizational complexity, I am not referring to the fact that organizations are broken up into sub-organizations where the responsibilities and reporting relationships among the leaders of those organizations may be arcane. I am using the term in a more formal sense, the sense used by researchers in the field of complex systems.
A complex system is one composed of a number of active agents that interact with one another. The agents may all obey the same rules, or they may each be guided by a unique set of rules. The key characteristic of complex systems is that the interactions of all these agents, each acting according to its own rules, produces something called “emergent behaviors.” An emergent behavior is one that arises from the interaction of all the agents with each other, not from any rules designed to produce that behavior.
Put simply, the organization as a whole acts as if it had some overall collective purpose. That may sound good at first, but the key thing about emergent behaviors is that they are very rarely predictable and there is no assurance they help achieve the goals of the organization’s leadership.
A simple example of emergent behaviors is the way a school of fish moves. Hundreds or thousands of fish move as a single group, turning or diving to avoid obstacles or predators, moving as if there were one lead fish directing their actions. But there is no leader; those behaviors emerge spontaneously from the various instincts of each individual fish combined with their interactions with each other and their environment. An excellent explanation of emergent systems is Growing Artificial Societies: Social Science From the Bottom Up (Epstein & Axtell, 1996).
Why enterprise architecture has not helped
Over the past couple of decades, organizations have tried to overcome complexity using enterprise architecture. The results have been disappointing, and I am hardly the first person to notice. Some authors have asserted that enterprise architecture is broken and in need of reform, while others have questioned whether the field is dead due to its consistent failure to deliver on its promises.
While both of those articles diagnose the problem correctly (i.e., enterprise architecture is not delivering the promised value), they do not quite diagnose the cause correctly. The cause of the problem is that enterprise architecture as traditionally practiced is based on the assumption the enterprise can be centrally planned and managed by a small architecture team.
Every enterprise architecture framework on common use today can trace its heritage back to the Zachman Architecture Framework, first published in 1987. The Zachman Framework and all its progeny are based on the assumption that we can start with enterprise-level requirements and progressively decompose them into implementation details that can be handed off to developers who will build the systems that will fulfill those requirements.
There are three main problems with this approach to enterprise architecture. First, it assumes that we have the time to specify high-level requirements and progressively decompose them over time until we are ready to begin system implementation. This approach made sense in the days when systems tended to be large, custom-built affairs. But this approach is not tenable in an environment where most or an organization’s needs are fulfilled by off-the-shelf software and the remainder are built using Agile development methods.
In today’s dynamic marketplace, a faster, more responsive architecture methodology is essential. There are a number of techniques that try to marry traditional architecture with Agile development, but while they have proven reasonably effective for system implementation, they have not shown commensurate value when applied at the enterprise level.
The final reason traditional architecture frameworks are failing at the enterprise level is that they fail to take emergent behaviors into account. These architecture frameworks assume that the enterprise they are modeling can be centrally directed, and if the architects are sufficiently careful in laying out their plans, the enterprise will function in a predictable manner. Most often, the enterprise exhibits a stubborn refusal to conform to the architects’ predictions (i.e., the emergent behaviors were not predicted by the architects).
Making complexity work for you
If traditional enterprise architecture is not delivering the promised benefits, then it is time to try another way. This is not to say that enterprise architecture in general is a bad idea, but rather that we need something other than traditional frameworks to make enterprise architecture effective.
First and foremost, the enterprise architecture should focus on the goals of the organization. Most organizations do not have the creation of detailed models of the organization and its systems as a primary goal. Instead, their goals are more likely to be increasing sales, eliminating duplicative systems, reducing administrative overhead, or similar business-centric goals. These goals must be the focus of the enterprise architecture. Building and deploying systems is not a goal, it is a means to reach one or more goals.
An enterprise architecture that is focused on system implementation is focused on the wrong things; it needs to focus on achieving the real goals of the organization. Toward that end, from the enterprise perspective all systems are black boxes that consume inputs, produce outputs, and generate some effect. All that matters to the enterprise is that the effect advances one or more business goals, that the inputs are available within the organization, and the outputs are usable (either in themselves or as inputs to another system). The enterprise architecture should focus on documenting how those systems and their interactions contribute to achieving the organization’s goals.
Second, the enterprise architecture must take account of the organization’s emergent behaviors. How the architects predicted the systems would be used and how people would respond to policy or organizational changes does not matter; what matters is how they did respond. Are systems being used for unexpected tasks? Are users ignoring systems and doing the work manually?
Observing and understanding these emergent behaviors will give the architecture team valuable information about how the organization is progressing toward achieving the organization’s goals (or not progressing, as the case may be). As the architecture team begins to understand these behaviors, they can respond appropriately, supporting and reinforcing positive behaviors and initiating efforts to change negative behaviors. Instead of a modeling exercise, the enterprise architecture effort becomes an operational tool that leaders can use to steer the organization.
Finally, measuring and responding to changes in enterprise behaviors and progress toward the established goals over time is essential. The goals must be clearly defined and measurable to ensure accurate measures of progress are possible. Behaviors must be monitored over relatively long terms, usually months to years. People do not respond to policy changes or new systems instantaneously. It takes time for them to adapt their individual routines and behaviors; as each person adapts the organization’s emergent behaviors will also adapt. By observing how the organization’s behaviors change, the leadership team can respond appropriately.
Trying to eliminate complexity is an impossible task, and traditional approaches to enterprise architecture have proven ineffective in dealing with it. By thinking about enterprise architecture in a new way, we can make that complexity work for us, and harness emergent behaviors to help achieve an organization’s goals.
Originally published in CIO.
