Why Architecture Matters

Grapple With Something

A response is our written or verbal answer or our reaction to something. Some action comes ahead of a response being necessary. In its simplest form you can think of a question as the action that precedes an answer, where the answer is the response.

I have been wondering if in a situation where there is no action, if the lack of action, what I will call "whitespace" here, if that constitutes an action. Is non-action an action? Does non-action require a response?

Think about a recent conversation you have had, could be with anyone, there are inevitable pauses in the conversation, where the silence is awkward, it is in effect whitespace, do you fill it or do you sit in the silence. In a situation like this are you disciplined enough to stay quiet or do you have to fill the void. Do you walk away, and the conversation is over?

I dont know that I have the answers, but I have some thoughts.

I wonder too, if whitespace is something we create because of our inherent predisposition to play defense and wait to see what will fill up the whitespace so that we can respond. There is always someone or something waiting to fill the whitespace, we wait on them to do that work for us.

What I am grappling with is a recurring question, or theme, in our industry and it is around data governance. There are many questions about this, many think it is a tool, but I think of it as a response to a problem that has been created over time and a problem that generally gets promulgated each time a new effort related to data is undertaken. I have written about this topic before and will probably do so many times in the future because it is a topic that is not settled nor in my opinion well understood. I think there are decades of white space that this area of policy is trying to fill, it is complex and nuanced, and maybe some change in how we think about it is required.

Let me attempt to create an analogy that we can all visualize and agree upon. If you were to set about building a home or a commercial building in the US right now, you would have to go through a series of required compliance steps. This is permitting and inspections mainly, but for many buildings you also have to submit engineered drawings or architectural drawings. The architecture and engineering typically go hand in hand and the permitting and inspections go along with those plans generally. All of this is to ensure that the house or commercial structure have reasonable foundations and mechanicals. Things like stud spacing, mechanical chases, and plumbing standards can be expected to be adhered to. Now there are variations in some of this such as not putting plumbing in exterior walls in colder climates, etc. My point is that there is general adherence to some standard. Architecture and engineering come up front working with the needs of the occupants to ensure that the structure meets the need. Then comes the build and the structure, and finally the finish work. The permitting and inspections follow along as the process moves along. Generally, this is the build process custom or prefab.

A few definitions here to set the stage are necessary.

Governance - the act or process of governing or overseeing the control or direction of something

Policy - a set of ideas or a plan of what to do in particular situations that has been agreed to officially by a group of people, a business organization, a government, or a political party

Data Governance - Data governance is a system of policies, procedures, and practices that ensures data is managed effectively and responsibly throughout its lifecycle. It encompasses the processes, roles, and technologies needed to guarantee data is accurate, reliable, secure, and accessible for its intended use. Essentially, it's about making sure data is a valuable asset, not a liability.

Now, the build process associated with a home or commercial structure is very much like the build of a business system. Business systems all track data in one form or another. Those business systems should be treated like the build of a home or commercial structure in my opinion, there should be data architecture and engineering up front, then should be a build process; both of which are worked in parallel with the data governance and policy aspects.

This all seems reasonable until you look at reality. Reality in business systems, generally, is quite divergent from the form of building noted above. I am not naive to the fact that some home and commercial building skit the standards and governance as well, but by and large there is compliance within that arena, business systems not so much. As a result, data governance and its resulting policy then become a response to things that were not planned for, unintended consequences, and white space.

Governance and policy in general are a response to things we did not plan or prepare for. By virtue of it being a response, it is always late to the party and outdated the minute it is put in place. Governance fails because the rate of information being generated, good or bad, results in outdated governance before it is completed or issued.

We have seen examples of procurement data tracked in SharePoint InfoPath forms with PO Numbers and Requisition Numbers that when requested to be put into a data warehouse, where all the other procurement data resides, result in questions about governance and policy regarding that data. There are a million questions that could be asked, but the main one is what was the architecture that led to procurement data in SharePoint, and where was the governance and policy when this was happening? There is already data governance within the financial system where the procurement data exists, putting that same procurement data into another system outside of the financial system results is unnecessary data governance issues because the architecture was not adhered to as business decisions were made. It would be the equivalent of building the first floor of a house on one lot and then putting the second floor of the house on another lot in a completely different subdivision.

The world of business systems and there resulting data architecture are not taken nearly as seriously as they need to by the businesses they serve. Data architecture and the decisions around where that data goes needs to be taken as seriously by business, it is something that is not to be thought of after the fact. The approach we see is that governance is attempting to be applied after the fact. Governance and policy applied after the fact is seen as optional and without any real validity. Additionally, its optionality leads to sprawl, which then leads to more governance and policy, it is a vicious cycle. Data governance should be considered as integral to architecture, and architecture should be front and center as business systems are designed and / or purchase of systems are contemplated.

I had the opportunity to sit in on a Salesforce demonstration recently, all their products and how they could integrate and solve all the customer problems. The part of the demonstration that particularly caught my attention was the discussion of the MuleSoft application and its ability to have custom connectors for all the different pieces of software out there that contain critical data for the business. I sat there and I wondered if anyone else wondered how we got to this point where tooling was so easily purchased throughout the enterprise and nary a thought was given to architecture and how that data would ultimately be integrated into the overall enterprise data architecture. It feels like as long as the software claims to have an API it is good to go.

Well, you may be wondering at this point, what is the point of all this rambling about governance and policy. The issues with data governance are complex, and we have made them needlessly complicated because not enough attention is paid to architecture and how that simplifies governance. We are in a response cycle endlessly, needlessly, businesses buy tooling to fill white spaces left by the absence of architecture, and that results in too much governance that is not easily understood or adhered to; therefore, it is relegated to the afterthought pile. This workflow results in more response; it is a vicious cycle. Quite simply the absence of data architecture at the business level has led to horrible inefficiencies and the need to apply data governance from the rear. You would never build a house and have the framing inspected after the drywall was up, but we do this with business systems all the time.

The lack of data architecture is a lack of action that creates white space. I am convinced that the lack of action is an act in and of itself. That white space is filled with tooling, which is then followed by more tooling. The result of that white space and tooling requires a response as the sprawl spirals out of control. The perceived way to regain control is to apply data governance not data architecture. The missing piece within the enterprise is not data governance, but data architecture. Apply good data architecture principles and good data governance will come as a result, they are two sides of the same coin, a partnership. It is more work up front, and the work is necessary and difficult, but will result in few tools, and fewer battles for control after the fact.