Using Data Mesh to Organise Data Management

We recently visited Sander Kerstens to talk about his Data Mesh implementation at Vanderlande. Data Mesh is a new approach to organising enterprise data. It aims to make managing and using data easier for everyone involved. 

Traditionally speaking, data management is organised through a centralised team that is responsible for all enterprise data. Data Mesh decentralises this, by distributing this responsibility across smaller teams (called domains) within the enterprise. Instead of having central policy and standards that are applied across enterprise teams, teams define their own policies and standards instead. 

In Data Mesh, each domain is responsible for the data they generate, the domain decides how their data is managed, processed and shared with other domains. All domains work together in a networked architecture. In turn, allowing for greater collaboration and ability.

A core principle behind the Data Mesh philosophy is one that we often write about: treating data as a product. As a product, there should be clear documentation and standards that describe that data is used and maintained. Much like in traditional data management, Data Mesh stresses the importance of good metadata.

By empowering smaller teams to take ownership of their data and work more closely with other domains, Data Mesh can help organizations to scale and innovate more quickly and efficiently. It bases data management hygiene factors on its principles, rather than having a central data governance team dictate how teams should act. 

This introduces a different way of thinking, which may be more suited to modern enterprises. This depends on the culture of the enterprise, though. One approach is not necessarily better than the other, both have their own strengths and weaknesses which are outlined below. 

Traditional Data Management

ProsCons
Greater control and consistency Potentially slow and inflexible 
Close alignment with business strategyMay not need team/domain specific requirements
Close alignment with regulatory requirementsMay not need team/domain specific requirements

Data Mesh

ProsCons
Agile and responsive to changing business needsCan help foster innovation and collaboration between teams
May present challenges around data quality and consistencyComplex to implement in terms of culture and technical debt

Leading through ownership of personal data, this is what you should know!

#innovation #data

Enabling or blocking? Sovereignty of personal data.

Within the digital world, individuals are mostly viewed as — potential — consumers (obviously already a high share) or patients (currently growing share). The data of individuals needs to comply to the regulations within the country or region where the data is collected, i.e., it needs to fit with privacy and security.

Companies are building views on individuals, based from the name, address, email etc, which have been provided through every registration to an online service. As well as online behaviour, e.g., through tracking cookies. These centralised views or centralised identities are stored within silo-based platforms. Neither personal data or individual behaviour are well portable. This means that your digital identity exists in many small pieces with several companies knowing different information about you. This also means that you have to create a unique password for every profile you make, which can be cumbersome, and many tend to use the same password more than once. All of this creates security risks, since your personal data is being stored and managed by many entities and because a password breach might give access to several of your accounts.

An attempt to address these issues is federated identities. Individual identities are managed in a company or government centralized system. The system then distributes the data from the individual to a digital service. Examples where this is in use is within banks, insurers, retail and health. A federated identity enables easier digital activities through a single-sign-on solution However, a federated identity is still silo-based, since it only can be used with web services that accept this solution.

“………That’s right, SSI sets data ownership at the individual level.”

A next generation of identity solutions that is currently being developed and taken into use is self-sovereign identities (SSI). This type of digital identity is a user-centric identity solution that allows you to be in control of your data and only share the strictly relevant information. An example would a situation where you need to prove that you are of age. With an SSI you can document that you are over 18, without disclosing your exact age. Or documenting that you have received a specific vaccine, without disclosing information about all the vaccines you have ever gotten or other sensitive health data. Other examples are sharing that you have graduated to your — future — employer, your medical record with a hospital and your bank account with a store. In your own personal vault if you like (also: a ‘holder’ or ‘wallet’), you own and manage your data. That’s right, SSI sets data ownership at the individual level. Data ownership would resolve a large topic, that often proofs to be a blocker for companies to fulfill their digital ambitions. From this vault you decide to which companies & organisation you want to share your personal data to be defined per specific purpose. For this purpose, personal data needs to be classified (e.g., in accordance with privacy & security regulations) which data is open for all, which is private and which is secure data. The vault provider needs to have good technical solutions (e.g., with verifiers and encryption), a sufficient governance regime and controls in place to support this.

SSI will mean that individuals need to understand what ownership comprises of, what potential risks are and what good practices are to share data. Data literacy should be extended from mostly companies to more individuals. And companies should prevent technical, legal, ethical, fairness and security pitfalls (see also: 10 principles for SSI), e.g, for transparency for systems & algorithms as well as data monetization.