D8A directors

31/05/202331/05/2023

How to make data governance practical using datasets

Nowadays, even when you have a catalog implemented that scans your data, the number of search results on the word ‘customer’ is too high and far from relevant. Scrolling through the first page of results makes you wonder what would suffice your data needs. Is the data properly described, do you know who to go to for questions and are you even allowed to use privacy sensitive data for your use case?

Documenting and maintaining these basic attributes of your data is already challenging and time consuming in itself. And at what granularity do you need to do this? If you listen to legal and compliance; every single attribute of every table of every system must be properly protected. This seems like an impossible task. And it is!

How can you practically approach this?

Start organising your data into datasets!

Classifying and defining all your data attributes is a cumbersome and time consuming task. At some point you should do it for at least the data you share. But waiting for all this to be done is not an option. Business must go on. So how can we overcome the big pile of work in the mean time? The answer is to start organising your data into datasets.

What do we mean by dataset?

There are varying opinions of what a dataset may be. At D8A Directors we consider a dataset as a collection of physical data objects that needs to be managed (governed) together. This can consist of a single or a selection of tables that make up the customer master data or the collection of tables that hold your inbound shipments. It may also be used for unstructured data like video’s or images that were collected during a particular event.

Then governance is applied at all the data together in these datasets. When you combine data into datasets that needs to be physically managed together; you don’t need to manage every individual attribute. The question then is what are these governance requirements that you should think about to combine data?

Dimensions to seperate data into datasets

To be honest, the dimensions to take into account need to be determined in your organisation by the Data Governance department in their policies. The legal grounds that allow you to process data vary per organisation, so there’s no one single set of dimensions. However, to get you started, here’s an example of good practice dimensions:

Source system
Data product
Geographical Region due to regional legislation
Responsible data owner (single ownership for accountability)
Purpose limitations
Retention term
Business & privacy sensitivity classification

The goal of a dataset is to be able to group physical data for purposes of:

Logically organizing data
Data Ownership and Data Governance
Request and approval against purpose limitations
Granting and Managing Security / Data Access Rights
Managing Data Retention and Disposal
Classifying Data for Privacy
Classifying Data for Geographical Regions
To make Reuse of data easier
Archiving Data to cheaper Storage tiers

Let’s make it practical

Decomposing datasets distinguishes conceptual data models, data objects and the attributes (from a logical model). These should then be linked up to the actual physical data.

An example is shown below on how data privacy classification of attributes affect datasets. Classsifying all attributes is a lot of work while you can manage your data already to a sufficient extent when this is classified at dataset level:

The dataset is an abstraction at a logical level that makes it easier to govern a collection of physical data. With the right information at dataset level like business owner, purpose limitation, business sensitivity, privacy sensitivity, retention term etc, you can start managing access to this data.

Note that the dataset is a logical grouping of data. This does not necessarily mean you must physically seperate different datasets to comply with governance requirements. An example when you must physically seperate the data is in case of a regional requirement to store data within country borders.

Data products & datasets

Taking it one step further: how do datasets relate to data products?

As per the HR example, you can consider this to be a single data product. A data product can expose access to different datasets that the data product holds.

To optimize the storage, you could expose dataset as (virtual) views on the actual data.

22/05/202322/05/2023

Do you want to discover a new data revolution?

Data service organisation - to enable business requirements

Central for de-central data teams: start acting as a (temporary) data owner to service business requirements.

Set up data service desks to be flexible for business requirements.

Data is often seen as the raw material to produce new products, frequently with analytics and AI as the innovative machinery enabling the end-result. Recent years have proven that the value of data more often serves as an enabler of multiple business results, leading to efficiency savings, profits, and the ability to maintain existing markets while expanding into new ones

Data as fundamental oil

Whether it is automated payments and invoicing, online customer interactions, or digital manufacturing, data is the underlying oil that can make your business operations run smoothly.

Or does it? Why is it that, despite the abundance of data, businesses often run sub-optimally, sometimes even relying on manual activities, in this digital age?

“So, if data is the new oil,

why isn’t everything running smoothly?”

All companies generate, receive, and process data to some extent. Data is abundant these days. So, if data is the new oil, why isn’t everything running smoothly?

Here is why: the data itself is complex, and the usage of data is complex. Many companies have tried to resolve this combined complexity through centralized standardization. Many projects aimed at establishing a single data model have become famous, often leading to disappointments. Alternatively, data solutions seek refuge in technology, often resulting in an increase in applications, which can add to the complexities instead of alleviating them. Above all, centralized standardization requires control, which does not adequately serve the business.

Move from control to empower

The very essence of any business is flexibility, the ability to innovate and develop new products and markets. The business needs to be facilitated by data. So, move from controlling towards empowering.

Empowering means understanding that there is no one-size-fits-all when it comes to data, as in the above-mentioned complexities. Data is very similar to machinery. Just compare the oil for ball bearings to petrol. They share the same raw material but differ in volume, characteristics, substance, and processing for different purposes.

How do we see the solution?

With the extensive rise in data volume, complexity, and velocity, a central data team supported by data stewards and architects is no longer sufficient. It requires more decentralized teams that can facilitate specific business needs while adhering to central requirements. Use the motto: control only where needed – for example, using one standard product or client ID across systems, and facilitate where possible, such as adding an additional product ID to support a regional process. Of course, this requires more effort. It is evident that any additional data requires more maintenance. However, the benefits for the business are immediate and significant. There is no need for the business to change processes, systems, or reporting. Immediate possibilities emerge to make more local variations of products and insights, facilitating specific market requirements. This approach maintains the possibility of working with central initiatives and the option to upgrade or downscale data where possible without affecting central requirements.

“The answer is easy, the deployment is more complex “

How do you facilitate this? The answer is easy; the deployment is much more complex. Have dedicated data teams in place with a close relationship to the business. That data team should consist of senior, well-trained data management experts, data analysts, and data engineers to facilitate and guide the local solutions, including the link to central platforms. The team should be able to answer business questions through a so-called service desk. Such a service desk requires a thorough understanding of the business processes and systems and the translation of data requirements into existing (or missing) data within systems. Preferably, the service desk should have the capability to identify, flag, and resolve regulatory questions on privacy, financial legislation, and health legislation. Make sure that the service desk is enabled by a ticketing workflow, including a dashboard displaying their effort and impact. Finally, that data team should be able to guide the business stakeholders in the best approaches and solutions. Don’t expect business stakeholders to deliver data requirements; they will have business requirements. If you didn’t know better, this team almost acts like a data owner.

Of course, some data needs to be strictly governed and controlled. There are overarching business requirements (e.g., insights in sales volumes) that require consistency and quality of trusted data. Identify these key data elements and manage them with a strict and tight regime. These key data elements can. for example, be linked to key reporting, identified as being used for most processes, or be the primary key within multiple systems. Current data volumes can make this identification a tough job. A good start can be using the Dublin Core standard to identify the right regime. The standard uses the following guidance:
– which data is related to which process, system, product and report?
– where is data being used?
– why is the data needed (purpose)?
– Who uses the data?
– How is data labelled and referenced?
– What is the relevance of the data (e.g., static or dynamic)?
– How is data related to other data?

New way

Teams acting as (temporary) data owners is a new, almost revolutionary way of looking at data. The traditional view, based on data standards (e.g., DAMA, DCAM, ISO), all revolves around governance and ownership. That view is based on having data ownership within the business. If you step away from that theoretical view and fall back on lessons learned, then don’t expect business stakeholders to take up sufficient data ownership. For decades, they have perceived data as a by-product.

Most business stakeholders will stay away from data ownership simply because it is unknown territory for them. It is up to the data team to translate do’s and don’ts regarding data and take up data ownership for the business. In theory, this might even support the embracing of data ownership by business stakeholders through the principle of show and tell.

So, a team of experts is required. Such a team goes beyond the effort of some companies to “simply” assign a data steward who reports on the content of specific fields within systems. Companies should build dedicated teams across the organization, which will often need to invent the wheel themselves. The way of working will differ per objective. For project goals, make sure you can act fast, agile, and dedicated. For sustainable solutions, ensure that you stay completely aligned with company standards (and enhance a few where needed) to avoid the “not invented here syndrome.” For any purpose, make sure you take the time to understand and align data, data requirements, and business requirements. And actually build solutions – not just on paper, but within apps, databases, data pipelines, and systems.

All of this will require a solid, robust, and senior data leadership team which can manage, sustain, guide and facilitate data responsibilities. Invest in that team.

For examples on data standards, visit: DAMA, DCAM or ISO

Contact D8A to get started

16/05/202316/05/2023

Using Data Mesh to Organise Data Management

We recently visited Sander Kerstens to talk about his Data Mesh implementation at Vanderlande. Data Mesh is a new approach to organising enterprise data. It aims to make managing and using data easier for everyone involved.

Traditionally speaking, data management is organised through a centralised team that is responsible for all enterprise data. Data Mesh decentralises this, by distributing this responsibility across smaller teams (called domains) within the enterprise. Instead of having central policy and standards that are applied across enterprise teams, teams define their own policies and standards instead.

In Data Mesh, each domain is responsible for the data they generate, the domain decides how their data is managed, processed and shared with other domains. All domains work together in a networked architecture. In turn, allowing for greater collaboration and ability.

A core principle behind the Data Mesh philosophy is one that we often write about: treating data as a product. As a product, there should be clear documentation and standards that describe that data is used and maintained. Much like in traditional data management, Data Mesh stresses the importance of good metadata.

By empowering smaller teams to take ownership of their data and work more closely with other domains, Data Mesh can help organizations to scale and innovate more quickly and efficiently. It bases data management hygiene factors on its principles, rather than having a central data governance team dictate how teams should act.

This introduces a different way of thinking, which may be more suited to modern enterprises. This depends on the culture of the enterprise, though. One approach is not necessarily better than the other, both have their own strengths and weaknesses which are outlined below.

Traditional Data Management

Pros	Cons
Greater control and consistency	Potentially slow and inflexible
Close alignment with business strategy	May not need team/domain specific requirements
Close alignment with regulatory requirements	May not need team/domain specific requirements

Data Mesh

Pros	Cons
Agile and responsive to changing business needs	Can help foster innovation and collaboration between teams
May present challenges around data quality and consistency	Complex to implement in terms of culture and technical debt

Material Science with Materials Zone

Ori Yudilevich (Chief Technology Officer at Materials Zone) on: the history of Materials Zone, the company and its product. Ori explains how Materials Zone’s Materials Informatics platform applies material science techniques to save costs. He also explains what challenges with regards to data they usually come across.

D8A Directors

Material Science with Materials Zone

00:00 / 25:49

Trends and Developments in Data Analytics Consulting

Ryan Price (Executive Data & Artificial Intelligence at Avanade) on: trends and developments within data analytics consulting. Ryan talks about how Avenade approached data and AI on a global scales and discusses his views regarding important topics around data and AI.

D8A Directors

Trends and Developments in Data Analytics Consulting

00:00 / 19:47

30/09/202230/09/2022

Tailor made D8A Academy trainings

Members and partners of D8A Directors can bring their years of experience as hands-on training to your organisation.

Are you looking for inspiration, guidance and practical how-to’s in building your data driven organisation?

Some inspiration on topics:

Guiding change through enterprise data architecture practice
How metadata management enables you to find, understand, govern, trust and share your data
Growing a federated data governance in your organisation made practical
Where and how to start with data quality management with immediate cost reduction in business operations
How to transition the perspectives of data security, privacy & compliance from ‘cost center’ to ‘profit center’ when done right
How to organise data products in data mesh
How to realize data observability embedded into your data pipelines with databricks and deltalake
and other

Don’t hesitate to get in touch with your case for a trailor made training!

Data-driven asset management

Jan Leenaars(Product Owner at Rijkswaterstaat) on: data-driven asset management, how data can be leveraged to manage assets, common pitfalls and lessons learned. Jan talks about how users & developers can work effectively together to create reliable insights on time. Listen to hear how Rijkswaterstaat has faced data-driven asset management challenges and how they overcome these challenges.

Date with D8A

Data-driven asset management

00:00 / 18.43

How to Become a Successful Data Professional

Remi Verhoeven (Senior Manager at KPMG) on: the character traits, technical skills and soft skills required in order to become a successful data profession. Listen to hear about the ways data professionals can improve their skills to increase their chances in the recruitment process.

Date with D8A

How to Become a Successful Data Professional

00:00 / 33:31

15/03/202228/11/2022

Launching synthetic data within your company? Understand results and possibilities!

The concept of synthetic data generation is the following; taking an original dataset which is based on actual events. And create a new, artificial dataset with similar statistical properties from that original dataset. These similar properties allow for the same statistical conclusions if the original dataset would have been used.

Generating synthetic data increases the amount of data by adding slightly modified copies of already existing data or newly created synthetic data from existing data. It creates new and representative data that can be processed into output that plausibly could have been drawn from the original dataset.

Synthetic data is created through the use of generative models. This is unsupervised machine learning based on automatically discovering and learning of regularities / patterns of the original data.

Large volume synthetic data as twin of original data

Why is synthetic data important now?

With the rise of Artificial Intelligence (AI) and Machine Learning, the need for large and rich (test & training) data sets increases rapidly. This is because AI and Machine Learning are trained with an incredible amount of data which is often difficult to obtain or generate without synthethic data. Large datasets are in most sectors not yet available at scale, think about health data , autonomous vehicle sensor data, image recognition data and financial services data. By generating synthetic data, more and more data will become available. At the same time, consistency and availability of large data sets are a solid foundation of a mature Development/Test/Acceptance/Production (DTAP) process, which is becoming a standard approach for data products & outputs.

Existing initiatives on federated AI (where data availability is increased by maintaining the data within the source, the AI model is sent to the source to perform the AI algorithms there) have proven to be complex due to differences between (the quality) of these data sources. In other words, data synthetization achieves more reliability and consistency than federated AI.

An additional benefit of generating synthetic data is compliance to privacy legislations. Synthesized data is less (but not zero) easy directly referable to an identified or identifiable person. This increases opportunities to use data, enabling data transfers to cross-borders cloud servers, extend data sharing with trusted 3^rd parties and selling data to customers & partners.

Relevant considerations

Privacy

Synthetisation increases data privacy but is not an assurance for privacy regulations.

A good synthethisation solution will:

include multiple data transformation techniques (e.g., data aggregation);
remove potential sensitive data;
include ‘noise’ (randomization to datasets);
perform manual stress-testing.

Companies must realize that even with these techniques, additional measures such as anonymization can still be relevant.

Outliers

Outliers may be missing: Synthetic data mimics the real-world data, it is not an exact replica of it. So, synthetic data may not over some outliers that the original data has. Yet, outliers are important for training & test data.

Quality

Quality of synthetic data depends on the quality of the data source. This should be taken into account when working with synthetic data.

Black-box

Although data synthetization is taking center stage in current hype cycles, for most companies it is still in the pioneering phase. This means that at this stage the full effect of unsupervised data generation is unclear. In other words, it is data generated by machine learning for machine learning. A potential double black box. Companies need to build evaluation systems for quality of synthetic datasets. As use of synthetic data methods increases, assessment of quality of their output will be required. A trusted synthetization solution must always include good information on the origin of the set, its potential purposes for usage, its requirements for usage, a data quality indication, a data diversity indication, a description on (potential) bias and risk descriptions including mitigating measures based on a risk evaluation framework.

Synthetic data is a new phenomenon for most digital companies. Understanding the potential and risk will allow you to keep up with the latest development and ahead of your competition or even your clients!

25/01/202230/03/2023

Do you know the physical location of your data?

“Where is your data stored?” is not asked often enough. An interesting topic, brought up by an owner of a Dutch cloud company during a radio program.

It surprised me. As a Data Professional who has lost count of the number of times I have asked this question. Why is this still not a common concern?

When I ask the question, the reactions vary. Often times the answer is a simple “I don’t know” When I do find the right specialist to tell me these details, the conversation goes something like this:

“Where is the data stored?”
“In the cloud.“
“Yes, but where are those servers located?”
“In Europe”
“Yes, what country in Europe?”
“In The Netherlands. “
“Yes, do you know in which city?”
“Yes, in Amsterdam.”

The tedious process goes to show that it’s a question that is not asked often enough. But it matters.

It’s interesting, it seems we don’t often enough realise that although data is digital, it always has a physical aspect to it. Much like us, the data has to live somewhere. In turn, cyber security is not just about the digital security of who can access your data, but also the physical one. What physical measures are in place to ensure that no one breaches your data centres? Do you know who can access the building where your data lives? However, security is but one concern when it comes to the physical location of data.

Laws and regulations mandate where data may be stored and processed. And processing includes ‘viewing’ data. GDPR, for instance, requires that data is stored in Europe. That means that some international cloud providers are automatically a non-option, when they don’t have data centres in Europe. If your data centres are located in a different country, you’re automatically dealing with cross-country transfers. Be vocal about this towards your cloud provider. If you don’t decide where your data is physically stored, they will. Then it’s out of your control. And most likely not in line with legal requirements.

Did you know that there are countries that demand their data is stored and processed locally only?

Location may also influence the reliability of you data. If your data centres are located in an area where power outages are common, you will be dealing with limited availability. Or if it is a politically unstable region, your data centre may be put out of the running all together. Next to that, the further away the physical location, the higher the risk for limited connectivity. Both the quality of the network and the distance can greatly impact connectivity. All these aspects can influence your ability to deliver the data-driven digital products and services your customers are paying for. In other words, thinking about the location of your data is business critical.

So my takeaway for you, start asking the question: do you know where your data is stored?

Data strategy for small and medium-sized enterprises

Geneviève Meerburg (Director SME Services at van Spaendonck) on: implementing a data strategy within her organisation. Geneviève shares how the importance and value of data organically grew, leading to a concrete need for a data strategy. Listen to hear how van Spaendonck approached truly living through the principles set out in the data strategy and how it helped create new services for their clients.

Date with D8A

Data strategy for small and medium-sized enterprises

00:00 / 26:27

23/11/202124/11/2021

Take the confusion out of your metadata discussions

Business metadata

All good data starts with business metadata. This business metadata is the information we need to build a dataset. There is someone in the business who approved the collection and processing of data in the first place. He/she also provides requirements and descriptions on what is needed. The challenge is that this information is often not managed good enough throughout time which leads business metadata quality to decrease. And thereby decrease good data. And that affects your AI solutions, BI reporting and integration with platforms.

Become aware of the necessity and value of business metadata to enable support on data requests, make it findable and also understandable!

When we know what business stakeholders want, we can design and implement this into physical form through technical metadata. We can now build the solution or buy it of the shelf and map it to the business metadata.

Operational metadata

Now that we know what data we need, what it means and have a place to store and process data; we can start doing business. Doing business will generate operational metadata. Operational metadata is very valuable in monitoring our data processes. We get insights in what data is processed, how often, the speed and frequency. This is great input in analysing the performance of our IT landscape and see where improvements can be made. Further we monitor the access to systems and data. When we take it a step further we can even start analysing patterns and possibly spot odd behaviour as signals of threats to our data.

Step into the driving seat capturing and analysing your operational metadata and become pro-active in controlling your IT landscape!

Social Media metadata

Finally we take the social metadata as an inspiration. This is where the value of your data becomes even more tangible. Value is determined by the benefit the user experiences. The way that he uses the data is then an indicator of value. Thus if we start measuring what data is used often by many users, this data must be important and valuable. Invest in improving the quality of thatdata to improve the value created. Behaviour is also a good indicator to measure. How much time is spent on content and which content is skipped quickly. Apparently that content doesn’t match up with what the user is looking for.

Measure social metadata to analyse what data is used often by many. It is likely to be more valuable than other data.

Business metadata

Governance metadata
All metadata required to correctly control the data like retention, purpose, classifications and responsibilities.
– Data ownership & responsibilities
– Data retention
– Data sensitivity classifications
– Purpose limitations

Descriptive metadata
All metadata that helps understand and use and find the data.
– Business terms, data descriptions, definitions and business tags
– Data quality and descriptions of (incidental) events to the data
– Business data models & bus. lineage

Administrative metadata
All metadata that allows for tracking authorisations on data.
– Metadata versioning & creation
– Access requests, approval & permissions

Technical metadata

Structural metadata
All metadata that relates to the structure of the data itself required to properly process it.
– Data types
– Schemas
– Data Models
– Design lineage

Preservation metadata
All metadata that is required for assurance of the storage & integrity of the data.
– Data storage characteristics
– Technical environment

Connectivity metadata
All metadata that is necessary for exchanging data like API’s and Topics.
– Configurations & system names
– Data scheduling

Operational metadata

Execution metadata
All metadata generated and captured in execution of data processes.
– Data process statistics (record counts, start & end times, error logs, functions applied)
– Runtime lineage & ETL/ actions on data

Monitoring metadata
All metadata that keeps track of the data processing performance & reliability.
– Data processing runtime, performance & exceptions
– storage usage

Controlling (logging) metadata
All metadata required for security monitoring & proof of operational compliance.
– Data access & frequency, audit logs
– Irregular access patterns

Social metadata

User metadata
All metadata generated by users of data to
– User provided content
– User tags (groups)
– Ratings & reviews

Behavior metadata
All metadata that can be derived from observation to
– Time content viewed
– Number of users/ views/ likes/ shares

Don’t mention AI!

Bart Rentenaar (Enterprise Data Lead at Athora) on: implementing data innovation within his organisation. Bart shares examples of use cases that inspired him to get start with data innovations, the framework that employs to structure initiatives, examples of data innovations he implemented and the team that made that possible. Listen for tips when starting out with the implementation of data innovations.

Date with D8A

Don't mention AI!

00:00 / 39:53

Trusted data awareness

Arjan Pepping (Corporate Data Manager at MN) on: creating awareness around trusted data and the role of data in control for a pension provider. Listen for the golden tip on implementing data awareness.

Date with D8A

Trusted data awareness

00:00 / 14:59

Good design teams embrace data quality

Marinka Voorhout (Director at Philips) on: data quality in design is becoming a pre requisite for innovations on data. Listen to practical approach tips and ideas to take data quality into account in user interfaces.

Date with D8A

Good design teams embrace data quality

00:00 / 24:34

Trailer: Date with D8A

In this trailer of the Date with D8A podcast series, Simone from D8A explains the idea behind the D8A initiative and the what, why and how of the Date with D8A podcast.

Date with D8A

Trailer: Date with D8A

00:00 / 4:13

21/09/202130/03/2023

Owning the use of data

Why your company needs a Chief Data Officer.

It is time to increase acknowledgement of the importance of a chief data officer.

As companies move towards working data-driven, monetizing data in new and enhanced services and products is essential. Traditionally heavy regulated industries, e.g. financial and health, first focused on bringing their data in control. Their efforts concentrated mainly around data quality management, data privacy, data governance and E2E trusted data lineage. These efforts are often led — or owned — by a Chief Data Officer (CDO).

In this article, we advocate to shift or extend this focus of the chief data officer towards data in control AND data in use.

CDO’s define and communicate the companies vision on data management and data use. Through this vision, the CDO gives direction, guidance, advocates for change and sets priorities for running projects. Most companies CDO’s have to some extent achieved this for data management. The extended focus of Chief Data Officers, which we advocate for in this article, contains standard processes for the design, prototyping, development, productizing and use of data & insights products & services. Furthermore, it is the CDO who defines a standard set of technology to be used to support these processes and create these solutions. Where needed, this is based on the data management foundation as implemented by the CDO in previous years.

The Chief Data Officer ideally combines business expertise, technology background and analytics/BI. Extended by a common commercial sense, understanding of production processes and knowledge of relevant 3rd party partners to cooperate with. Organisations without an ‘extended CDO’ will experience difficulties and potential delays in reaching their data-driven goals — in accordance with new developments in the market. Without strategic guidance and steering, there is an increased risk that departments and units will define their own standard processes, set of technology and data-driven products and services. Making it harder to leverage pre-existing data foundations as well as cross-unit collaboration to enable effective market penetrations. Teams will struggle to escalate and address growing concerns as sufficient C-level representation is missing.

Concluding, companies benefit from a Chief Data Officer with a focus on data in control and data in use. Top-down ownership and alignment of data initiatives, standardisation of processes and data tooling and a clear escalation path for growing concerns are necessary to succeed as a data-driven company.