// PoC: A New Data pipeline for ‘Zorginstituut Nederland’
Zorginstituut Nederland plays a crucial role in the Dutch healthcare system by ensuring healthcare remains accessible, effective and affordable. The organization relies heavily on data for decision-making, making proper data governance essential. To improve their data management capabilities, they approached us to develop a proof of concept for a new data pipeline within the structure of a modern data platform.
The Challenge
Zorginstituut Nederland asked us to create a data pipeline for one of their data streams, based on SQL, in a reusable way. A key requirement was that the state of the data warehouse at any given moment in history should be retrievable. This would allow them to reproduce past analyses and ensure that historical data remained accessible and accurate over time.
Additionally, the developed software needed to meet several key requirements:
- Efficient Data Ingestion – The ability to seamlessly integrate diverse data sources.
- Data Quality & Cleaning – Ensuring the data is reliable and ready for analysis.
- Optimized Database Performance – Structured for efficient querying and reporting.
- Traceability – The ability to track the origin of each data point.
Zorginstituut Nederland also requested research into different platform options, specifically comparing Microsoft and SAS technologies for data processing and ETL workflows.
Our Solution
We developed a proof of concept using Microsoft SQL Server, focusing on a structured approach with three key phases:
- Staging Layer – A landing zone where data is ingested, cleaned, and validated. In this layer, the raw data is stored to allow traceability of the data.
- Data Warehouse – Built following the Kimball methodology, ensuring a structured, efficient, and scalable foundation.
- Data Marts – Enabling tailored reporting and analysis for different business needs.
To populate the Staging Layer, we created two separate ETL pipelines (Extract, Transform, Load):
- Microsoft Azure Data Factory – A modern, cloud-based approach to ETL.
- SAS Viya with SAS Flows – Providing additional flexibility and integration options.
As part of our research, we conducted a comparison between Microsoft and SAS solutions, evaluating their performance, usability, and scalability. This provided Zorginstituut Nederland with valuable insights to inform their long-term technology strategy.
With this proof of concept, Zorginstituut Nederland now has a solid foundation to develop a scalable and well-governed data platform. Our approach ensures their data is clean, traceable, and optimized for analysis – empowering them to make data-driven decisions with confidence.
Interested? Let us help YOU become a data driven organisation. Contact us today to discover the possibilities!
Contact:
Eric Mathura
E-mail: eric.mathura@notilyze.com
Mobile: +31 6 53640514
