Improving forecast reliability - Use case

Forecasting is often seen as a solved problem until it needs to work in production. Many organizations struggle to turn promising models into reliable, actionable tools due to challenges in data quality, model transparency, and scalability. At Notilyze, we help organizations bridge the gap between experimentation and production, turning forecasting into a robust, trusted decision-making capability.

Problem description 

Companies who want to bring forecasting models into production, see the following challenges: 

• Accuracy forecast too low to create actionable outcomes; this can be caused by a lack of statistical knowledge resulting in suboptimal model choices, using the wrong information from data, using low-quality input data,  

• Moving from pilot to production: including data in the pilot that is not available when deploying the model, using models that cannot be explained to the stakeholders (management board) and are therefore not transparent enough to be supported by the board, or using open source tools that are labour-intensive to maintain and keep track of; updates of python packages that break dependencies within the project, vulnerabilities in code that the company is not maintaining themselves, etc. 

Specifically for a drinking water company with smart metering (IoT) in their drinking water system, their developed model was not good enough to be brought into production. Also data cleaning was not structurally done and the model was built with open source packages that rely on maintenance by the open-source community. This can lead to unwanted situations after deploying code (vulnerabilities that can be exploited by malware and not be solved internally).  

Solution 

Notilyze has a Data Analyst team consisting of MSc-level members in Econometrics, Applied Mathematics and AI. With the knowledge of forecasting models and their statistical foundations, proper model choices can be made. By leveraging SAS Viya a large set of potential model candidates can be explored and deployed, all within the same platform and without dependencies on code of someone else. Also the full analytics life cycle (development, ETL pipeline and connections to databases, deployment and continuous evaluation) is supported, which places the ownership of the entire data and analytics pipeline with the organization; where it belongs. 

Specifically for a drinking water company with smart metering (IoT) in their drinking water system, the team was able to decrease the errors with a factor 3 by improving both the model choice and by improving and standardizing the data cleaning. A MAPE (Mean Average Prediction Error) of 7.5% was obtained (both in training and test set), resulting in a model that could predict the population within tourist cities by leveraging the IoT data, combined with weather data and event calendars, in a transparent and explainable model. After pushing the accuracy with the initial data to the max in a great collaboration with the customer, the customer also understood the value of good input data and managed to improve the model on their side as well by obtaining better input data for model training. 

How? 

The collaboration with the customer is centered around delivering Analytics as a Service; with our Cloud, Analytics and IT team Notilyze fills the knowledge gaps at the customer end. On the other hand, we collaborate with the experts within the organization; for example their subject matter knowledge, database administrators or data engineers. In sprints we determine the focus and business needs with the customer, estimate the workload and develop the ETL pipeline and algorithm, as well as documentation and visualisation of the results. After evaluation on the test data set, the model can be deployed on a server hosted by the customer or a SAS Viya server hosted by our Cloud team in the Netherlands, ensuring ISO27001 compliance and data not leaving the EU. 

In the case of a drinking water company with smart metering (IoT), we distilled the business needs from the desired forecasting problem. After creating an ETL pipeline and forecasting algorithm with SAS Viya, we connected their Snowflake database and created APIs to integrate the model results in the monitoring software of the customer.  

Challenge 

The main challenge is to obtain a transparent, accurate model. Most of the times more complex models give better forecasting accuracy, introducing a trade-off between accuracy and interpretability. The former is important to create actionable outcomes, while the latter is important to create trustworthy results. By creating insights in both aspects we managed to reassure the customer the outcomes can also be trusted by using future data as input that was not tested yet. 

About Notilyze

We are Notilyze, a SAS® Partner with only one goal: to help organizations with data analytics, and become more data-driven and successful. Our aim is to bring data analytics within reach of every department and every company. We do this by offering SAS® as a Service. Notilyze Cloud is the quickest solution to start with AI on SAS® Viya®. In addition, we offer IT solutions to properly integrate analytics into existing processes, and the experience of our SAS® specialists.