NHG guidelines and Huisarts & Wetenschap: one technology, two entirely different challenges

In practice they turned out to be two entirely different projects.

"The approach is different. No two RAG systems are alike. You run into challenges specific to each product. Doing two projects like this at the same time really shows you it isn't a matter of switching it on and being done."

The difference is in the stakes. With the NHG guidelines, an AI application quickly touches on medical practice; the user, a GP, uses it to check their own clinical decisions against the profession's guidelines. That changes the context. With Huisarts & Wetenschap, the AI is there to make the journal's content easier to search.

Within Europe the Medical Device Regulation (MDR) applies. That means software influencing medical decisions can be classed as a medical device, which brings strict requirements around reliability, verifiability and risk.

That has a direct effect on how you develop the solution.

Two applications, two kinds of use

Although both solutions are about opening up knowledge, the use is completely different.

Huisarts & Wetenschap is a large archive of research and practical articles, and opinion pieces too. Valuable, but not easy to search. GPs don't search by title or author. They have a question about recent medical research and want an answer fast, with the option to click through to the source.

The NHG guidelines form the basis for medical practice. There are around 150 guidelines that together determine how a GP acts. Those documents are long, and the information is spread across several guidelines.

So with Huisarts & Wetenschap you help people search and discover. The NHG guidelines give recommendations for clinical practice.

And that is exactly where the difference starts to matter.

A different approach because the stakes differ

With Huisarts & Wetenschap the approach was relatively straightforward. The tool could be developed quickly, tested with editors and then put live. The impact of mistakes is limited. A weaker answer doesn't directly change how a doctor treats a patient.

With the NHG guidelines it is a different story. You can't simply go live there. You have to be able to explain where an answer comes from and how it was arrived at. In practice that meant:

• clear source references
• always a link to the original guideline
• no interpretation by the tool, only by the GP

The AI supports; it doesn't take the decision out of the doctor's hands. That boundary had to be clear.

Scale and coherence make it complex

The NHG guidelines aren't only large, they are interconnected. The value lies precisely in combining several guidelines.

A question about the NHG standard on COPD also touches on the NHG treatment guideline for quitting smoking, for example. You have to be able to make those connections. At the same time, a language model can only process a limited amount of text at once. So you can't simply feed it everything in one go.

The solution lay in splitting questions up and retrieving relevant information in a targeted way. That lets you combine several guidelines without losing sight of the whole.

That sounds technical, but it is mostly about control. You want to know exactly which information is used when.

Testing from different perspectives

The NHG guidelines were tested step by step. First an internal group of GPs, then increasingly large external groups of GPs.

Each group looked at it differently:

• experts watched for completeness and nuance
• GPs also wanted speed and applicability

That made clear that reliability isn't only about the content. It is about how the tool gets used.

With Huisarts & Wetenschap it comes down to interpretation

With Huisarts & Wetenschap the challenge lies elsewhere.

The archive is large and varied. Not every article carries the same weight in the same context. An opinion piece is something different from a research article. How current something is plays a part too. You don't want to present an old article as a current answer.

So they chose:

• more emphasis on recent content in AI answers
• a distinction between different kinds of articles
• offering AI answers and search results separately

Users can choose how they search. That turned out to matter, because not everyone wants to use AI the same way.

Monitoring comes with the territory

Both projects showed that the work doesn't stop at launch. AI opens a new channel where users ask questions. That gives insight into what works and what doesn't.

So inside the NHG they keep looking at questions, patterns and situations where answers fall short, and adjust the solution accordingly.

That calls for a different way of working. You don't build it and hand it over, you keep improving it.

What this case shows

This case shows that AI in healthcare is not an off-the-shelf solution. The same idea can turn out completely differently depending on the context.

With the NHG guidelines the emphasis is on care and control, because you sit close to medical practice. With Huisarts & Wetenschap the emphasis is on accessibility and interpretation.

And that is exactly where the link with the MDR comes in. The closer you sit to decision-making, the stricter the requirements become and the more carefully you have to work. That makes projects like these more complex. And more interesting at the same time.