Why your AI assistant gives outdated answers

When the AI assistant on your website gives an outdated answer, the first thought is often that the model is no good. Usually it's something else. An assistant like that answers from your own content. The system reads in your website, finds the documents that cover the question, and lets the model write an answer from them. If something is wrong with what it retrieves, the answer is wrong too.

A modern assistant copes with that better than you might expect, mind you. Give it three documents of which one is out of date, and there is a real chance it notices and gives the right answer anyway. That isn't free. The model first has to work out which source still applies, and that costs time the visitor experiences as waiting. So messy content doesn't only make your assistant less reliable, it makes it slower.

I work on the side of vragen.ai where that content comes in. This piece is about what goes wrong there, and why it is more often a crawling problem than an AI problem. Most of the examples below are ones we have had to repair ourselves.

Three crawlers, one problem

Right now there are probably three kinds of crawlers reading your website. Googlebot, to index you. The crawlers of AI companies, combing the web. And, if you have an AI assistant on your own site, the crawler you hired for it yourself.

Technically those three do the same thing. They fetch a URL, get HTML back and try to pull text and structure out of it. A page one of them can't read, the others can't read either. A sitemap that is broken for Google is just as broken for us.

The difference is in how fast you notice:

• Googlebot

  Months

  A page that doesn't get read slowly slides down the rankings. By the time you see it in Search Console, it has been that way for a while.

• Third-party AI crawlers

  Never

  You get no notification and no report. Whether ChatGPT has read your page is something you can't check.

• Your own assistant

  Tomorrow

  A visitor asks a question and gets last year's answer. That lands in your inbox, with the question alongside it.

That last column is why I find this interesting. An AI assistant on your own content is the fastest quality check on your website there is. Within a day it tells you which page can't be found, which information is duplicated and which question you don't answer anywhere. For exactly the same problems, Google gives you a slight drop in rankings six months later, without explanation.

So the work you do to get your assistant giving good answers largely improves your findability too. That is because it comes down to the same HTML.

Your sitemap is broken more often than you think

A sitemap is a list of URLs that should be on your site. Nearly every CMS generates one automatically, which is exactly why nobody ever looks inside it. These are the problems we run into most in practice, all of them in sitemaps the CMS considered perfectly fine.

The sitemap lists http:// while the site runs on https://. This is by far the most annoying one, because it looks so innocent. Every URL in the list then leads to a redirect: the crawler requests the http version, gets redirected to https and has to start again. Ingestion becomes twice as slow and your server gets twice the number of requests. On a large site that is the difference between an hour and half a day.

Entries without a <loc>, or with whitespace around it. A line break or a couple of spaces inside the element, and the URL that comes out doesn't exist. A strict parser skips it, a lenient parser requests a nonsense URL.

A lastmod in a homegrown date format. That date determines when a page gets fetched again. If it can't be read, the crawler loses its most important clue that something has changed.

URLs from another domain. Often a leftover from a migration or a link with an external tool. The crawler is then handed pages that don't belong to your site.

<!-- Three errors in four lines -->
<url><loc>http://www.example.com/contact</loc></url>
<url><loc> https://www.example.com/about-us </loc></url>
<url><lastmod>11-08-2026</lastmod></url>

And then the point that often gets left out: a sitemap is a clue, not the truth. It tells you what your CMS thinks should be there. What is actually there only becomes clear once you fetch it. A good crawler uses the sitemap as a starting point and then checks for itself, because otherwise it inherits your CMS's mistakes.

If you want to know how our crawler handles this and how fast changes come through, that is in the crawler documentation.

Structured data configures your assistant

If you ever added schema.org markup to get a nicer search result in Google, you also made ingestion by AI systems easier without knowing it. That JSON-LD block in your HTML is the only place on a web page that explicitly states what something is. The rest is formatting you have to infer from.

On our side, the system takes a document's title, author and summary straight out of that structured data when it's there. Since early August it goes a step further: when you create a new source, an agent looks at your pages and proposes where metadata should come from. If your site has clean JSON-LD, it points at that and you're done.

If it isn't there, the agent works back to find which piece of HTML holds the publication date and proposes a CSS selector. That works, but it is more fragile: a theme change shifts a selector like that, and then your dates quietly stop updating. Structured data doesn't shift.

Things go wrong on our side too, by the way. In July there were document titles in the knowledge base with “ in them, where a quotation mark belonged.

The cause was in our own HTML sanitization. It converts punctuation and accents into a safe code, and “ is such a code. Fine for ordinary text, but it did the same inside the JSON-LD block, and that conversion doesn't belong there. So the title came out of the structured data correctly and got damaged on the way.

Nobody sees errors like that, until a visitor gets an answer with an odd title underneath it. The model had nothing to do with it.

The crawler has to be able to get in

Before ingestion starts, the crawler has to get through the door. There are two layers there, and most organizations only know the top one.

The first is robots.txt. Research firm Writgo scanned the five hundred most visited .nl domains on 1 August 2026: 21.7 percent block at least one AI crawler there, with GPTBot the most excluded at 16.8 percent. That is a legitimate choice; not everyone wants their content ending up in a language model. It is worth knowing that a rule like that can reach wider than intended, especially when it was written using a generic list of bot names.

The second layer is the one that research explicitly says nothing about, and the one we run into most in practice: your firewall. Cloudflare, a WAF, bot fight mode, rate limiting. Those systems don't look at your robots.txt and decide for themselves. A crawler that sends a proper user agent and sticks to the rules looks, to a filter like that, exactly like the traffic it is meant to stop.

You recognize it by 403 errors in your server logs. On our side you see it in the message "system temporarily paused": if ingestion hits too many errors in a short time, we stop automatically. That is deliberate. We would rather not keep hammering a site that can't handle the requests at that moment. The steps to solve it, including the settings for Cloudflare, are on that same documentation page.

Only your knowledge, in four formats

One thing gets easily buried in this story. vragen.ai uses only the sources you provide. Nothing is pulled in from elsewhere on the internet, and the model doesn't top things up from whatever it once read. If the answer isn't in your content, there is no answer. That is why this whole piece is about ingestion: it is the only place knowledge comes in.

Content comes in three ways. Through the crawler, which reads your website and keeps track of changes. Through a direct connection with your CMS, using the API or one of the plugins. Or through a feed, both RSS and Atom. For documents we process HTML, PDF and Word, both current DOCX and older DOC.

Having PDFs on that list is convenient, and worth a warning at the same time. A web page has structure a machine can read: headings are headings, a table is a table. In a PDF that is often gone. What remains is text at coordinates, and the structure has to be inferred from it. Researchers behind OHRBench, a benchmark with over eight thousand document images presented at ICCV 2025, looked at what that does to the answer quality of RAG systems. Their conclusion about the available techniques for reading documents is blunt: none is good enough to build a reliable knowledge base on. Errors made while reading carry through into the answer, and nothing further down the chain repairs them.

The practical lesson: if important information only exists in a PDF, a normal web page is almost always the better source. That goes for your findability too, incidentally.

Where technology stops and editing begins

Up to here it was about things you can repair. Now the part that has no technical solution.

Say the 2019 guideline is still online, and the revised 2024 version is there as well. Both cover the question someone asks. Both come from your own website, so both are equally reliable. Which of the two applies now is a judgment about substance. Your organization can make that call and a system can't.

This is a known and stubborn problem. A research group at Google published CONFLICTS in June 2025, a benchmark for exactly this kind of contradiction between sources. The outcome: language models often don't resolve such conflicts well. What does help is making the model reason about the conflict explicitly. Picking a better model and hoping it goes away doesn't help.

You have two levers here, and both cost something. You can switch the old version off, so it drops out of the running. That costs no extra time, but you have to know in advance which page is out of date. With a handful of guidelines that works, with thousands of documents it doesn't. Or you let the agent judge: it weighs how old a document is and looks for a more recent version when needed. That does scale, but it costs a reasoning step, and the visitor waits slightly longer for an answer.

It also helps to know that removing works differently than people expect. Deleting takes a document out now, but the crawler picks it up again on a later round if it is still on your site. Switching off remembers that you deliberately don't want it. So for an archive that has to stay online but doesn't belong in answers, switching off is the right lever. And when a page genuinely disappears from your site, count on a day's delay; if the subject keeps coming back after that, other pages still link to it. There is more about that in managing knowledge base content.

That clean-up work is exactly the work that makes your site more findable too. Duplicate pages, outdated versions and content you can't reach from anywhere: a search engine struggles with all of it just as much.

What you can check today

Six things that aren't a project and that help both your assistant and your findability.

• Open your sitemap and check whether the URLs are on https, whether the lastmod dates are right and whether any odd domains have crept in.
• Search your server logs for 403 errors from crawlers. That is your firewall stopping something.
• Check what your robots.txt says and whether those rules are still what you meant.
• See whether your most important pages have structured data, and whether it holds a decent title and date.
• Find the subjects that have two versions online and decide which one applies.
• Check whether information that really matters exists only in a PDF.

If you are already getting outdated answers from an AI assistant, that list is your diagnosis. And if you're still weighing whether something like this fits your organization: you can try our own widget on our own content, including the sources under every answer. What you read above is exactly the work sitting underneath it.