Managing content in your knowledge base
You decide what's in your knowledge base. This page explains how content comes in, how to add a source or individual page, and how to disable, delete or reindex something. How the crawler reads your site is covered in The crawler and your content.
How does content get into your knowledge base?
Content comes in three ways:
-
Via the crawler. The crawler reads your website and keeps track of changes. This is the default for most customers.
-
As a separate document. You manually add a single page or file.
-
Via the API or a plugin. Your CMS pushes documents in directly, for example with the WordPress plugin, the Drupal module or the API.
The difference matters when managing your content: you refresh a crawled source differently than a document that comes in via the API (see "Having content reindexed" below).
Adding a source or website
In principle you can add a new website or source yourself in your environment. In practice we like to look over your shoulder here, so we can immediately check that the crawler works well on the new site. If in doubt, email service@vragen.ai.
Want to add individual pages within an existing source that the crawler doesn't find on its own? You can, using that source's starting points (the initial URLs the crawler searches from). An added page usually appears in your knowledge base within a day.
Working with multiple websites or languages? You can put them in separate environments, or in one environment where you decide per agent which sources it may use. See Agents and agent mode.
Disable or delete?
If you want to keep a page out of the answers, choose deliberately between disabling and deleting. The difference matters:
-
Deleting removes the document now, but the crawler can pick it up again on a next round. After that it's back in your knowledge base.
-
Disabling keeps the document out of your knowledge base permanently. The system remembers that the page has already been crawled and that you deliberately switched it off, so it won't come back by itself.
For content you want to keep out permanently, disabling is the right choice. Deleting is meant for content that can really go and is no longer on your website.
If you want to remove an entire website source, check with us first: otherwise the individual documents remain. Individual pages that disappear from your website are usually picked up by the system within a day; with a direct API connection it's faster.
Deleted content still shows up
Removed content from your website but still see it in an answer? We check for deleted pages daily, so it can take up to a day before a page has completely disappeared from your knowledge base.
Does a topic keep coming back after that? Often other pages still link to it, which makes the assistant assume it still exists. This is best fixed in the content on your site itself.
Having content reindexed
If a page no longer matches your website, you can have it reindexed. Note the difference:
-
Crawled sources you refresh yourself via your environment: log in (the Login button is at the top of this website), open your knowledge base and request a new crawl for the page that's out of date.
-
Documents via the API or a plugin aren't updated with the reindex button. They're updated as soon as your CMS pushes an update. If something goes wrong there, see Connecting via the API or get in touch.
Flagged answers help us improve
If you come across an answer that's wrong, flag it in your inbox with a short note. We use those signals to adjust the answers. More on this in Testing and evaluating.
If you get stuck, email service@vragen.ai; we're happy to take a look.