Docs Content & answers

The crawler and your content

A crawler, also known as a web crawler or bot, is an automated program that visits websites and analyzes their content. Crawlers are often used by search engines to index web pages, but they can also be used for other purposes, such as gathering information for AI systems.

What does the vragen.ai crawler do?

The vragen.ai crawler is used to retrieve and process relevant information from your website. This enables our AI to give accurate and up-to-date answers to the questions users ask. The crawler follows the guidelines set in the robots.txt file and respects standard crawling intervals to minimize server load.

How does your content stay up to date?

The crawler revisits your website regularly and automatically picks up changed pages. You can expect a change on your website to be reflected in the answers within 1 to 2 hours.

Don't want to wait, or want to check whether a page was indexed correctly? Log in to your environment (the Login button is at the top of this website) and open your knowledge base. There you can see which pages and documents have been indexed and request a new crawl yourself. Not working, or is it a trickier case? Email service@vragen.ai.

User Agent

Our crawler identifies itself with the following User Agent:

Vragenai

You can recognize our crawler in your website's server logs by this User Agent.

IP addresses of the crawler

The vragen.ai crawler uses a fixed IP range to send requests to your website. This makes it possible to manage access at the server level.

All crawler requests currently come from the following IP addresses:

87.233.245.192/28

Note: the IP addresses used may change in the future. If a change occurs, you will receive a notification so you can make any necessary adjustments to your configuration.

How can you manage the vragen.ai crawler?

Robots.txt settings

You can manage the behavior of the vragen.ai crawler by adding rules to your robots.txt file. For example:

User-agent: Vragenai
Disallow: /private-folder/

This prevents the crawler from visiting the /private-folder/ directory.

Adjusting the crawl rate

Our crawler respects crawl-delay directives if they are defined in robots.txt. For example:

User-agent: Vragenai
Crawl-delay: 10

This sets the crawler to wait at least 10 seconds between consecutive requests.

Allowing the crawler through your firewall

Firewalls and bot protection (such as Cloudflare) protect websites against unwanted bots, but in doing so they can also block legitimate crawlers like the one from vragen.ai. By whitelisting the crawler, you ensure that your content can be indexed without interruptions.

As a rule, always combine the User Agent and the crawler's IP range (both are listed above). A rule based on the User Agent alone is too broad: anyone can send that text along. The combination only lets our real crawler through.

Step-by-step guide for Cloudflare

  1. Go to the Cloudflare Dashboard, log in and select the right website.

  2. Navigate to Security > WAF (Web Application Firewall).

  3. Go to Custom Rules and click Create Rule. Give the rule a name, for example: Whitelist Vragenai Crawler.

  4. Set up the rule with two conditions, combined with And:

    • Field: User-Agent, Operator: contains, Value: Vragenai

    • Field: IP Source Address, Operator: is in, Value: 87.233.245.192/28

  5. For Action, choose Allow and click Deploy Rule.

If you prefer to work with IP Access Rules, that's also possible: go to Security > WAF > Tools, add the crawler's IP range under IP Access Rules and set the action to Allow.

Is the crawler still being blocked?

Check whether other rules are active that block bots, such as Bot Fight Mode or Rate Limiting, and adjust those settings if needed. Using a different firewall or WAF than Cloudflare? The recipe is the same: allow the combination of User Agent and IP range. You can recognize a block by rejected requests (for example 403 errors) from the User Agent Vragenai in your server logs.

What if I run into problems?

If you have questions or experience unwanted crawl behavior, you can contact our support team at service@vragen.ai.

Didn't find what you were looking for?

Ask your question directly to vragen.ai.

What exactly is vragen.ai?

Example answer by vragen.ai

vragen.ai lets visitors ask their question on your website and gives them a reliable answer straight from your own content, with the source included. You decide which sources the AI uses.

Source: How it works

This is an example. The interactive widget could not load here, for instance because of a script blocker or a slow connection.

Create your demo Try it yourself

You are asking an AI assistant from vragen.ai. Answers come from our own content, with the source included. Why we mention this (Dutch)