Vragen.ai crawler
A crawler, also known as a webcrawler or bot, is an automated program that visits websites and analyzes their content. Crawlers are often used by search engines to index web pages, but they can also be deployed for other purposes, such as gathering information for AI systems.
What does the Vragen.ai Crawler do?
The Vragen.ai Crawler is used to retrieve and process relevant information from your website. This enables our AI to provide accurate and up-to-date answers to questions that users ask. The crawler follows the guidelines set in the robots.txt file and respects default crawling intervals to minimize server load.
User Agent
Our crawler identifies itself with the following User Agent:
You can recognize our crawler in your website's server logs by this User Agent.
IP addresses of the crawler
The Vragen.ai Crawler uses a fixed IP range to send requests to your website. This enables management of access at the server level.
All crawler requests currently come from the following IP addresses:
Note: The IP addresses used may change in the future. If a change occurs, you will receive a notification so that you can make any necessary adjustments to your configuration.
How can you manage the Vragen.ai Crawler?
Robots.txt settings
You can manage the behavior of the Vragen.ai Crawler by adding rules to your robots.txt file. For example:
This prevents the crawler from visiting the /private-folder/ directory.
Adjusting crawl speed
Our crawler respects crawl-delay guidelines if these are defined in robots.txt. For example:
This sets the crawler to wait at least 10 seconds between consecutive requests.
What if I encounter issues?
If you have questions or experience unwanted crawling behavior, you can contact our support team via info@vragen.ai.
