Whitelist the crawler in Cloudflare

Cloudflare protects websites against unwanted bots and attacks, but sometimes this can block legitimate crawlers like the Vragen.ai Crawler. By whitelisting our crawler, you ensure that our AI can correctly index your content without being interrupted by security measures.

How to recognize the Vragen.ai Crawler?

Our crawler uses the following User Agent:

Learn more about our Crawler

Steps to whitelist the Vragen.ai Crawler in Cloudflare

1. Logging into Cloudflare

• Go to Cloudflare Dashboard and log into your account.

• Select the appropriate website for which you want to whitelist the Vragen.ai Crawler.

2. Create Firewall Rule

1. Navigate to Security > WAF (Web Application Firewall).

2. Go to Custom Rules and click on Create Rule.

3. Name the rule, for example: Whitelist Vragenai Crawler.

4. Set the rule as follows:

   • Field: User-Agent

   • Operator: contains

   • Value: Vragenai

5. Choose at Action: Allow.

6. Click on Deploy Rule to save the settings.

3. Whitelist the IP address of the crawler (optional)

As an additional measure, you can whitelist IP addresses used by our crawler:

1. Go to Security > WAF > Tools.

2. Add a new IP address under IP Access Rules (contact us for the current IP ranges).

3. Set the action to Allow and click on Add.

Frequently Asked Questions

What if the crawler is still blocked?

Check whether other rules blocking bots are active, such as Bot Fight Mode or Rate Limiting. Adjust these settings if necessary.

How often does Vragen.ai crawl my site?

Our crawler follows the guidelines in your robots.txt file and adapts based on server load.

Can I adjust the crawl speed?

Yes, you can set a Crawl-delay in your robots.txt file.

Support

By following the steps above, you ensure that the Vragen.ai Crawler can properly index your website without Cloudflare blocking the traffic. If you have any questions, please contact our support team via support@vragen.ai.