Ahrefs Site Audit and Cloudflare: fixing those annoying 404 errors

At Robuust, we handle this for you. All websites we host are properly configured for SEO tools. Want to know how to fix this yourself? Read on.

The quick fix

Don't feel like reading the whole story? Here's what you need to do:

In Cloudflare:

1. Go to Rules > Configuration Rules
2. Create a new rule with this expression:

(http.user_agent contains "AhrefsSiteAudit") or (http.user_agent contains "AhrefsBot")

3. Set Email Obfuscation to Off
4. Deploy

Done. Your next Ahrefs audit will be clean. Want to know why this works and what the safest approach is? Read on.

What's going on?

You know the drill: you run a site audit in Ahrefs and suddenly you see dozens (or hundreds) of "Page has links to broken page" warnings. All 404 errors. And they all point to URLs that look something like this:

/cdn-cgi/l/email-protection#a8c9cccdc6e8cfc5c9c1c486cbc7c5

These aren't actual broken links on your website. It's a side effect of two features that don't play nice together:

Cloudflare's Email Obfuscation replaces email addresses on your website with encrypted versions. This protects against spam bots harvesting email addresses. Real visitors see [email protected] as expected, but the HTML contains a cryptic /cdn-cgi/ link.

The Ahrefs crawler sees those links, tries to follow them, gets a 404, and flags them as broken pages.

The result: a polluted audit report where you can't spot the real issues anymore.

Why robots.txt doesn't work

Your first thought might be: "I'll just block /cdn-cgi/ in robots.txt."

That doesn't work. The crawler still discovers the links in your HTML. It just can't follow them. Result: you still get broken page warnings, because a link to a blocked URL is also considered a problem by Ahrefs.

The solution lies in Cloudflare itself: make sure email obfuscation isn't active when Ahrefs comes crawling.

Option 1: User-Agent rule (quick and simple)

The easiest solution is a Configuration Rule that recognizes the Ahrefs crawler by its user-agent.

1. Open your Cloudflare dashboard and select your domain
2. Go to Rules > Configuration Rules
3. Click Create rule
4. Choose a name, for example "SEO crawlers - disable email obfuscation"
5. Click Edit expression and paste:

(http.user_agent contains "AhrefsSiteAudit") or (http.user_agent contains "AhrefsBot")

6. Scroll down, find Email Obfuscation and set it to Off
7. Click Deploy

That's it. The rule will be active within a few minutes.

Bonus: You can add other SEO tools to the same rule:

(http.user_agent contains "AhrefsSiteAudit") or (http.user_agent contains "AhrefsBot") or (http.user_agent contains "Screaming Frog") or (http.user_agent contains "SEMrush")

Option 2: IP whitelist (more secure)

User-agents can be spoofed. A malicious bot could pretend to be Ahrefs to bypass your email protection. In practice this rarely happens, but if you want maximum security, use IP whitelisting.

This method requires you to create IP Lists in Cloudflare (under Manage Account > Settings > Lists). Not all plans support this.

Creating the IP list

1. Go to Manage Account > Settings > Lists
2. Click Create list
3. Name it ahrefs-crawlers
4. Select IP as the type
5. Add all Ahrefs IP ranges:

5.39.1.224/27
5.39.109.160/27
15.235.27.0/24
15.235.96.0/24
15.235.98.0/24
37.59.204.128/27
51.68.247.192/27
51.75.236.128/27
51.89.129.0/24
51.161.37.0/24
51.161.65.0/24
51.195.183.0/24
51.195.215.0/24
51.195.244.0/24
51.222.95.0/24
51.222.168.0/24
51.222.253.0/26
54.36.148.0/24
54.36.149.0/24
54.37.118.64/27
54.38.147.0/24
54.39.0.0/24
54.39.6.0/24
54.39.89.0/24
54.39.136.0/24
54.39.210.0/24
92.222.104.192/27
92.222.108.96/27
94.23.188.192/27
142.44.220.0/24
142.44.225.0/24
142.44.228.0/24
142.44.233.0/24
148.113.128.0/24
148.113.130.0/24
167.114.139.0/24
168.100.149.0/24
176.31.139.0/27
198.244.168.0/24
198.244.183.0/24
198.244.186.193
198.244.186.194
198.244.186.195
198.244.186.196
198.244.186.197
198.244.186.198
198.244.186.199
198.244.186.200
198.244.186.201
198.244.186.202
198.244.226.0/24
198.244.240.0/24
198.244.242.0/24
202.8.40.0/22
202.94.84.110
202.94.84.111
202.94.84.112
202.94.84.113

Creating the rule

1. Go to your domain > Rules > Configuration Rules
2. Create a new rule
3. Select IP Source Address > is in list > ahrefs-crawlers
4. Set Email Obfuscation to Off
5. Deploy

Now email obfuscation is only disabled for traffic that actually comes from Ahrefs' servers.

Which option should you choose?

| Situation | Recommendation | |-----------|----------------| | Small business website | User-Agent rule (Option 1) | | Webshop with customer data | IP whitelist (Option 2) | | Enterprise / strict security requirements | IP whitelist (Option 2) | | No access to IP Lists | User-Agent rule (Option 1) |

For most websites, the user-agent method is fine. The risk of someone specifically targeting your site by spoofing the Ahrefs user-agent is negligibly small.

After the fix

Run a new Site Audit in Ahrefs. The /cdn-cgi/l/email-protection warnings should be gone. If you still see them, wait a moment - Cloudflare rules can take a few minutes to become active.

Still seeing issues? Check if:

• The rule is actually deployed (not in draft)
• You used the correct expression
• Email Obfuscation is actually set to "Off" in the rule

More about Cloudflare

This article is part of our Cloudflare series:

• Speed up your website with Cloudflare CDN - Performance optimization
• Secure your website with Cloudflare - DDoS protection and firewall
• Cloudflare DNS Management - DNS configuration explained
• Cloudflare Page Rules - Redirects and caching configuration

Related

• Getting Started with SEO - SEO basics for beginners
• Core Web Vitals Explained - Performance metrics that matter