Why we still recommend blocking AI crawlers

In a world where AI is blatantly stealing your content, we’re looking for every opportunity to protect it. That’s why we introduced our Terms of Content Use, and why we’ve been blocking AI crawlers through the Raptive Ads WordPress plugin by default since 2024. 

Since then, the question we’ve heard most from creators is this: does blocking AI crawlers hurt my traffic?

While we were confident that blocking AI crawlers was the right move—we wouldn’t have recommended it otherwise—we continued collecting data to remain sure.

After tracking data for the past year across the 6,000+ sites in the Raptive network, we can confidently say that blocking AI crawlers does not affect your traffic or search rankings.

What does it mean to block AI crawlers? 

When you block an AI crawler—sometimes referred to as an AI bot—you’re telling it via your website’s robots.txt file not to index or scrape your content for training AI models. This doesn’t block legitimate traffic or stop Google’s standard search crawler from finding your site. It only limits specific AI crawlers from collecting and potentially using your content without consent or compensation. 

Blocking AI crawlers prevents your original content from being used to generate AI outputs that could compete directly against you for your traffic. And perhaps most importantly, it signals support for the development of responsible, ethical AI. 

We’re not alone in this recommendation. Publishers like The New York Times, Wall Street Journal, Vox, and Reuters have all blocked AI crawler access. 

The data supports blocking AI crawlers

From June 2024 to May 2025, we tracked traffic patterns across thousands of Raptive creator sites to understand the real impact of AI crawler blocking. 

Here’s what we found:

• Sites that blocked one or more AI bots saw no statistically significant change in traffic compared to sites that did not block bots. The average traffic variation across cohorts was within 1%, which is what we expect to see due to variables like core updates and changes in search behavior.
• There was no evidence that blocking AI bots led to negative traffic performance.

As of July 2025, approximately 40% of Raptive sites block GPTBot and nearly 10% block Google-Extended. 

The decision is yours

We believe that you should always have full autonomy over your decisions. The Raptive Ads WordPress plugin blocks AI crawlers by default, but you can adjust these settings at any time and choose which crawlers to block or unblock. 

You can review and update your AI crawler blocking preferences in your Raptive Dashboard or the Raptive Ads WordPress plugin. If you’re not using the plugin, here’s how to manually block common AI crawlers.

From our end, this isn’t a set-it-and-forget-it recommendation. We’ll continue analyzing the data, and if anything changes, our creators will be the first to know. Given the most current data, we continue to recommend blocking AI crawlers as a straightforward and proven way to protect your work without sacrificing traffic or visibility.