The web optimization Bots That ~140 Million Web sites Block the Most

We appeared on the whole variety of web sites blocking the bots. There are various methods to dam bots with robots.txt, and this accounts for all of them together with:

• Express blocks, the place the bot is talked about and disallowed
• Basic blocks, the place all bots could also be blocked
• Any situations the place a directive allowed the bot, after blocking all bots

Caveats: this doesn’t embody every other block varieties akin to firewalls or IP blocks.

As I discussed earlier, essentially the most blocked bot is MJ12bot from Majestic. I think there are a pair causes for this.

1. They’re a distributed crawler, which means you may’t search for or block them by IPs, which makes them much less trusted.
2. They’ve been crawling the online for longer.
3. They’ve a smaller person base than extra widespread web optimization instruments and subsequently much less leverage to take away any blocks.

Listed below are essentially the most blocked web optimization bots:

And the full web sites blocking web optimization bots:

Right here’s the knowledge:

It will get somewhat extra sophisticated to research. For the above, we appeared on the important robots.txt file for an internet site, however each subdomain can have their very own set of directions. If we take a look at the ~461M robots.txt in whole, then essentially the most blocked web optimization bot is SemrushBot at 5.76%. Listed below are the highest 5:

1. SemrushBot: 5.76%
2. Dotbot (Moz): 5.34%
3. MJ12bot (Majestic): 4.96%
4. BLEXBot: 4.88%
5. Ahrefsbot: 4.67%

For this measure, we’re wanting solely at instances the place a specific bot is disallowed. It doesn’t embody any total disallow statements or instances the place solely sure bots could also be allowed. In these instances, web site house owners went out of their strategy to particularly block sure bots.

Majestic’s bot is essentially the most focused, adopted by Moz’s bot.

Listed below are essentially the most blocked web optimization bots by express mentions:

Listed below are the variety of web sites explicitly blocking web optimization bots:

Right here’s the knowledge:

We appeared on the prime 1M websites by DR, which aligns to websites with a DR >45. Semrush is essentially the most blocked adopted by Majestic and Moz.

Right here’s the way it breaks down for every particular person bot in numerous classes of internet sites. The highest 3 are:

1. Autos_and_Vehicles: 39%
2. Books_and_Literature: 27%
3. Real_Estate: 17%