Quantcast
Channel: phpBB.com
Viewing all articles
Browse latest Browse all 2592

[3.3.x] Support Forum • Re: Rate Limiting GoogleBot?

$
0
0
Quote from another site (non-public topic):
Google ignores a lot of directives, e.g., nofollow, noindex & crawl-delay (in robots.txt). However, they do honor 'disallow' statements in robots.txt.

Recently I added these lines to robots.txt & this resulted in a fairly dramatic decrease in Google (and other crawler) traffic:
The lines in question are not for phpBB syntax, but I added the phpBB equivalents to my robots.txt.

Code:

Disallow: *sid=Disallow: */index.php?p=
Offhand I don't know how effective they are with Google bots, since Google hasn't been hammering my site anyway. They should work though.

Statistics: Posted by Gumboots — Fri May 02, 2025 6:25 pm



Viewing all articles
Browse latest Browse all 2592

Trending Articles