PDA

View Full Version : Googlebot...

Joe
08-12-2006, 10:23 PM
How do I get this shit off of my site? I keep seeing it accessing the forum when I check my control pannel:

crawl-66-249-66-82.googlebot.com
Cuckoo
08-12-2006, 11:58 PM
You can use a robots.txt file. That's what I use since I have no use for any of the search engines indexing my forum.
Cringer
08-13-2006, 12:03 AM
robots.txt file is what I use. I didn't have a problem with it until the Imperial League's last offseason. Between the heavy every day usage of the board by owners, combined with the draft utility, combined with a big jump of spiders/googlebots in that month, I went over my bandwidth usage. Thankfully the webhost gave me some extra bandwidt for free that month.

let me know GWB, and I will send you mine. they are easy to do.
Joe
08-13-2006, 07:06 AM
yeah Cring, email it to me
Icy
08-13-2006, 01:02 PM
George, you usually will love to be crawled by google as it means it will index your forum threads and will display them in google related searches. It will mean more visitors to your forum. It's free advertisement so unless you want your forum totally private, you should be so happy for being crawled.

If you don't want to be crawled, use the robots.txt to tell google which pages you don't want indexed.
Coder
08-13-2006, 01:08 PM
George, you usually will love to be crawled by google as it means it will index your forum threads and will display them in google related searches. It will mean more visitors to your forum. It's free advertisement so unless you want your forum totally private, you should be so happy for being crawled.

If you don't want to be crawled, use the robots.txt to tell google which pages you don't want indexed.


Well for a commercial website this is certainly true, but what happened in the Imperial League was definately not a good thing. Not only did we get an increase in "spam-accounts", but like Cringer said, the bandwidth usage went haywire, causing us to more than double our regular usage.

I personally don't think it's important to have fantasy-leagues forums crawled by Google.. our aim is not to have more visitors, but to supply a service to the people who are in the league.
Cringer
08-13-2006, 01:51 PM
Emailed it to you GWB.
Icy
08-13-2006, 02:24 PM
Well for a commercial website this is certainly true, but what happened in the Imperial League was definately not a good thing. Not only did we get an increase in "spam-accounts", but like Cringer said, the bandwidth usage went haywire, causing us to more than double our regular usage.

I personally don't think it's important to have fantasy-leagues forums crawled by Google.. our aim is not to have more visitors, but to supply a service to the people who are in the league.

Totally agree, i was not sure about what kind of forum did he own.