Comments on: What Is A Robots.txt File?

By: Caroline’s Favourite Blogs & Links #9 | Caroline Middlebrook

Caroline’s Favourite Blogs & Links #9 | Caroline Middlebrook — Mon, 04 Feb 2008 15:52:57 +0000

[...] does not suck. Also Garry is another blogger who provides workable, informative posts such as this guide to the robots.txt file which I really need to absorb and apply myself. Finally, Garry invites his readers to ask him [...]

By: Technology Talk - 01/13/2008

Technology Talk - 01/13/2008 — Sun, 13 Jan 2008 23:31:52 +0000

[...] Conn offers his thoughts on having a Robots.txt. It took me awhile to get the hang of them, but I’ve enjoyed a lot of success with a [...]

By: Technology Talk - 01/13/2008

Technology Talk - 01/13/2008 — Sun, 13 Jan 2008 10:01:35 +0000

[...] Conn offers his thoughts on having a Robots.txt. It took me awhile to get the hang of them, but I’ve enjoyed a lot of success with a [...]

By: Garry Conn

Garry Conn — Sun, 13 Jan 2008 03:06:02 +0000

Thanks for the comments and feedback so far everyone, this is great stuff. Kyle thanks for showing us your file. Jason L… you have a very valid point

@ Zath,

Can you contact me by email and let me know where this forum is located? I’d like to check it out and see if I can help you out a little bit before you shut it down.

By: Zath

Zath — Sat, 12 Jan 2008 18:13:17 +0000

I’ve done a few bit of work on my robots file, I spent a good few months trying to sort it out - I was trying to exclude stuff, but because of the order I placed it in the file, it wasn;t being picked up - once I sorted that out though, it’s worked quite well so far.

My ultimate aim is try and keep content to being shown on the individual pages only and index other pages of interest.

I’ve had a vbulletin forum for years now that my friends and I have mainly used, however I’ve now decided to drop the forum at my next renewal time in March since it doesn’t get enough use for the money it costs per year - with that in mind I’ve now added the forum pages to the robots file in the hope that once I do remove it, Google doesn’t penalise the rest of the site when the hundreds of forum pages suddenly disappear from the site!

Overall, in the last few months I’ve gone from around 1900 indexed pages (with lots of blog and forum duplications) down to around 1000 pages - this should continue to drop now all forum pages are being removed.

How big a difference it makes, I can’t say for sure, but I’ve now got to the point where keeping my Google indexed pages is a project and a challenge in itself!

By: Josh Spaulding

Josh Spaulding — Thu, 10 Jan 2008 21:48:36 +0000

I’m glad you mentioned it because I need to update mine. I use it to disallow archives, trackback links, images, wp-admin etc. etc.

nofollow is somewhat effective, but not all SE’s acknowledge it.

@ Frank - It’s not just about duplicate content. It’s also to consolidate good pages in order to get the spiders deeper into your blog and to direct the flow of authority to those money pages.

By: Jason L

Jason L — Thu, 10 Jan 2008 21:40:56 +0000

Just keep in mind that anyone can read your robots.txt file. If you are disallowing directories that entices others to attempt to access them to figure out what you don’t want to share. =) It’s a common practice; which is why I recommend that if you use subdirectories to do development work that you have a separate testing server.

By: Kyle Eslick

Kyle Eslick — Thu, 10 Jan 2008 14:52:35 +0000

You can view my Robots.txt file here.

It is actually a pretty standard one, though I disallowed a few files that are unique to my site that the search engines didn’t need to see like redirected affiliate links and such.

One of my News Year’s resolutions was to go through and update it, which hasn’t happened yet.

By: Frank H M

Frank H M — Thu, 10 Jan 2008 11:26:17 +0000

So far I haven’t used robots.txt on any of my blogs. I know that I have to create one to prevent indexing of the wp* directories, but for the content itself, I have so far relied on using excerpts for tag, category and archives to prevent the same content from showing up in too many places.
If I then switch to using excerpts on the front page as well, I might not need to spend too much time on a robots.txt?

These are my thoughts on robots.txt at the moment. (Being a newbie in blogging, they are of course subject to change, if somebody makes a convincing argument for changing it).