Earlier this year if you asked me about search engine sitemaps, I would have told you, “Yes, you need a sitemap.xml and you need to make it top priority!” but now I question the effectiveness of having one.
To be honest, the process in how I started to question this is largely because I just got lazy and didn’t feel like taking the extra time to create a sitemap or install the WordPress sitemap generator.
As time continued to pass by, I didn’t notice any difference in the frequency in which content got indexed on sites with sitemaps compared to sites without sitemaps.
I run a lot of sites and today, most I haven’t even bothered with installing a sitemap, or even registering in the many webmaster programs made available by the search engines.
Long story short… I just blog. I blog and I blog and I blog and all my content gets indexed.
Now, with all this said, I do feel that a robots.txt without a doubt is necessary. This controls where robots are allowed on your site.
A sitemap, I really don’t think it helps or harms you. I am sure that programmers of the major search engines understand that only a small percentage of people really know what a sitemap.xml file is. I am sure this is taken into consideration.
Lastly, I have spent a little time researching various sites and pages that pull up in the SERPs for random keywords. You would be amazed to discover the amount of sites that return first in the index that don’t have sitemaps.xml files.
Did you enjoy reading this post? I want to invite you to subscribe to my blog so that you won't miss my next article. Subscribing to my blog is totaly free. Simply, enter your email address in subscription box below and immediately you will start to receive copies of my articles sent to you by email. Your email address is never shared and you can unsubscribe anytime.
Date/Time: 12-10-2007 05:54:05 Comment #7176
It would be awesome if you did a little tutorial on how to create and use Robots.txt.
Date/Time: 12-10-2007 07:35:42 Comment #7179
-Garry, How do you know those sites do not have a sitemap? Sitemaps can be named differently, they are not like the robots file. I disagree with you on this one. Sitemaps are probably the only thing Google, MSN & Yahoo agree on, and that is good enough for me.
Date/Time: 12-10-2007 11:57:47 Comment #7190
Just reporting the facts. I have sites with and without sitemaps… Also I have sites with and without even robots.txt files.
More sites don’t have sitemaps than do. Just out of my sheer laziness I have neglected to take the time to implement sitemaps as well as even robots.txt files on many of my sites.
After a few months time, I don’t see much if any difference in indexing.
You very well might be right.. I’ll give you that. But based off the performance I am seeing in my sites that have them verses not, I don’t have much of a priority to continue to invest the time into adding them.
Robots.txt I see as being very useful. But a sitemap, at least for the time being, I see as being optional.
Date/Time: 12-10-2007 12:41:30 Comment #7191
Garry, you have got to learn, the POINT is this, most people do not know all the OTHER stuff you do, THEY NEED SITEMAPS, you may not. I think you do.
You have to remember that you are not the typical person reading your site.
And sitemaps are not intended to make you index better or more, they are a roadmap for the bot, and most people need all the help they can get.
I have even tested putting a file in the sitemap that was not linked anywhere on a site to see if it got picked up, and it did. That alone tells me the spiders are using it.
Date/Time: 12-10-2007 12:49:47 Comment #7194
I’ve been setting up sitemaps using the generator plugin since I started blogging and hope that it does make a difference!
The robot.txt file is something I’ve been struggling with for a while until recently – i had the order of the bots wrong, so GoogleBot basically indexed all the folders despite all my attempts at varying how I set it up.
I couldn’t believe my simple error once I figured it out! I was kicking myself for days after that!
Date/Time: 12-10-2007 13:02:53 Comment #7195
-Zath, I tried every plugin I could find for Sitemaps, none of them did what I thought they should. They produce a sitemap based on what is in Wordpress, not on an actual crawl of the site. I highly recommend using GSiteCrawler instead. And make sure to put the autodiscovery of the Sitemap in the Robots.txt file.
Date/Time: 12-10-2007 16:10:20 Comment #7202
Thanks David, I’ll take a look at your suggestion and have a proper read up on what sitemaps should be doing as I’ve just taken it as read that the recommended plugins were doing it right in the past.
Date/Time: 12-10-2007 16:46:30 Comment #7206
@ David,
I may know more than the average blogger… however, that in itself is the exact point I am trying to make. Many people don’t know about sitemap.xml nor do they know about robots.txt. So, with that being said, what are search engines to do? Only index and rank the sites that uses these better than ones without?
I have removed sitemap.xml as well as robots.txt from this blog in fact and things are running great. This blog here is no different that any other average or typical blogger’s blog.
I blog and post content and use tags. There is nothing any more special about this blog then any other blog online.
I haven’t seen any results to speak up after investing the time into creating sitemaps. Robots.txt… yeah I can see how people would want that. But, in the case of GCDC… screw it. I am blogging and search engines can pick up my content and readers can read it if they want. I am serious… no sitemap an no robots.txt and I have had more pages index in the last two days than I typically get in an entire month. My SE traffic has doubled and I haven’t seem much of a change in regular reader traffic (no drops)… feedburner remains the same.
I think sitemaps can come in handy… but I think bloggers need to focus on other opportunities first such as correctly naming their blog post titles and assuring the proper balance of exact phrase words as links in their posts.
Date/Time: 12-14-2007 18:13:18 Comment #7486
Hi Garry,
I’m a little confused. In your post you said, “I do feel that a robots.txt without a doubt is necessary.” But in the comments it looks like you’ve removed that file as well. Please clarify your stance. Do you think robots.txt still serves a purpose?
Thanks,
Terry
Date/Time: 12-14-2007 22:00:14 Comment #7491
-Garry, ok, being open minded is something I believe in, so I have removed sitemaps and robots.txt from a handful of sites to test, including one of my large sites, so I will let you know how that goes.