SEO question about the robots.txt file?

Question

Hi, I made the silliest SEO mistake I have ever done. I disabled spiders to come to my website by doing:

User-agent: *

Disallow: /

But I had to do

User-agent: *

Disallow:

I made this mistake last Thursday and discovered it on Sunday. But for these few days the organic ranking of my website dropped significantly.

My question is, how long will it take for Google to crawl back my pages and most importantly to get the same search positions I had before?

Thanks!

Answer 1

I dont think this can take effect in just 3 days. There must be some other facts which dropped your ranking.

Answer 2

Hi

I don't think that your ranking dropped for this mistake in just a couple of days. It could be an search algorithm change that did that. To make sure try these 2 experiments.

Sign up for Google webmaster tools and verify your site. You can then see when Googlebot last accesed your site. (This is if you don't have access to the logs of your server to check this out.

Secondly you can check the 2 main engines Google/ Yahoo/ Msn(Live) for cache. Simply run this search:   cache:www.yoursite.com   and see when did they last update their database with your website. This will help you see if they came back afterall.

If you have trouble with this and they are not coming back for a period of time, try updating your website with some fresh content, and then submit a new xml sitemap if you haven't done allready to all 3 of them. A new submission to all the rest engines will also get things going but this is if you haven't made a submission over the last 3 months. If so, do not submit again as you might loose some more listings for spamming.

In good will, you should have escaped any Google Filters for these couple of days so you should really worry about this mistake, but for discovering the real reason why your rankings dropped.

Hope this helps.

Answer 3

As you have disabled your robots.txt file, spiders can crawl and index the webpages. To get the spider crawl and index your websites, add some new content to your website. Reason, if you add any new content, Search Engine spiders will automatically crawl your website to update the info in the data base.

In addition to this, submit your site map to the Google site map centre to get your website crawled and indexed.

To regain the Search Engine position, perform Social Bookmarking and build relevant authorized back links. It would be great If you build links from .org, .edu, .gov. Having the authorized back links will keep your website rank top in the search engines.

Answer 4

It depends on how often Google visits your website. Just keep checking your website logs to see when the spider comes back.

In the meantime you can submit a sitemap via webmaster tools to try and get the googlebot to come back quicker. Here's more info on that: http://www.hooverwebdesign.com/blog/sear...

Answer 5

Hi,

i am sorry for what had happened from your mistake..

any how there are some updates in google algorithm last few weeks so that might be reason for the dropping your sites organic ranking, to get back to raise again you can do some link building and summit your site to search engines ans they will again crawl your site..

Answer 6

This site rankfirstprocom will help you to get better search engine rankings. I have had great results time and time again. Check it out!