robots.txt files

[philjohns]

Ok, I've been redesigning my website as some of you may know and it will be located on my server for a while before it goes live - however Ive already realised it is indexed in google.

By including a robots.txt file in my root directory (i.e. www.philjohnsimaging.co.uk/robots.txt) I can stop a file or secondary directory of my site being indexed right?

i.e. I want Phil Johns Imaging - Plymouth Graphic and Website Design and all of its contents to not be indexed or searched by robots.

In my robots.txt file in my root directory would I need to have the following? If not, could someone correct me please ;)




User-agent: *
Disallow: /website_redesign/




Thanks,

Phil

[Toon]

That's correct and log in to webmaster tools to delete the indexed files from google

[philjohns]

Ok, cool.

So rpesumably if i wanted to block robots from searching more than one secondary folder it would be:


User-agent: *
Disallow: /website_redesign/
Disallow: /other_directory/



Thanks

[tommylogic]

Correct.. or you could have specific folders within a hierarchy if you wanted to allow a robot to crawl a portions of a directory

User-agent: *
Disallow: /website_redesign/other_directory/

[philjohns]

Hmm - now im slightly confused! lol

Basically, I want to block the following directories of my website:

Phil Johns Imaging - Plymouth Graphic and Website Design

www.philjohnsimaging.co.uk/sisterswebsitedesign


What do I need to put in my robots.txt file to block these directories from being crawled and then added to search engines?

[tommylogic]

User-agent: *
Disallow: /website_redesign/
Disallow: /sisterswebsitedesign/