Disallow search engine access your sub-domain pages

Submitted by thehong on Sat, 08/02/2008 - 14:22

Big problem with SEO one my previous article: There are maybe many URL for one page, that is so bad for SEO. To fix that problem, we should disallow search engines to access your sub-domain pages. How to do that?

Create file robots-subdomain.txt contains:

  1. # This example keeps all robots out
  2. User-agent: *
  3. Disallow: /

Open your .htaccess file add these lines, when search engines request robots.txt from sub-domain, we redirect it to robots-subdomain.txt

  1. RewriteEngine on
  2.  
  3. RewriteCond %{REQUEST_URI} /robots.txt
  4. RewriteCond %{HTTP_HOST} !^www.your-domain\.com$ [NC]
  5. RewriteCond %{HTTP_HOST} ^(.+)\.your-domain\.com$ [NC]
  6. RewriteRule ^(.*)$ robots-subdomain.txt [L]

Comments

Beware

Submitted by SEO Marketing Vietnam (not verified) on Fri, 04/09/2010 - 00:50.

Beware of abusing robot.txt. May be you will lost some important ranking.
SEO Marketing Vietnam

Excellent

Submitted by SEO Marketing Vietnam (not verified) on Mon, 04/19/2010 - 18:00.

Thanks alot, that is what i am looking for

Thanks you

Submitted by SEO Marketing Vietnam (not verified) on Mon, 04/19/2010 - 18:02.

I would like to add that, if the site has htaccess.txt, it should be changed to .htaccess

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <sup> <br> <p> <h3>
  • Lines and paragraphs break automatically.
  • Link to content with [[some text]], where "some text" is the title of existing content or the title of a new piece of content to create. You can also link text to a different title by using [[link to this title|show this text]]. Link to outside URLs with [[http://www.example.com|some text]], or even [[http://www.example.com]].
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>. The supported tag styles are: <foo>, [foo]. PHP source code can also be enclosed in <?php ... ?> or <% ... %>.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.