| Robots.txt Generator Highlights |
|
|
|
Import your site structure
|
|
Enter your domain to retrieve your site's structure from Yahoo Site Explorer. Then just click on a directory to disallow access.
|
|
|
|
|
|
|
Choose the type of robot
|
|
Easily choose which type of search engine crawler you want to disallow access to. Block all of them, or choose from web search, images, ads, archivers or the bad spiders.
|
|
|
|
|
|
|
Copy & paste your robot rules
|
|
Just create a robots.txt file with the content generated and upload to your website root.
|
|
|
|