Tell the crawlers to index all your pages using robots.txt
Before indexing pages, the spiders read the content of the robots.txt file. There you can instruct the robots to ignore and not index certain files and directories. If they don’t find a robots.txt file, they usually proceed with indexing the website.
However, if you don’t have a robots.txt file, the server will keep returning 404 errors to the robot. And if you don’t want that, you can create a robots.txt file where you specify that the robots should index everything on your website:
User-agent: *
Disallow:
If you want to prevent the robots from indexing any of your pages, use the following:
User-agent: *
Disallow: /
February 7th, 2006 at 7:07 am
Well i had added my website to many search engines n for crawlers but i can’t see it listed ever i don’t know why can u comment on it.
February 7th, 2006 at 7:24 am
That’s because you’re using IFrames. Google, MSN Search and the other search engines get very confused when it comes to frames. So unless you remove them, chances are you’ll never get indexed with the right content.