If it's one thing we have learned about robots, it is that there is absolutely no pattern to them. Most robots are stupid and wander randomly. For example, 50% of robot hits to my sites ask for the robots.txt page and
then go away never asking for anything else. Then they come back a week later, ask for the same thing and then go away, again. This happens over and over again for months. You will never never figure it out. What are
they doing? If they wanted to see if the Web site was really a Web site, they could just Ping it. This would be much faster and much more efficient. They seldom visit another page and if they do, they ask for one other page
every visit or so. Some come in and issue rapid-fire requests for every page in the Web site. How rude! You have to quit worrying so much about robots. It takes 6 months before they request enough pages to do you any
good.  We really quit thinking about them a long time ago. Build a lot of pages correctly, and, if you have reciprocal links to them, the robots will find them someday.

Try this: Go to AltaVista and type into the search box "link:YourSite.com" (Leave off the www). This will list the reciprocal links to your Web site.

Try link:crownjewels.com and you get 136 links to it. Think about this now:
The robots say to themselves, "Here is a site that must be popular or why
would so many Web sites SIMILAR to it have it's link on their pages?" Remember
that only SIMILAR sites with SIMILAR THEMES would probably have a link to
your site. They give more importance to this than you submitting your link
to them. Wouldn't you?

Go to heavily trafficked sites matching your Web site's Themes and use AltaVista
to find out how many reciprocal links they have. This will prove to you
we are right.

Search engines are nothing more than a measure of reciprocal links to your
site. The problem is, you are constantly having to fight for your positioning
in the search query listings. Forget about that. Leave the fighting to people
who are able to spend 24 hours a day trying to trick everybody. Quit trying
to compete with the large organizations pouring millions into their marketing.
Completely forget about Search Engines after submitting to them and go after
the reciprocal links. The Search Engines will then believe you are a heavily
visited site because you will be. You will now be getting the traffic you
so richly deserve.

Search engine visitors to your site, are oftentimes not qualified visitors. Too many visitors pop into your home page for 2 seconds and then leave . You know how it is. We all do it when we are using the search engines. Either it wasn't the information we were looking for, or they had this huge graphic on this stupid portal page, which just took forever to load. These visitors shouldn't even count, but they get counted as 12-18 hits in your server logs . Hits are requests to the server. One page request can incur a lot
of hits: requests to the page itself plus the graphics, each count as a hit.  Reciprocal links bring in qualified visitors. These are visitors who were  already on a Web site which had matching Themes to yours. They already have
a good idea of what type of site you are. They will come into your site and actually stay awhile. These visitors should count as double credit, they are so good. We know which type of visitor we would rather have.

How do you get people to WANT to put your link on their Web sites? Why would
a similar site put a link to your site on theirs? Simple, you have similar Themes. You are similar, but not competition.

There is one very important lesson to be learned from this crazy robot behavior.
You need to make the navigation in your Web site so easy that a visitor
can find any page within 2 clicks of your home page.

One way of doing this is installing hidden Dot Links. Dot links are little periods that are linked to other pages which are not really noticeable on your page if you put it as a period. Although they are not easily seen by the human eye, they are a link that a robot can follow in your Web site. When you do this, robots can find your pages faster and more easily.