Currently, the leading search engines include Google, Yahoo! and Bing search. They each use a similar crawler to find pages and rank them according to certain algorithms. They each also have tools for webmasters to optimize their rankings in the search engines properly. Not all websites will be indexed in the search engines.
Yahoo! has a paid service that guarantees search engine crawling. They do not guarantee page ranking but only that the website will be crawled and ranked. The Yahoo Directory requires that webmasters manually submit their pages to the search engine.
Google will automatically crawl web pages. Like other search engines, not all pages will be ranked on the search engine. Google has webmaster tools like XML Sitemap that will help webmasters ensure that all their pages are found and indexed properly.
Bing Webmaster Tools allow webmasters to determine the crawl rate of the search engine, track their pages’ status and submit a sitemap making it easier for tracking search engine data.
Search engines can be instructed to stay away from certain pages. This could include shopping carts or user specific information that shouldn’t be shown in the search engine ranks. This is done with meta tags that specifically tell the spiders not to crawl the page. It’s found in the robots.txt file.
To increase page ranking and prominence in the page ranking system of search engines, a web creator can include meta data that increases the likelihood that visitors will find their sites. Relevant keywords in the content are essential too.
Cross linking is important within the website to increase and improve the page’s visibility within search engines. Content creators in the past used reciprocal links to rank higher in the search engines. Reciprocal links lost some of their importance while resource links have gained value. Resource links provide added quality and relevance which Google has stated ranks well.