Search Engine Spiders

A search engine is a computer program that:

1. Uses a proprietary and sophisticated algorithm to search a large collection of documents in a data base
2. Contains the keywords (that were entered by the person doing the searching)
3. Returns a list of documents that contain one or more of the keywords, to the searcher.

The way this information gets into the database of a search engine is from spiders programs that crawl around the internet and retrieve all of this information from websites.  These programs are called “Spiders,” “Robots” or “Crawlers,”   Just like a carriage is useless without a horse to pull it, a search engine would not work without a spider to retrieve all of the data for it.

Spiders, Robots and Crawlers

Search engines, like Google, have their spider programs searching the web constantly.   As they search, they retrieve significant information on each page that they "crawl" and then  copy this information into a huge database.  Then when a web surfer goes to Google and types in a query, the search engine can quickly retrieve this information from the database.

Google has over 8 Billion pages stored in its web database.

When you use a search engine, it looks in its database, not on the Web, to find sites or pages that match your query.

A very important element in getting your site spidered is to have numerous inbound links (links from other sites that point to your site).  Web spiders will follow these links to your site and not only will you get indexed because of these links, the more links you have pointing to your pages, the higher will be it's ranking in the search engines.

The more thorough a job a spider does of crawling your site, the more information it will pick up about your site, and the better are your chances that your information will appear in search results.

1. Place text links at the bottom of your homepage to all your main internal pages (this is not necessary for every page).
2. Create a Sitemap page that has text links to all your internal pages and have a link on your home page pointing to the Sitemap page.
3. Be sure that every page on your website that you want a spider to crawl has a link pointing to it from another page on your site.  Spiders crawl a site by following the internal links of the site, so if a page on your site does not have a link pointing to it, then the spiders will not be able to find it. 
   
   This could happen by accident, or you might have a special page that you set up for special visitors and have given them the address to this page in an email or some other advertising, but don't have a direct link to it from the rest of your site.

I will cover many more ways on getting good search engine rankings in the topic on Search Engine Optimization.

Spiders are Text Readers

The reason text links "to your site" and "on your site" are so important is that Spiders read text and follow text links but they don't read images.  Any pictures, buttons, gif files, or images of any kind on your site will not help with your ranking because a spider can't read them since they are not text.

Images are strictly for the eyes of the visitors.  They get people's attention, help explain or clarify things and make your site attractive, but they won't make any difference to a Web spider.  There is a way you can make a spider pay attention to a graphic and I'll talk about it in the next lesson on Search Engine Optimization.   Even if you have a graphic that has words in it, a spider will not pick these up because they are part of an image and not actual text.

In the next topic on Submitting your site to Search Engines and Directories, I'll talk about why it is important that you get your site submitted and give you some pointers in doing it.