Search Engine Positioning
with Spiders & Directories
Search engine is a blanket term that covers two types of indexing systems used in the positioning of web sites. One type of search engine is known as a robot, spider or crawler, and indexes web sites through an automated program. The other type is known as a directory, which is powered by humans - usually volunteers.
Crawlers aka Robots aka Spiders
Robots aka spiders aka crawlers are computer programs with algorithms, or formulas that tell them to crawl through a site and index its content. If your site is designed with good navigation, the robots will crawl through all of your pages, figure out how they are linked and index your site.
After the site is indexed and a search is conducted, the accompanying search algorithm analyzes the content, title tags, meta tags and filenames, among other factors. If your site is optimized for the search engines, and these components are relevant to your keyword search phrase, this is where that information will be noted and your site will be ranked accordingly to the search engine positioning.
Robots crawl through the web periodically. If changes are made to your site, they will be noted and analyzed in the crawl by the search engine following the changes. Depending on the changes, your positioning with a particular keyword search phrase could be affected.
Each crawler has its own methods and algorithms when it comes to positioning a site, and this information is a trade secret.
Directories
Humans catalog directories, which are organized and positioned in a hierarchy of topics and sub-topics. Each topic and sub-topic is assigned to an editor, and search engine positioning is done manually.
Submission to a directory begins with selecting the correct category that best represents your site. Once you have located the right section, you submit from that page. Usually, A title and short description of your site and maybe some keywords are requested along with your domain name. Upon submission, your site is reviewed by the editor and either posted or rejected.
There are no periodic crawls with directories, so changes to your web site are not noted unless you resubmit your site manually.
