Search Engine Basics and Anatomy - SEO 101 Course

SEO 101 Course, Part 1

What Is a Search Engine?

Okay, so you know the basic concept of a search engine. Type a word or phrase into a search box and click a button. Wait a few seconds, and references to thousands (or hundreds of thousands) of pages will appear. Then all you have to do is click through those results to find what you want. But what exactly is a search engine, beyond this general concept of "search-find"?

It's a little complicated. On the back end, a search engine is a piece of software that uses algorithms to find and collect information about web pages. The information collected is usually keywords or phrases that are possible indicators of what is contained on the web page as a whole, the URL of the page, the code that makes up the page, and links into and out of the page. That information is then indexed and stored in a database.On the front end, the software has a user interface where users enter a search term - a word or phrase - in an attempt to find specific information. When the user clicks a search button, an algorithm then examines the information stored in the back-end database and retrieves links to web pages that appear to match the search term the user entered.

The process of collecting information about web pages is performed by an agent called a crawler,spider, or robot. The crawler literally looks at every URL on the Web that's not blocked from itand collects key words and phrases on each page, which are then included in the database thatpowers a search engine. Considering that the number of sites on the Web exceeded 100 millionsome time ago and is increasing by more than 1.5 million sites each month, that's like yourbrain cataloging every single word you read, so that when you need to know something, youthink of that word and every reference to it comes to mind.

Anatomy of a Search Engine, and How it Works

By now you probably have a fuzzy idea of how a search engine works, but there's much more to it than just the basic overview you've seen so far. In fact, search engines have several parts. Unfortunately, it's rare that you find an explanation describing just how a search engine is made - that's proprietary information that search companies hold very close to their vests - and that information is vitally important to succeeding with search engine optimization(SEO).

Query interface

The query interface is what most people are familiar with, and it's probably what comes to mind when you hear the term ‘‘search engine.’’ The query interface is the page, or user interface, that users see when they navigate to a search engine to enter a search term.There was a time when the search engine interface looked very much like the Ask.com page.This interface was a simple page with a search box and a button to activate the search, and not much more.

Today, many search engines on the Web have added much more personalized content in an attempt to capitalize on the real estate available to them. For example, Yahoo! Search, is just one of the search services that now enable users to personalize their pages with a free e-mail account, weather information, news, sports, and many other elements designed to make users want to return to that site to conduct their web searches.One other option users have for customizing the interfaces of their search engines is a capability like the one Google offers. The Google search engine has a customizable interface to which users can add different gadgets. These gadgets enable users to add features to their customized Google search home page that meet their own personal needs or tastes.

Search has even extended onto the desktop. Google and Microsoft both have search capabilities that, when installed on your computer, enable you to search your hard drive for documents and information in the same way you would search the Web. These capabilities aren't of any particular use to you where SEO is concerned, but they do illustrate the prevalence of search and the value that users place on being able to quickly find information using searching capabilities.

When it comes to search engine optimization, Google's user interface offers the most potential for you to reach your target audience, because it does more than just optimize your site for search: If a useful tool or feature is available on your site, you can enable users to have access to this tool or feature through the Application Programming Interface (API) made available by Google.Using the Google API, you can create a gadget that users can install on their Google Desktop,iGoogle page, or FireFox or Chrome browser. This enables you to have your name in front of users on a daily basis.

For example, a company called PDF24.org offers a Google gadget that enables users to turn their documents into PDF files right from their Google home page once the gadget has been added. If the point of search engine optimization is ultimately to get your name in front of as many people as possible, as often as possible, then making a gadget available for addition to Google's personalized home page can only further that goal.

This graph figure shows the 5 most used search engines internationally, it also shows the competition between Google and the other 4, more than eight in every ten searches made globally are done through Google.

Search engine results pages

The other sides of the query interface, and the only other parts of a search engine that's visible to users, are the search engine results pages (SERPs). This is the collection of pages that are returned with search results after a user enters a search term or phrase and clicks the Search button. This is also where you ultimately want to end up; and the higher you are in the search results, the more traffic you can expect to generate from search. Specifically, your goal is to end up on the first page of results - in the top 10 or 20 results that are returned for a given search term or phrase. Getting there can be a mystery, however. We’ll decode the clues that lead you to that goal throughout the book, but right now you need to understand a bit about how users see SERPs.

Let's start with an understanding of how users view SERPs. Pretend you're the searcher. You go to your favorite search engine - we'll use Google for the purposes of illustration because that's every one's favorite, isn't it? Type in the term you want to search for and click the Search button.What's the first thing you do when the page appears?Most people begin reading the titles and descriptions of the top results. That's where you hook searchers and entice them to click through the links provided to your web page. But here's the catch: You have to be ranked close enough to the top for searchers to see those results page titles and descriptions and then click through them, which usually means you need to be in the top 10 or 20 results, which translates into the first page or two of results. It's a tough spot to hit.

This figure shows the percentage of search engine users in UK.

Also Google is far from any other competition

There is no magic bullet or formula that will garner you those rankings every time. Instead,it takes hard work and consistent effort to push your site as high as possible in SERPs. At the risk of sounding repetitive, that's the information you'll find moving forward. There's a lot of it, though, and to truly understand how to land good placement in SERPs, you really need to understand how search engines work. There is much more to them than what users see.

Crawlers, spiders, and robots

The query interface and search results pages truly are the only parts of a search engine that the user ever sees. Every other part of the search engine is behind the scenes, out of view of the people who use it every day. That doesn't mean it's not important, however. In fact, what's in the back end is the most important part of the search engine, and it's what determines how you show up in the front end.

If you've spent any time on the Internet, you may have heard a little about spiders, crawlers,and robots. These little creatures are programs that literally crawl around the Web, cataloging data so that it can be searched. In the most basic sense, all three programs - crawlers, spiders,and robots - are essentially the same. They all collect information about each and every web URL.

This information is then cataloged according to the URL at which they're located and are stored in a database. Then, when a user uses a search engine to locate something on the Web, the reference sin the database are searched and the search results are returned.

Databases

Every search engine contains or is connected to a system of databases where data about each URL on the Web (collected by crawlers, spiders, or robots) is stored. These databases are massive storage areas that contain multiple data points about each URL. The data might be arranged in any number of different ways and is ranked according to a method of ranking and retrieval that is usually proprietary to the company that owns the search engine.

You've probably heard of the method of ranking called PageRank (for Google) or even the more generic term quality scoring. This ranking or scoring determination is one of the most complex and secretive parts of SEO. How those scores are derived, exactly, is a closely guarded secret, in part because search engine companies change the weight of the elements used to arrive at the score according to usage patterns on the Web.

The idea is to score pages based on the quality that site visitors derive from the page, not on how well web site designers can manipulate the elements that make up the quality score. For example, there was a time when the keywords that were used to rank a page were one of the most important factors in obtaining a high-quality score.

That's no longer the case. Don't get me wrong. Keywords are still vitally important in web page ranking. However, they're just one of dozens of elements that are taken into consideration,which is why a large portion of Part II of this book is dedicated to using keywords to your advantage. They do have value; and more important, keywords can cause damage if not used properly.