Unlocking the Mystery of Search Engines

Imagine a library with 30 billion documents with no centralized 'card catalog' and no librarians. Then imagine millions of people adding documents to this collection every day, without telling anyone.

That's the Internet we live with today. Your website is the handful of pages you add to this vast, unstructured library.

So instead of asking why your site isn't #1, you should be asking “how the heck does anyone ever find my website, at all?”

Since search engine optimization is one of those topics that, at first glance, seems to be “a riddle, wrapped in a mystery, inside an enigma”, I thought I'd use this article to unlock some of the mystery before getting down to the nuts and bolts.

When you have a working knowledge of how search engines really work today, you'll know most of what you need to know to attract qualified buyers to your site.

How search engines work their magic

Before a stamp collector can find your website, 4 things must happen:

  1. The search engine needs to find your website, using a program called a web crawler. A crawler starts with a list of 'seed' websites. It reads those pages and finds links to other websites. Then it reads those pages and again follows any links it finds. This continues, through thousands or even millions of links and pages, until finally – hopefully – the crawler finds a link to your website.
  2. Once it finds your site, the crawler needs to read all the text on your site into the search engine's database.
  3. Then the search engine needs to index all that text. Indexing is like creating a back-of-the-book style index for every word and phrase on your site. This is where your keyword list fits in. If you don't use a keyword on your website, it won't be in the search engine's index and the search engine won't find your site when someone searches for that keyword.
  4. Finally, the search engine needs to rank your pages by quality against all the other pages on the Internet, so searchers get the best pages first.

Steps 1-3 are pretty simple and even early search engines did a good job with them. Step 4 was the tricky bit. How could a mere computer decide which site had higher quality content?

In the early days, Yahoo! used human editors to rank sites. Other search engines tried to automate the process, with mixed results.

Then two guys, named Larry Page and Sergey Brin, thought they could build a better search engine by analyzing the links between websites.

Their technique, called PageRank™, is illustrated in the diagram below. In essence, Page and Brin's search engine interpreted a link from page A to page B as a vote by page A for page B.

But not all votes were equal. Votes from pages that were themselves “important” were worth more than votes from “unimportant” pages, and helped make the pages they voted for more important.

This works because when you add a link from your website to another, you're telling your customers “Here's a website I think you'd like”. It's your vote for the other website.

Page and Brin used this simple idea to build a search engine called 'BackRub'. Luckily, they renamed it before it became the most popular search engine in the world. Otherwise we'd be 'BackRubbing' for stuff, instead of 'Googling' for it.

Putting this new information to work

We'll get into the details next time, but here's the take away:

  • You need to include your keywords in your website text, so your site is indexed for those keywords
  • You need need to get more links to your website from other sites that Google regards as “important”

And here are two exercises you can do right now:

  • To find out how many websites link to yours, go to Google and type the following into their search box: "link:http://www.yourwebsitename.com"
  • To get an approximation of your site's current Google PageRank, go to http://www.checkpagerank.com and type in your website address.

Then try the same with your competitor's sites!

Is all this information really useful? Definitely! By focusing his website (discussed 2 issues ago) and using the right keywords (last issue), one of our clients rocketed to the first page of Google listings for his primary keyword, “European Stamps” in less than one year. Your site can, too.

Next month, we'll use your new-found understanding to look at some practical steps you can take to attract qualified buyers to your website. Until then, have a great month SELLING ON THE WEB.