« Wholesale Loose Diamonds Online | Main | Diamonds, Gems, and Designer Jewels At The Oscars 2006! »
How Search Engines Find Your Site
I just found a really awesome article by Lara Ewen on how search engines crawl cyberspace looking for websites.
This is a must read for anyone that wants a good basic idea of the dynamics of Search Engine Marketing and Search Engine Optimization.
If you own or operate a diamond or jewelry website, read the rest of this article!
In order to better understand how to use search engines to market a site, it’s important to learn how search engines actually search. According to SearchEngineWatch.com, a leading search engine marketing site, there are essentially two types of search engines. Some use web-indexing robots, known as “spiders” or “crawlers,” to search or “crawl” the web for links and keywords, and others use actual human editors, who find and rank sites. Most of the larger search engines today use spiders because this automated process facilitates the search process, which can involve regularly scanning literally billions of web pages.
HOW IT WORKS
Some search engines that use spiders include Google, Yahoo! and Ask Jeeves. Paul Loeffler, public relations specialist for Ask Jeeves, explains the process: “The Ask Jeeves crawler finds pages by following links from other pages. If the Ask Jeeves crawler has not visited a URL, it’s because it didn’t discover any link to that website from other pages that we visited.”
But while most spiders perform essentially the same work, search engines vary widely in terms of the criterion they use. Google, for instance, uses proprietary software called PageRank. According to the company, this software “performs an objective measurement of the importance of web pages by solving an equation of more than 500 million variables and 2 billion terms. Instead of counting direct links, PageRank interprets a link from Page A to Page B as a vote for Page B by Page A. PageRank then assesses a page’s importance by the number of votes it receives…. There is no human involvement or manipulation of results.”
Ask Jeeves, on the other hand, sorts its crawler results based on community experts. “The Ask Jeeves search technology analyzes the web as it actually exists — in subject-specific communities,” says Loeffler. “Our algorithm goes beyond link popularity, which ranks pages based on the sheer volume of links pointing to a particular page. Instead, we determine popularity among pages considered to be experts on the topic of your search. This is known as subject-specific popularity.”
MISSED CONNECTIONS
So the spiders crawl the web looking for links and votes and recommendations. But sometimes, a site simply gets missed. There are several reasons why this might happen. “When our spiders miss a site,” according to the Google website, “it’s frequently for one of the following reasons: The site isn’t well-connected through multiple links to other sites on the web; the site launched after Google’s most recent crawl was completed or the design of the site makes it difficult for Google to effectively crawl its content.”
“Well-connected” essentially means that other credible sites frequently reference a site. This is as much about offline marketing as it is about hiring a firm to “optimize” a site’s chances of being found by a spider. “What you have to realize is that in the offline world, a brand like Zales didn’t just develop overnight; it took years of working to build,” says Aaron Wall, search engine consultant and author of SEO Book, an e-book that addresses issues of search engine optimization. “In the online world, it’s the same. The search business model won’t work if it just trusts new sites it knows nothing about.” This means that in order to get those much-coveted links that will increase a site’s search engine ranking, it’s important to do plenty of offline marketing as well as online promotion.
If a site has done the marketing needed to garner credible links — and the stress is on “credible” — but has still been missed, it may just be a question of being patient until the next few crawls. There are no hard and fast numbers about how often a search engine crawls because it’s a fully automated system. “Our crawl process is algorithmic,” says Google. “Computer programs determine which sites to crawl, how often and how many pages to fetch from each site.” The frequency with which a site gets crawled is based on many variables, including the current popularity of that site’s topic, the amount of marketing a site has been doing and the number of links to said site from other sites. If a website is growing in popularity, automated search engines will eventually pick it up.
The only exception to this is if the site design isn’t geared to attract spiders. A web designer must follow certain guidelines in order to enable a website to be recognized by these web-indexing robots. “Owners and webmasters of a site should research online resources that provide tips and helpful information on how to best create a website and set up a web server to optimize how search engines look at web content and how they index and trigger based upon different types of search keywords,” says Loeffler. Google offers webmasters and designers a plethora of information on how to do just that. The best tip, of course, is to create content that users will want. After that, it’s important to understand that spiders see web pages as text only. “Don’t use images to display important names, content or links,” says Google. “Our crawler doesn’t recognize text contained in graphics.” Any competent web designer will know how to create alternate text messages to accompany images in order to get recognized by spiders, as well as how to use “meta tags” — code within a website that highlights a site’s key words and phrases — judiciously to emphasize a given website’s content.
PAY-PER-CLICK
All of the above information relates to website owners who wish to be found by search engines organically. “There are two main ways to be listed in a search engine,” explains Loeffler. “One is organically, which means a site is naturally crawled by a search engine. The other is by contacting a search engine to participate in a site submission program or sponsored web results program, also known as Pay-Per-Click (PPC).” Organic listings cannot, by their very nature, be controlled, and any company that claims to be able to raise a site’s status within a search engine’s results should be approached with some trepidation. “The goal of a search engine is to match the end-desires of its users,” says Wall. “But there are people out there using automated bots — robots — to submit thousands, or hundreds of thousands, of comments to blogs, for instance. People are manipulating the link graph. And some people partner with premium publishers who sell links. If you stay on-topic, I don’t think it’s too bad. But search engines are getting better at understanding quality links and low-quality links.” Many of the most reputable search engines will actually remove a site from their results if that site is actively ‘spamming’ their link-indexing process. “Don’t participate in link schemes designed to increase your site’s ranking,” says Google. “In particular, avoid links to web spammers or ‘bad neighborhoods’ on the web, as your own ranking may be affected adversely by those links.”
In order to avoid potentially unscrupulous methods of polluting the linking index, many website owners opt for a PPC program. And while the ins and outs of working with PPCs are sometimes complex, the basics are simple. “In PPC advertising, also known as sponsored search advertising, advertisers place bids with search engines for keywords to secure prominent placement in sponsored search listings that appear across the top, side, and bottom of the computer screen,” says Gaude Paez, a spokesperson for Yahoo! Search Marketing. Another example of PPC is Google’s AdWords, as is the Ask Jeeves “Featured Sponsor” program. All of these PPCs offer website owners the opportunity to increase their site traffic without resorting to unethical linking.
But whether a site owner uses design savvy, offline marketing or PPC, there’s no doubt that being at the top of a search result is one of the best ways to help potential customers find what they’re looking for — you.
