Search Engine Optimization: Preparing your pages for the search engines

Before you submit your pages to the search engines, it is crucial that you make sure they are search-engine friendly. Here are some basic tips on what to do.
For more detailed tips on search engine strategies, I highly recommend Search Engine Watch. It is the best resource on the net for learning about search engines.

HelpHelp
Login to Account
Account Name:

Password:


(Or click here to Create New Account)

First time here? Before you begin, you might want to read the Search Engine Submission Tool Tutorial. There is also a comprehensive Help & FAQ page available.

Forgot your account name or password? Click here to cure your amnesia!

Secret Net Tools
To give you a taste of how cool the Secret Net Tools are (you have to contribute to get access to all of them), I've enabled your access to several of the simpler ones. Enjoy!

PrettyPrint
Are You Listed?
License Pl8 Genr8tor

Google (my favorite search engine)

Getting Started
Getting Started:
Welcome (Home Page)
Site Help
Site FAQ
Site Map
List of Search Engines
Create New Account

Tutorials: (read in this order)
Preparing your Pages for the Search Engines
Using Tooter to submit your site
Getting listed in Yahoo (and the other big indexes)
Paying for Traffic (Pay-Per-Click)
Improving your Search Engine Rankings
How to (not) win Awards

Articles:
Trends in Submission
Virtual Vexation
The 500 Sites Fallacy
Page 1 Paranoia
Clickthrough Conundrums
The Gullibility Virus
A Marketing Course that Works?!
Accepting Credit Cards Online
Links of Interest
Glossary of Useful Terms
Random Nonsense
Woodhead's Law
Cool Domain Names
Why I let you decide how much to pay

Moral obligations?
Be Supportive!
Add your Award or Index
Make Money Slow! (by linking to me)
Suggestion Box
Advertising & Sponsorship

About the Site:
What's New?
How it works
Promotion Monitor
Our Awards
Best User Sites
User Comments

Believe it or not, 11.13% of the people who use SelfPromotion.com support the site!
Step 0 - Make sure your site is useful!

I'm always amazed by how many people miss this simple concept. They spend a huge amount of time trying to get good search engine rankings and lots of visitors, and then what do the visitors find? A poorly designed, badly written website. If there's one bit of advice I wish I could drum into everyone who visits this site, it is this:

"It isn't how many 'hits' you get, but how many 'sales' you make."

Making your pages "people-friendly" is as important as making them "search-engine" friendly. It's usually much easier to double the effectiveness of your website than it is to double your traffic, and the bottom line result is the same. At the bottom of this article, I'll give you some advice on how to do that.

But for now, let's talk about what you can do to help out the search engines.

Step 1 - Determine your Key Phrases

People get obsessive about their keywords. This is wrong. It is difficult if not impossible to get high rankings based on keywords. Instead, you need to think about keyphrases.

The easiest way to do this is ask yourself "what would someone trying to find me type in when they search?" Make a list of these. Try them out on the search engines -- pretend to be someone looking for your product or service.

If your business is geographically restricted, then your keyphrases should reflect this. For example, if you are a a real-estate broker in Wilmington, North Carolina, then the key phrase "buying real estate" is a waste of time; instead, the more specific phrase "buying real estate in wilmington north carolina" is what you want to be thinking about.

Think about variations on the key phrases and write them down. Continuing with our example:
The above is only a partial list, but you get the idea. You can also get a good idea of what keyphrases and page design techniques work well by looking at other pages that do well in the searches you've tried. I discuss how to do that in more detail in my Improving Your Search Engine Ranking article. Note that this sample list is just a list of possible keyphrases -- we're not going to use all of them because we won't have room.
One of your fellow users, Stephen Sherman, pointed out an interesting subtlety about keyphrase selection. Let's assume you find a keyphrase that you think people will type in a lot. Try it, and look at the results. If the results seem to be "on topic", then people are likely to drill down several pages to find a listing that is just right for them. This kind of keyphrase is one you want to target, but if you don't get on page one, you'll still get traffic. If, on the other hand, the results are mostly irrelevant (or full of spammed listings), then people will rarely look at page 2, or even more than the first few listings. These keyphrases are thus not as valuable. This doesn't automatically mean you shouldn't try to target it -- none of the criteria are absolutes -- but it does mean that it will be more difficult to get a useful listing with that keyphrase.

Two great resources for finding out what keyphrases are the most effective are the Overture.com Search Suggestions Page and WordTracker. On Overture's search suggestion page, you just type in a very general keyphrase (like "real estate") and it tells you all of the more specific keyphrases that relate to that keyphrase and how many hits they got.

Using our example of selling real estate in Wilmington, NC and entering "real estate" into the Overture tool, I found that in October 1999, the broad keyphrase "real estate" was searched for 67016 times (on Overture). However, you'd never want to target that keyphrase, because you don't care if someone in Kalamazoo wants real estate; you're only interested in people who want houses in Wilmington. Looking down the list of results, I found that "north carolina real estate" got 489 searches. That's a possibility. Doing a search suggestion on "wilmington" revealed several hundred hits on "wilmington north carolina," "wilmington nc" and related topics. Zooming in even further, getting a suggestion on "wilmington real" found 36 hits on specific queries related to real estate in wilmington, NC. These queries obviously don't get a huge amount of traffic, but because they're so focused on what you're trying to do, they're often the best ones to target.
Side note: If you've read my Overture and Adwords tutorial or my page on Search Engine Trends then you know I think pay-per click services like Overture can be worthwhile. The top position on those nice, juicy specific queries on Overture can often be had for 10 cents each. Yeah, you might only get 10 visits a month by sponsoring "wilmington real" but each one would be looking for exactly what you're selling - at a cost of a dime per pre-qualified visitor! Recent changes at Overture have made it a lot less valuable for highly targeted advertisers, however (you used to be able to get those terms for a penny!). Read these other pages for more information.

Those of you who have contributed and gained access to the Secret Net Tools will want to check out the Keyword Susser tool, which will do multiple queries using the Overture.com Search Suggestions system and merge the results. It's a really great way to find out what your keyphrases actually are.

WordTracker goes a bit further. It helps you develop lists of relevant keyphrases, ranked by their popularity. It then queries the major search engines to determine which keyphrases are the least competitive. It's usually not much use targeting a popular keyphrase (lots of searches) if there are millions of other pages that contain that keyphrase. On the other hand, a relevant keyphrase that only gets a few searches a day but which has only a few pages competing for it is a good candidate, because it will be much easier to get a high ranking. WordTracker has a free trial that will give you a lot of information, and additional services available by subscription - including some great tools for working with Overture.
Disclaimer: WordTracker pays me a commission on any income generated from clickthroughs from SelfPromotion.com, but as usual, I donate all such income to the Salvation Army in order to preserve my editorial independence.

My advice is to use the Overture tool to get a rough idea what your keyphrases should be (and find ones you might not have thought about), and then use WordTracker to determine which ones you really should be targeting - and (this is key!) to rank them in order of importance.

OK. At this point, you know what your best keyphrases are. You've got your list. You've checked it twice. Now it's time to use it!

Step 2 - Crafting your <TITLE> tag

Most people make the mistake of using a page title that's good for people but lousy for the search engines. Big mistake. A title like "Bill Phillips - Real Estate Broker" is a disaster! The golden rule is this: All your most important keyphrases should be in the TITLE tag. So what you do is look at your keyphrases, make a list of all the important words, and create a title tag that uses them. Also, keep in mind that browsers only display the first few words of a title tag (whatever fits into the title bar of the window). So while the first sentence of your title tag should be "human readable", the rest can be just a list of keyphrases.

There is some debate as to whether a very long title tag is a good thing or a bad thing when it comes to search engines. Some people are concerned that a very long title tag might result in the search engines deciding the page is a "spam page." I'm waffling on this issue right now. Based on the available (scanty!) evidence, my advice is to keep the title between 15 and 20 words. But you might want to try longer title tags on some of your pages, just to see what happens! So Bill Phillips might have a title that looks like this:

<TITLE>Real Estate in Wilmington, North Carolina - New Hanover County - Buying Selling Renting Houses Homes Apartments Commercial Property Office</TITLE>

The reason for this is that the three most important places to have keyphrases and phrases are your title tag, your meta tags, and your first paragraph. You want them to all contain the same important words; this increases your keyphrase density and improves your rankings.

Step 3 - The Meta Tags

The fabled Meta tags are important to getting good rankings, and on many search engines, the page title (often truncated) and the Meta Description tag are what gets displayed.

Meta tags go in the <HEAD> section of the HTML page (the same section as the <TITLE> tag). The Meta Description tag should contain a short description of the web-page. If you think of your webpage as a news story, then the title tag (the first part of it anyway, not any keywords you tacked on) is the headline, and the meta description tag is the lead paragraph. In many search engines, your search results will simply be your title tag followed by your meta description tag, so make sure they work together to explain what's on the page. The format of a Meta description tag is simple. It looks like this:

<META name="description" content="whatever you want to place here">

So, in our example, we might use:

<META name="description" content="Real Estate in Wilmington, North Carolina - Buying, Selling & Renting of Houses, Homes, Apartments, Commercial Property and Office Space">

My advice on the length of this description is keep it between 100 and 200 characters. Remember: the description tag should be written for humans to read. It should not be a list of keywords, and should be longer (perhaps 50% longer) than your title tag.

The other Meta tag is the Meta Keywords tag. What you do is take your keyphrases, and enter them in the order you think is most appropriate, separated by commas. Don't repeat a keyphrase, and don't repeat any individual word more than 5 times or so. This may mean that you can't use some of your better keyphrases.

The reason why you don't want to repeat any particular word more than 5 times is that some search engines may penalize you for doing this. Search engines aren't as sensitive to keyword repeating as they used to be (most of them ignore extra repeats), but play it safe. The exception is common "noise" words like "the", "in", "a", "and" and so on. Most search engines ignore them. Leave them in, but don't worry if you have more than 5 of any of them.

If you've got a lot of keyphrases that really are relevant to your site, the best thing to do is build "theme" pages devoted to a particular keyphrase or set of keyphrases. This is good for you, good for your visitors, and appreciated by the search engines. Use the most important keyphrases on your homepage.

Some people get confused about whether to use commas between phrases, and whether to capitalize keywords. The truth is, some search engines pay attention to the commas, some don't. But the ones who don't treat them as "white-space". So just use commas as appropriate, but don't waste a character putting a space after the comma. Similarly, just capitalize words as you might expect people to normally use them. Most search engines will ignore the capitalization, but it can't hurt to help out those that make note of it.

In addition, some search engines are sensitive to word order, others just to presence of the keywords on the page. So for some search engines, "buying real estate" is not the same as "real estate buying". This means it is a good idea to word your phrases in the way you think most people are going to type them in.

If you want to get really fancy, play the cunning comma trick. The search engines that don't pay attention to commas sometimes pay attention to sequences of words. So if you can put two keyphrases together with a comma between them, and the last words of the first keyphrase coupled with the first words of the next keyphrase make up one of your keyphrases, then you've gotten 3 keyphrases for the price of two! Normally, however, this is difficult, so don't waste too much time over it.

Keep your keywords meta-tag length between 200-400 characters. Unfortunately, this means you may not be able to include all of your key phrases in your meta keywords tag even if you don't repeat a word too often. The theme pages concept deals with this also. After pruning away, our sample keywords tag might look like this:

<META name="keywords" content="real estate in wilmington north carolina,buying real estate in wilmington north carolina,selling real estate in wilmington north carolina,renting real estate in wilmington north carolina,real estate broker in wilmington north carolina,new hanover county,south-east north carolina,house broker,apartment broker,home sales,apartment rental">

Step 4 - The first paragraph

The first paragraph of your page should recapitulate and expand upon everything in your title and meta tags. You need to have all those keyphrases in it. However, since this is going to be read by people, it needs to be written with them in mind. This is where you introduce yourself to your visitors, so you want to make a good impression.

Try to put this first paragraph as close to the <BODY> tag as possible. Avoid putting graphics or other HTML in front of your first paragraph as much as you can. I don't have a banner ad on my homepage for this reason. Also, use the <H1> or <H2> tag to emphasize your opening sentence (but make sure it looks tasteful!). Bill Phillips might use the following opening paragraph:

<H2>Are you interested in buying, selling or renting real estate in Wilmington, North Carolina?</H2><BR>
If so, you've come to the right place. My name is Bill Phillips, and for the last 10 years, I've specialized in helping my clients find the perfect home, apartment or commercial space in beautiful New Hanover County. Please allow me to be your guide.

Step 5 - Don't Go Overboard - and whatever you do, don't put up spam pages!

You clearly want to have your important keyphrases on your page more than once, because this is what gives the search engines a clue as to what your page is really about. But you don't want your keyphrases to appear too many times, because that might make the search engines think your page is a spam page trying to rank highly for a particular phrase.

The question then becomes, how much is too much? And the answer is, nobody knows for sure, and it's going to be different from search engine to search engine. Rumor has it that Google likes pages with less than 13 repeats of a keyphrase, for example.

My advice is to try and keep the number of repeats of important phrases down to 10 or less; this means all instances, in title, meta tags, and the text of the page. Sometimes this simply isn't possible, because the phrase is so integral to your topic, so don't get paranoid about this. Just keep it in mind.

There are certain classes of sites and pages that the big guys consider spam, and either won't list, or will penalize. The major indexes consider the following kinds of sites to be spam and will not list them: In addition, the major search engines are actively penalizing/banning sites that employ the following techniques: People who repeatedly submit spam sites to the big guys have not only been blacklisted, but in some cases, their previously submitted (and legitimate) sites have been removed. So be nice to the Indexes, and they'll be nice to you. And credit where credit is due: Chris Sherman's SearchDay Newsletter is the place to find out what works -- and what doesn't -- with the search engines. Do yourself a favor and check it out.

Putting it all together

Combining steps 1-5, we get some HTML that looks like this

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html40/loose.dtd">

<HTML>
<HEAD>
<TITLE>Real Estate in Wilmington, North Carolina - New Hanover County - Buying Selling Renting Houses Homes Apartments Commercial Property Office</TITLE>
<META name="description" content="Real Estate in Wilmington, North Carolina - Buying, Selling & Renting of Houses, Homes, Apartments and Commercial Property">
<META name="keywords" content="real estate in wilmington north carolina,buying real estate in wilmington north carolina,selling real estate in wilmington north carolina,renting real estate in wilmington north carolina,real estate broker in wilmington north carolina,new hanover county,south-east north carolina,house broker,apartment broker,home sales,apartment rental"> </HEAD>
<BODY>
<H2>Are you interested in buying, selling or renting real estate in Wilmington, North Carolina?</H2><BR>
If so, you've come to the right place. My name is Bill Phillips, and for the last 10 years, I've specialized in helping my clients find the perfect home, apartment or commercial space in beautiful New Hanover County. Please allow me to be your guide.

... rest of your html
</BODY>
</HTML>

At this point, you're probably wondering "What the heck is that <!DOCTYPE> tag that suddenly appeared?" (if you aren't, then either you already know what it is, or you are not reading this page carefully enough!)

Believe it or not, every page on your site should have a DOCTYPE definition right at the top of the page, before the <HTML> tag. DOCTYPE defines what flavor of HTML your page is written in, and helps browsers display the page correctly. Strictly speaking, if a page doesn't have a DOCTYPE, it isn't valid HTML, though of course browsers will try their best to display it (which is a good thing since 95% of the pages out there don't have them. Heck, as of this writing, even Google.com doesn't have a DOCTYPE, and they of all people ought to know better).

One nice advantage of having a correct DOCTYPE definition is that you can use w3.org's excellent HTML validation tool to check your HTML. This tool is the most horribly picky validator on the planet (and was written by the guys who define what HTML is). For more information on figuring out what your DOCTYPE should be, read W3.org's pages on DOCTYPE.

If this discussion of DOCTYPE is making blood drip from your ears, don't sweat it too much. Not having a DOCTYPE isn't the end of the world. Microsoft forgot to add them. But Apple did, which should tell you something!

Extra Credit!

Now you've got a decent webpage, with good content and meta tags. You're ready to submit to the search engines. Or are you?

What you've done so far are the basics. There's a lot more you can do, time and money permitting, to improve your odds of ranking highly. Here are some tips on what to do (and not to do), to get "extra credit" from the search engines.

Get your own domain name

This is the best investment you'll ever make. A domain name costs $20-30 a year, it's dirt cheap. There are three big reasons for getting your own domain name: I personally use Internet Names WorldWide for my registrations, they are an Australian company with good service and a great user agreement (you own your domain name, not them), though not the cheapest.

This is an absolute no-brainer. Get your own domain name!

Don't get hyped about long domain names

I've been getting a lot of questions about the new, longer domain names that are available. There is a lot of misinformation being passed around about them. The big lie is that if you have a domain name with lots of keywords in it (eg: add-url-register-website-promote-site-selfpromotion.com) you will get a higher ranking in the search engines.

This is flat out not true. NONE of the major search engines will significantly boost your rankings based on keywords in your url. Not one. This is what they said when Danny Sullivan, editor of the highly recommended Search Engine Watch Newsletter asked them, and I've confirmed it by experiment. If the search engines look at them at all, they simply add the url text to the rest of the page, so the added benefit of keywords in the URL is totally insignificant. Don't waste your money.

My advice is to try and go for a short, memorable domain name, either 1 word or 2 words combined, or with an i, e, i- or e- prefix. Make it easy to type, and easy to remember. You can see a list of 1-word domain names that were still available as of early January 2000 here. Maybe one of them will be just right for you.

If you insist on trying the keywords in URL, do it either using subdirectories (eg: http://selfpromotion.com/add-url/register-website/promote-site.html) or subdomains (eg: http://add-url-register-website-promote-site.selfpromotion.com/). You'll still be wasting your time, but at least you won't be wasting money!

Avoid Search Engine Tricks

Some "experts" advise trying to trick search engines by putting keyphrases in comments, putting them in text that is the same color as your background, and so on. I strongly advise that you not try these tricks. Bluntly, most of them don't work -- and the ones that do may stop working at any minute, as the search engines are constantly trying to detect and defeat them.

My philosophy is that you should try and help the search engines by making it as easy as possible to get a good idea of what your page is about. That way, as search engines get better and better at rating the contents of sites, your rankings will get better over time, with no effort from you.

I know this is sort of repeating what I just said a couple of screens higher on this page, but it bears repeating. If you try and fool the search engines, in the long run, you'll be the fool.

Got Links?

Once you have your pages up and running, and chock full of useful content, it's a very good idea to try and get other people to link to them. It's not enough to just get them in the search engines. There are three very good reasons for doing this: First, many search engines are now using link popularity (how many other pages link to your page) as a ranking criteria -- they figure that if other sites link to your page, it might be useful. Second, it's recently been revealed that Inktomi applies a ranking penalty to any url submitted through their free "Add URL" system, but removes it if their spider also finds the page by following a link from another site (they do this in an attempt to find and penalize "doorway" pages). And third, you'll get traffic from the websites that link to you.

Getting links isn't that hard. When you find a website that has content similar to yours, email the webmaster and ask for a link, pointing out why it would be appropriate. If he has content on his site useful to your visitors, link to him without even offering to trade links. Link to him, then email him and ask for a link back. A good site for learning the basics of getting links is Linking 101.

The biggest search engine to use link popularity is Google (who came up with the idea), and the two sites that generate the most link "value" on Google are Yahoo and Open Directory. For many people, the true value of the $299 a year cost of a Yahoo listing isn't the clicks from Yahoo, but the boost in their rankings on Google. Open Directory doesn't cost anything, but getting in can be time consuming. See my Yahoo tips page for more details.

Flash is a plague upon the net

Avoid using Flash in your website. Flash is a black box to the search engines, they can't see inside it. Anything presented by Flash is invisible to them. So that means it is invisible to people trying to find your product or service.

Flash is almost never used appropriately on the net. One of the few examples of good flash use is HowStuffWorks, where it is used to present little explanatory animations.

But time and time again, I see entire sites created in Flash. What's the point? Since the search engines can't see the Flash content, the chances are, these sites won't rank well for the searches they want to rank well. Which means they won't get traffic. Which means almost nobody will see the high-tech super-cool way-keen website that they paid a huge amount of money to a "web design expert" to create.

My advice is simple. Death to Flash. If a consultant recommends that you use Flash in a website, run for the door. If you can trample him or her on the way out, consider that a bonus. Just in case I'm not being clear, let me put it another way. Anyone who recommends extensive use of Flash in a website ought to be taken out and shot.

Javascript suffers from similar problems. It clutters websites, and doesn't work on all browsers. Use it only for absolutely essential functional activities. Never use it to create flashy effects.

A note about Framed sites

Many "experts" also say that using frames to construct your website can hurt their rankings. My experience is that this is not so, as long as you construct your frames properly. The trick is this: make sure that your <frameset> page has a proper title tag and meta tags. Similarly, your subframe pages should have the same ingredients (perhaps with modified contents), as well as a little bit of javascript that "pops" the user to the proper framed presentation if they surf into the subframe page. Here's some sample javascript that works with just about every browser:

<BODY onLoad="if (top == self) top.location.href = 'http://www.yourserver.com/yourframe.html';">

What this does is, when the page is loaded, if it finds that it is not in a frame, it redirects the browser to the proper frameset url, whatever that might be. A much more detailed explanation of how this can be done in various ways can be found in this excellent article.

Once you've got your pages configured, simply promote them all (the frameset page and the subframe pages) to the search engines. The MultiSubmitter tool, which you get access to by forking over $10 or more, makes this easy!

Hiding your JavaScript and CSS

If you are using Javascript or CSS in your pages, you probably stuck it at the top of your pages. Whups! Remember that search engines tend to rate what they find at the top of pages a bit higher -- and what do you have there? Pages of Javascript!

A better way to do things is put the Javascript (or CSS) in a separate file, and include it into your pages with a single tag. Here's an example of a <SCRIPT> tag that includes a file full of Javascript.

<SCRIPT type="text/javascript" src="http://someplace.com/javascript/myscript.js"></SCRIPT>

It takes up much less space, and you can reuse the same code on all your pages (and change it in just one place if you need to). Don't forget the </SCRIPT> tag or the browser might think the rest of your page is also Javascript! Similarly, for CSS, you'd use something like this:

<LINK REL=StyleSheet HREF="http://someplace.com/css/mystylesheet.css" TYPE="text/css">

Both of these tags would go in your <HEAD> section, after your title and meta tags.

Checking your HTML

Almost all websites have HTML errors, even those that appear to display nicely on your browser. Browsers do a pretty good job of being tolerant of errors, but even so, it's a good idea to make sure your HTML is as perfect as possible, as this increases the chance that your website will display the way you want it to on as many browsers as possible (both past, present and future).

I strongly recommend that you use a HTML Validator to check your pages, and my favorite is the W3.org HTML Validator. While it is a little hard to use and unbelievably picky (good!), it's by far the best I've found -- and given that it's written by the folks that define what HTML is, it does the job right. One thing to be aware of with this tool is that a single error can cause a cascade of error messages, so I typically fix the first error reported, then revalidate.

For links to other HTML tools and tutorials, look on my Links of Interest page.

What about a "robots.txt" file?

The robots.txt file is a special file you can place on your webserver to restrict access by some or all webcrawling robots to some or all of your site. You only need one if you want to place some areas of your website "off-limits" to robots. If your whole website is open to them, you don't need one.

You can only have a robots.txt file if you own your own domain, because they are always located in the same place (so the robot can find them!). Thus, my robots.txt file is located at http://selfpromotion.com/robots.txt.

If you feel you need a robots.txt file, then the complete specification how to create one can be found here.

One caution: some robots interpret a blank robots.txt file as meaning "don't crawl any pages on this website." So if you don't need a robots.txt file, don't have one (even a blank one!) on your server.

Meta Tag Software

While I have not personally used it, if you are looking for a Macintosh-based tool to manage the meta tags of many pages, you might want to check out Meta Tag Manager. Windows-based users who have found a good tool, please email me and let me know which ones you like.

Making your site "people-friendly"

OK, you've got all your keywords set up. You've made life easy for the search engines. Hopefully, you'll get a lot of traffic. Make sure you don't waste that traffic by making sure your site effectively sells your products.

The art of creating websites that sell effectively is beyond the scope of this website, but fortunately, I've found a couple of online marketing courses, one quite inexpensive, that do an excellent job of teaching you how to do it. I've reviewed both of them in detail, and both of them helped me make major improvements in this site. You can read about them here.

Graduate School: Shopping Cart Traps

This tip is only relevant if you're running a shopping cart or tracking your users individually as they move through your website. It's a very technical issue, but it nails a lot of advanced website operators, so I thought I would mention it. This problem does not apply to 99% of the people reading this page, so if it sounds like it's written in martian, don't worry about it.

Here's the problem in a nutshell. A visitor arrives at your site, www.coolstuff.com, and your super-duper advanced web-commerce server redirects them to a url that encodes an individual user context, either a url with a ? in it (eg: www.coolstuff.com/?CART=12345678) or a url with the users shopping cart ID encoded into it (eg: www.coolstuff.com/12345678/). The classic example of this kind of thing is Amazon.com; type "http://www.amazon.com" into your browser and you'll end up at someplace like "http://www.amazon.com/exec/obidos/subst/home/redirect.html/002-3104337-1035221"

These kind of urls give search engines apoplexy, and the most usual response is to not index them at all.

Another common trick to maintain a user context is to use cookies, in particular what I call the "Stupid ASP trick" of redirecting to the same webpage to set a cookie (so named because it's a standard trick used by Microsoft's ASP software). Search engines ignore cookies, so it doesn't work.

Individual user contexts cause huge problems for search engines, and they can cause huge problems for you as well. A whacked-out web spider going through your whole catalog adding all 327,123 of your products to a single shopping cart can really make your day exciting.

The solution is simple: (1) don't create an individual user context until you need to; don't create a shopping cart until the first item is ordered. (2) make sure all actions that create or change an individual user context are hidden behind POST forms -- because search engines don't follow such links.

In some cases, the software you are using doesn't allow you to do this. If so, consider running two versions of your site, the public www.coolstuff.com, and the "real" webcommerce version, "sales.coolstuff.com". The sites are identical, except that all the "buy" buttons from www.coolstuff.com go to sales.coolstuff.com, and there are no links back. So people who need a shopping cart shift from the public site to the sales site, but the search engines stay on the public side. Everyone's happy. PS: make sure that sales.coolstuff.com only handles requests that come with a referer from www.coolstuff.com!

Whew, that's enough technical stuff for now!

Next step on the site tour: Tutorial on how to use the search engine submission engine


If you would like to reprint this article in your online or paper newsletter, please contact me for permission.

About this site...
This site was developed on a Macintosh, programmed in WebSiphon, and served by WebStar. The author, on those exceedingly rare occasions when he does think, indeed thinks differently.

It looks (and works) better when you use version 4.0 or better of Netscape Navigator or Internet Explorer, but can be used with any tables-capable browser.

SelfPromotion.com ©1997-2005 Robert Woodhead, All Rights Reserved. SelfPromotion.com™, Tooter™, Secret Net Tools™, MultiSubmitter™, Rankulator™, BaldSpotCam™, and ShareService™ are trademarks of Robert Woodhead. To get help, report bugs, or make suggestions, go here.