Cookies — the small text files that a Web server can store on a site visitor’s hard drive — can often prove as indigestible to search engines as dynamic Web pages and session IDs. Imagine this scenario: You visit a site that is using cookies, and at some point, the server decides to store a cookie. It sends the
information to your browser, and the browser duly creates a text file, which it then stores on your computer’s hard drive. This text file might contain a ses
sion ID (as discussed in the preceding section) or something else. Systems that remember who you are so you don’t have to log in each time you visit the site are using cookies to do this.
Cookies are sometimes used for navigation purposes. For instance, you may have seen crumb trails, a series of links showing where you have been as you travel through the site. Crumb trails look something like this:
Home->Rodents->Rats->Racing
This is generally information being stored in a cookie and is read each time you load a new page. Or the server may read the cookie to determine how many times you’ve visited the site or what you did the last time you were on the site, and direct you to a particular page based on that information.
If you’re using Internet Explorer on Microsoft Windows, follow these steps to see what these cookie files look like:
1. Choose Tools➪Internet Options from the main menu.
The Internet Options dialog box appears.
2. In the Internet Options dialog box, make sure that the General tab is selected.
3. Click the Settings button in the Temporary Internet Files area.
The Settings dialog box appears.
4. In the Settings dialog box, click the View Files button.
A Windows Explorer window opens, displaying the directory containing your temporary files, including cookies. The files are named Cookies:
username@domainname.com. For instance, when I visited the nokiausa.
com Web site, the server created a cookie called Cookie:peter kent@
nokiausa.com.
5. Double-click any of these cookie files to view the file’s contents; a warning message appears, but ignore it and click Yes.
The cookie opens in Notepad.
Here are the contents of the cookie that was set by nokiausa.com; this cookie contains a session ID:
session_idA_da6Bj6pwoLg=_nokiausa.com/_1536_1240627200_
30394925_36557520_29597255_*
There’s nothing wrong with using cookies, unless they’re required in order to navigate through your site. A server can be set up to simply refuse to send a Web page to a site visitor if the visitor’s browser doesn’t accept cookies.
That’s a problem for several reasons:
A few browsers simply don’t accept cookies.
A small number of people have changed their browser settings to refuse to accept cookies.
Searchbots can’t accept cookies.
If your Web site demands the use of cookies, you won’t get indexed. That’s all there is to it! The searchbot will request a page, your server will try to set a cookie, and the searchbot won’t be able to accept it. The server won’t send the page, so the searchbot won’t index it.
How can you check to see if your site has this problem? Change your browser’s cookies setting and see if you can travel through the Web site.
Here’s how (for Internet Explorer):
1. Choose Tools➪Internet Options from the main menu.
The Internet Options dialog box appears.
2. In the Internet Options dialog box, click the Privacy tab.
3. On the Privacy tab, click the Advanced button.
The Advanced Privacy Settings dialog box appears, as shown in Figure 6-8.
4. Select the Override Automatic Cookie Handling check box — if it’s not already selected.
5. Select both of the Prompt option buttons.
You want to get prompts from both first-party cookies and third-party cookies. I recommend that you select Prompt rather than Block to make it easier to test your site. Now each time a server tries to set a cookie, you see a warning box, and you can accept or block the cookie at will.
6. Click OK to close the Advanced Privacy Settings dialog box.
7. In the Internet Options dialog box, click the General tab.
8. On the General tab, click the Delete Cookies button. Note that some sites won’t recognize you when you revisit them, until you log in again and they reset their cookies.
9. Click the OK button in the confirmation-message box.
10. Click the OK button to close the dialog box.
Figure 6-8:
The Advanced Privacy Settings dialog box.
Select these two option buttons.
Now go to your Web site and see what happens. Each time the site tries to set a cookie, you see a message box, as shown in Figure 6-9. Block the cookie and then see if you can still travel around the site. If you can’t, the searchbots can’t navigate it either.
Figure 6-9:
The Privacy Alert dialog box.
How do you fix this problem?
Don’t require cookies. Ask your site programmers to find some other way to handle what you’re doing with cookies, or do without the fancy navigation trick.
As with session IDs, you can use a User-Agent script that treats search-bots differently. If the server sees a normal visitor, it requires cookies; if it’s a searchbot, it doesn’t.