Helping webmasters from user to user

You have to have some kind of super-powers to keep up with all of the issues posted in our Webmaster Help Forum—that's why we call our Top Contributors the "Bionic Posters." They're able to leap through tall questions in a single bound, providing helpful and solid information all around. We're thankful to the Bionics for tackling problems both hard and easy (well, easy if you know how). Our current Bionic Posters are: Webado (Christina), Phil Payne, Red Cardinal (Richard), Shades1 (Louis), Autocrat, Tim Abracadabra, Aaron, Cristina, Robbo, John, Becky Sharpe, Sasch, BbDeath, Beussery (Brian), Chibcha (Terry), Luzie (Herbert), 奥宁 (Andy), Ashley, Kaleh and Redleg!

With thousands of webmasters visiting the English Help Forum every day, some questions naturally pop up more often than others. To help catch these common issues, the Bionic Posters have also helped to create and maintain a comprehensive list of frequently asked questions and their answers. These FAQs cover everything from "Why isn't my site indexed?" to diagnosing difficult issues with the help of Google Webmaster Tools, often referring to our Webmaster Help Center for specific topics. Before you post in the forum, make sure you've read through these resources and do a quick search in the forum; chances are high that your question has been answered there already.

Besides the Bionic Posters, we're lucky to have a number of very active and helpful users in the forum, such as: squibble, Lysis, yasir, Steven Lockey, seo101, RickyD, MartinJ and many more. Thank you all for making this community so captivating and—most of the time—friendly.

Here are just a few (well, a little more than a few) of the many comments that we've seen posted in the forum:

• "Thank you for this forum... Thank you to those that take the time to answer and care!"
• "I've only posted one question here, but have received a wealth of knowledge by reading tons of posts and answers. The time you experts put into helping people with their problems is very inspiring and my hat's off to each of you. Anyway, I just wanted to let you know that your services aren't going unnoticed and I truly appreciate the lessons."
• "Thank you very much cristina, what you told me has done the trick. I really appriciate the help as this has been bugging me for a while now and I didn't know what was wrong."
• "thank you ssssssssssoooo much kaleh. "
• "OK, Phil Payne big thanks to You! I have made changes and maybe people are starting to find me in G! Thanks to Ashley, I've started to make exclusive and relevant content for people."
• "If anything, it has helped me reflect on the sites and projects of days gone by so as to see what I could have done better - so that I can deliver that much more and better results going forward. I've learned that some things I had done right, were spot on, and other issues could have been handled differently, as well as a host of technical information that I've stored away for future use. Bottom Line: this forum rocks and is incredibly helpful."
• "I asked a handful of questions, got GREAT help while doing a whole lot of lurking, and now I've got a site that rocks!! (...) Huge thanks to all the Top Contributors, and a very special mention to WEBADO, who helped me a TON with my .htaccess file."
• "Over the years of reading (and sometimes contributing) to this forum I think it has helped to remove many false assumptions and doubts over Google's ranking systems. Contrary to what many have said I verily believe Google can benefit small businesses. Keep up the good work. "
• "The forum members are awesome and are a most impressive bunch. Their contribution is immeasurable as it is huge. Not only have they helped Google in their success as a profitable business entity, but also helped webmasters both aspiring and experienced. There is also an engender(ment) of "family" or "belonging" in the group that has transcended the best and worst of times (Current forum change still TBD :-) ). We can agree, disagree and agree to disagree but remain respectful and civil (Usually :-) )."
• "Hi Redleg, Thank you very much for all of the information. Without your help, I don't think I would ever have known how to find the problem. "
• "What an amazing board. Over the last few days I have asked 1 question and recieved a ton of advice mainly from Autocrat. "
• "A big thank you to the forum and the contributors that helped me get my site on Google . After some hassle with my web hosters and their naff submission service, issues over adding pages Google can see, issues over Sitemaps, I can now say that when I put my site name into the search and when i put in [custom made watch box], for instance, my site now comes up."
• "Thank you Autocrat! You are MAGNIFICENT! (...) I am your biggest fan today. : ) Imagine Joe Cocker singing With a Little Help from My Friends...that's my theme song today."
• "I've done a lot of reading since then and I've learned more in the last year than I learned in the previous 10. When I stumbled into this forum I had no idea what I was getting into but finding this forum was a gift from God! Words cannot express the amount of gratitude I feel for the help you have given me and I wish I could repay you some how.... I don't mean to sound so mushy, but I write this with tears in my eyes and I am truly, truly grateful..."

Are you new to the Webmaster Help Forum? Tell us a little bit about yourself and then join us to learn more and help others!



Dj Matioka

Answering your December Grab Bag questions

Webmaster Level: All

You asked and Matt Cutts answered. It's time to answer the latest round of Grab Bag questions! Here's the first answer, complete with Matt's new hairstyle:

We have a lot of videos ready to share, so we're not currently taking new questions for the Grab Bag. If you have a question that you would like answered, your best bet as always is to head to our Webmaster Help Forum, where you'll find plenty of knowledgeable webmasters, including some Bionic Posters!

To be kept up-to-date on our latest video releases, you can follow @googlewmc on Twitter, where we'll announce new videos and blog posts as they're published.

Dj Matioka

Test your webmaster know-how!

Webmaster Level: All

We thought it might be fun and educational to create a quiz for webmasters about issues we commonly see in the Webmaster Help Forum. Together with our awesome Bionic Posters, we've tried to come up with questions and answers that reflect recurring concerns in the forum and some information that may not be well known. Some things to keep in mind when taking this quiz:

• The quiz will be available to take from today until Wednesday, January 27 at 5PM PST.
• It doesn't cover all facets of webmaster problems that arise, and—as with any test—it is at best only a fun way to test your webmaster prowess ;). We leave discussion of specific cases to the forum.
• We've set up the quiz using our very own Google Docs. This means you won't see results right away, but we plan to write a follow-up blog post explaining answers and listing top scorers. Be sure to save your answers or print out your completed quiz before submitting! This way you can check your answers against the correct ones when we publish them.
• It's just for fun!

Now go take the quiz!

Dj Matioka

State of the Index 2009

Webmaster Level: All

At PubCon in Las Vegas in November 2009, I gave a "State of the Index" talk which covers what Google has done for users, web developers, and webmasters in the last year. I recently recreated it on video for those of you who didn't make it to the conference. You can watch it below:

And here are the slides if you'd like to follow along:

Dj Matioka

Google SEO resources for beginners

Webmaster Level: Beginner

Want to eat healthier and exercise more in 2010? That's tough! Want to learn about search engine optimization (SEO) so you can disregard the rumors and know what's important? That's easy! Here's how to gain SEO knowledge as you go about your new start to 2010:

Step 1: Absorb the basics

• If you like to learn by reading, download our SEO Starter Guide for reading while you're on an exercise bike, training for Ironman.
• Or, if you're more a video watcher, try listening to my "Search Friendly Development" session while you're cleaning your house. Keep in mind that some parts of the presentation are a little more technical.

• For good measure, and because at some point you'll hear references to them, check out our webmaster guidelines for yourself.

Step 2: Explore details that pique your interest

Are you done with the basics but now you have some questions? Good for you! Try researching a particular topic in our Webmaster Help Center. For example, do you want more information about crawling and indexing or understanding what links are all about?

Step 3: Verify ownership of your site in Webmaster Tools

It takes a little bit of skill, but we have tons of help for verification. Once you verify ownership of your site (i.e., signal to Google that you're the owner), you can:

• See more confidential information about your site, like crawl errors or messages from Google about your site
• Run through the Webmaster Tools checklist and learn new ways to monitor and manage your site
• Forward your Google messages to your personal email address

A sample message regarding the crawlability of your site

Step 4: Research before you do anything drastic

Usually the basics (e.g., good content/service and a crawlable site with indexable information) are the necessities for SEO. You may hear or read differently, but before you do anything drastic on your site such as robots.txt disallow'ing all of your directories or revamping your entire site architecture, please try:

• Researching material in our Help Center
• Checking for related posts in our Webmaster Forum
• Asking for feedback from the webmaster community -- we've got a super skilled group of Google employees, members, and Bionic Posters.
• 
  
• Dj Matioka

Introducing a new Rich Snippets format: Events

Webmaster Level: All

Last year we introduced Rich Snippets, a new feature that makes it possible to surface structured data from your pages on Google's search results. So far, user reaction to Rich Snippets has been enthusiastic -- after all, Rich Snippets help people make more informed clicks and find what they need even faster.

We originally introduced Rich Snippets with two formats: reviews and people. Later in the year we added support for marking up video information which is used to improve Video Search. Today, we're excited to kick off the new year by adding support for events.

Events markup is based off of the hCalendar microformat. Here's an example of what the new events Rich Snippets will look like:

The new format shows links to specific events on the page along with dates and locations. It provides a fast and convenient way for users to determine if a page has events they may be interested in.

If you have event listings on your site, we encourage you to review the events documentation we've prepared to help you get started. Please note, however, that marking up your content is not a guarantee that Rich Snippets will show for your site. Just as we did for previous formats, we will take a gradual approach to incorporating the new event snippets to ensure a great user experience along the way.

Stay tuned for more developments in Rich Snippets throughout the year!


Dj Matioka

Protect your site from spammers with reCAPTCHA

Webmaster Level: All

If you allow users to publish content on your website, from leaving comments to creating user profiles, you’ll likely see spammers attempt to take advantage of these mechanisms to generate traffic to their own sites. Having this spammy content on your site isn't fun for anyone. Users may be subjected to annoying advertisements directing them to low-quality or dangerous sites containing scams or malware. And you as a webmaster may be hosting content that violates a search engine's quality guidelines, which can harm your site's standing in search results.

There are ways to handle this abuse, such as moderating comments and reviewing new user accounts, but there is often so much spam created that it can become impossible to keep up with. Spam can easily get to this unmanageable level because most spam isn’t created manually by a human spammer. Instead, spammers use computer programs called “bots” to automatically fill out web forms to create spam, and these bots can generate spam much faster than a human can review it.

To level the playing field, you can take steps to make sure that only humans can interact with potentially spammable features of your website. One way to determine which of your visitors are human is by using a CAPTCHA , which stands for "completely automated public Turing test to tell computers and humans apart." A typical CAPTCHA contains an image of distorted letters which humans can read, but are not easily understood by computers. Here's an example:

You can easily take advantage of this technology on your own site by using reCAPTCHA, a free service owned by Google. One unique aspect of reCAPTCHA is that data collected from the service is used to improve the process of scanning text, such as from books or newspapers. By using reCAPTCHA, you're not only protecting your site from spammers; you're helping to digitize the world's books.

Luis Von Ahn, one reCAPTCHA's co-founders, gives more details about how the service works in the video below:

If you’d like to implement reCAPTCHA for free on your own site, you can sign up here. Plugins are available for easy installation on popular applications and programming environments such as WordPress and PHP.

Dj Matioka

Request visitors' permission before installing software

(Cross-posted on the Google Korea Blog)

Webmaster Level: All

Legitimate websites may require that their visitors install software. These sites often do so to provide their users with additional functionality beyond what's available in standard web browsers, like viewing a special type of document. Please note, however, that if your site requires specific software for your visitors, the implementation of this software installation process is very important. Incorrect implementation can appear as though you're installing malware, triggering our malware detection filters, and resulting in your site being labeled with a 'This site may harm your computer' malware warning in our search results.

If using your site requires a special software install, you need to first inform visitors why they need to install additional software. Here are two bad examples and one good example of how to handle the situation of a new visitor to such a site:

Bad: Install the required software without giving the visitor a chance to choose whether or not they want to install the software.

Bad: Pop up a confirmation dialog box that prompts the visitor to agree to install the software, without providing enough detail for the visitor to make an informed choice. (This includes the standard ActiveX control installation dialog box, since it doesn't contain enough meaningful information for a visitor to make an informed decision about that particular piece of software.)

Good: Redirect the new visitor to an information page which provides thorough details on why a special software installation is required to use the site. From this page the visitor can initiate the installation of the required software if they decide to proceed with installation.

Has your site been labeled with a malware warning in our search results due to a poorly implemented software installation requirement? Updating the installation process to ensure that visitors are fully informed on why the installation is necessary, and giving them a chance to opt out, should resolve this issue. Once you've got this in place, you can go to Webmaster Tools and request a malware review to expedite the process of removing any malware warnings associated with your site in Google's search results.


Dj Matioka

Running desktop and mobile versions of your site

(This post was largely translated from our Japanese version of the Webmaster Central Blog )

Recently I introduced several methods to ensure your mobile site is properly indexed by Google. Today I'd like to share information useful for webmasters who manage both desktop and mobile phone versions of a site.

One of the most common problems for webmasters who run both mobile and desktop versions of a site is that the mobile version of the site appears for users on a desktop computer, or that the desktop version of the site appears when someone finds them from a mobile device. In dealing with this scenario, here are two viable options:

Redirect mobile users to the correct version

When a mobile user or crawler (like Googlebot-Mobile) accesses the desktop version of a URL, you can redirect them to the corresponding mobile version of the same page. Google notices the relationship between the two versions of the URL and displays the standard version for searches from desktops and the mobile version for mobile searches.

If you redirect users, please make sure that the content on the corresponding mobile/desktop URL matches as closely as possible. For example, if you run a shopping site and there's an access from a mobile phone to a desktop-version URL, make sure that the user is redirected to the mobile version of the page for the same product, and not to the homepage of the mobile version of the site. We occasionally find sites using this kind of redirect in an attempt to boost their search rankings, but this practice only results in a negative user experience, and so should be avoided at all costs.

On the other hand, when there's an access to a mobile-version URL from a desktop browser or by our web crawler, Googlebot, it's not necessary to redirect them to the desktop-version. For instance, Google doesn't automatically redirect desktop users from their mobile site to their desktop site, instead they include a link on the mobile-version page to the desktop version. These links are especially helpful when a mobile site doesn't provide the full functionality of the desktop version -- users can easily navigate to the desktop-version if they prefer.

Switch content based on User-agent

Some sites have the same URL for both desktop and mobile content, but change their format according to User-agent. In other words, both mobile users and desktop users access the same URL (i.e. no redirects), but the content/format changes slightly according to the User-agent. In this case, the same URL will appear for both mobile search and desktop search, and desktop users can see a desktop version of the content while mobile users can see a mobile version of the content.

However, note that if you fail to configure your site correctly, your site could be considered to be cloaking, which can lead to your site disappearing from our search results. Cloaking refers to an attempt to boost search result rankings by serving different content to Googlebot than to regular users. This causes problems such as less relevant results (pages appear in search results even though their content is actually unrelated to what users see/want), so we take cloaking very seriously.

So what does "the page that the user sees" mean if you provide both versions with a URL? As I mentioned in the previous post, Google uses "Googlebot" for web search and "Googlebot-Mobile" for mobile search. To remain within our guidelines, you should serve the same content to Googlebot as a typical desktop user would see, and the same content to Googlebot-Mobile as you would to the browser on a typical mobile device. It's fine if the contents for Googlebot are different from the one for Googlebot-Mobile.

One example of how you could be unintentionally detected for cloaking is if your site returns a message like "Please access from mobile phones" to desktop browsers, but then returns a full mobile version to both crawlers (so Googlebot receives the mobile version). In this case, the page which web search users see (e.g. "Please access from mobile phones") is different from the page which Googlebot crawls (e.g. "Welcome to my site"). Again, we detect cloaking because we want to serve users the same relevant content that Googlebot or Googlebot-Mobile crawled.

Diagram of serving content from your mobile-enabled site

We're working on a daily basis to improve search results and solve problems, but because the relationship between PC and mobile versions of a web site can be nuanced, we appreciate the cooperation of webmasters. Your help will result in more mobile content being indexed by Google, improving the search results provided to users. Thank you for your cooperation in improving the mobile search user experience.


Dj Matioka

'New software version' notifications for your site

Webmaster level: All

One of the great things about working at Google is that we get to take advantage of an enormous amount of computing power to do some really cool things. One idea we tried out was to let webmasters know about their potentially hackable websites. The initial effort was successful enough that we thought we would take it one step further by expanding our efforts to cover other types of web applications—for example, more content management systems (CMSs), forum/bulletin-board applications, stat-trackers, and so on.

This time, however, our goal is not just to isolate vulnerable or hackable software packages, but to also notify webmasters about newer versions of the software packages or plugins they're running on their website. For example, there might be a Drupal module or Joomla extension update available but some folks might not have upgraded. There are a few reasons a webmaster might not upgrade to the newer version and one of the reasons could be that they just don't know a new version exists. This is where we think we can help. We hope to let webmasters know about new versions of their software by sending them a message via Webmaster Tools. This way they can make an informed decision about whether or not they would like to upgrade.

One of the ways we identify sites to notify is by parsing source code of web pages that we crawl. For example, WordPress and other CMS applications include a generator meta tag that specifies the version number. This has proven to be tremendously helpful in our efforts to notify webmasters. So if you're a software developer, and would like us to help you notify your users about newer versions of your software, a great way to start would be to include a generator meta tag that tells the version number of your software. If you're a plugin or a widget developer, including a version number in the source you provide to your users is a great way to help too.

We've seen divided opinions over time about whether it's a good security practice to include a version number in source code, because it lets hackers or worm writers know that the website might be vulnerable to a particular type of exploit. But as Matt Mullenweg pointed out, "Where [a worm writer's] 1.0 might have checked for version numbers, 2.0 just tests [a website's] capabilities...". Meanwhile, the advantage of a version number is that it can help alert site owners when they need to update their site. In the end, we tend to think that including a version number can do more good than harm.

We plan to begin sending out the first of these messages soon and hope that webmasters find them useful! If you have any questions or feedback, feel free to comment here.


Dj Matioka

Hard facts about comment spam

Webmaster Level: Beginner

It has probably happened to you: you're reading articles or watching videos on the web, and you come across some unrelated, gibberish comments. You may wonder what this is all about. Some webmasters abuse other sites by exploiting their comment fields, posting tons of links that point back to the poster's site in an attempt to boost their site's ranking. Others might tweak this approach a bit by posting a generic comment (like "Nice site!") with a commercial user name linking to their site.

Why is it bad?

FACT: Abusing comment fields of innocent sites is a bad and risky way of getting links to your site. If you choose to do so, you are tarnishing other people's hard work and lowering the quality of the web, transforming a potentially good resource of additional information into a list of nonsense keywords.

FACT: Comment spammers are often trying to improve their site's organic search ranking by creating dubious inbound links to their site. Google has an understanding of the link graph of the web, and has algorithmic ways of discovering those alterations and tackling them. At best, a link spammer might spend hours doing spammy linkdrops which would count for little or nothing because Google is pretty good at devaluing these types of links. Think of all the more productive things one could do with that time and energy that would provide much more value for one's site in the long run.

Promote your site without comment spam

If you want to improve your site's visibility in the search results, spamming comments is definitely not the way to go. Instead, think about whether your site offers what people are looking for, such as useful information and tools.

FACT: Having original and useful content and making your site search engine friendly is the best strategy for better ranking. With an appealing site, you'll be recognized by the web community as a reliable source and links to your site will build naturally.

Moreover, Google provides a list of advice in order to improve the crawlability and indexability of your site. Check out our Search Engine Optimization Starter Guide.

What can I do to avoid spam on my site?

Comments can be a really good source of information and an efficient way of engaging a site's users in discussions. This valuable content should not be replaced by gibberish nonsense keywords and links. For this reason there are many ways of securing your application and disincentivizing spammers.

• Disallow anonymous posting.
• Use CAPTCHAs and other methods to prevent automated comment spamming.
• Turn on comment moderation.
• Use the "nofollow" attribute for links in the comment field.
• Disallow hyperlinks in comments.
• Block comment pages using robots.txt or meta tags.

For detailed information about these topics, check out our Help Center document on comment spam.

My site is full of comment spam, what should I do?

It's never too late! Don't let spammers ruin the experience for others. Adopt security measures discussed above to stop the spam activity, then invest some time to clean up the spammy comments and ban the spammers from your site. Depending on you site's system, you may be able to save time by banning spammers and removing their comments all at once, rather than one by one.

If I spammed comment fields of third party sites, what should I do?

If you used this approach in the past and you want to solve this issue, you should have a look at your incoming links in Webmaster Tools. To do so, go to the Your site on the web section and click on Links to your site. If you see suspicious links coming from blogs or other platforms allowing comments, you should check these URLs. If you see a spammy link you created, try to delete it, else contact the webmaster to ask to remove the link. Once you've cleared the spammy inbound links you made, you can file a reconsideration request.

For more information about this topic and to discuss it with others, join us in the Webmaster Help Forum. (But don't leave spammy comments!)


Dj Matioka

Changes in First Click Free

Webmaster level: Intermediate

We love helping publishers make their content available to large groups of readers, and working on ways to make the world's information useful and accessible through our search results. At the same time, we're also aware of the fact that creating high-quality content is not easy and, in many cases, expensive. This is one of the reasons why we initially launched First Click Free for Google News and Google Web Search -- to allow publishers to sell access to their content in general while still allowing users to find it through our search results.

While we're happy to see that a number of publishers are already using First Click Free, we've found that some who might try it are worried about people abusing the spirit of First Click Free to access almost all of their content. As most users are generally happy to be able to access just a few pages from these premium content providers, we've decided to allow publishers to limit the number of accesses under the First Click Free policy to five free accesses per user each day. This change applies to both Google News publishers as well as websites indexed in Google's Web Search. We hope that this encourages even more publishers to open up more content to users around the world!

Questions and answers about First Click Free

Q: Do the rest of the old guidelines still apply?
A: Yes, please check the guidelines for Google News as well as the guidelines for Web Search and the associated blog post for more information.

Q: Can I apply First Click Free to only a section of my site / only for Google News (or only for Web Search)?
A: Sure! Just make sure that both Googlebot and users from the appropriate search results can view the content as required. Keep in mind that showing Googlebot the full content of a page while showing users a registration page would be considered cloaking.

Q: Do I have to sign up to use First Click Free?
A: Please let us know about your decision to use First Click Free if you are using it for Google News. There's no need to inform us of the First Click Free status for Google Web Search.

Q: What is the preferred way to count a user's accesses?
A: Since there are many different site architectures, we believe it's best to leave this up to the publisher to decide.

(Please see our related blog post for more information on First Click Free for Google News.)


Dj Matioka

Region Tags in Google Search Results

Webmaster Level: All

Country-code top-level domains (or ccTLDs) can provide people with a quick and valuable clue about the location of a website—for example, ".fr" for France or ".co.jp" for Japan. However, for certain top level domains like .com, .info and .org, it's not as easy to figure out the location. That's why today we're adding region information supplied by webmasters to the green address line on some Google search results.

This feature is easiest to explain through an example. Let's say you've heard about a boxing club in Canada called "Capital City Boxing." You try a search for [capital city boxing] to find out more, but it's hard to tell which result is the one you're looking for. Here's a screen shot:

None of the results provide any location information in the title or snippet, nor do they have a regional TLD (such as .ca for Canada). The only way to find the result you're looking for is to refine your search ([capital city boxing canada] works) or click through the various links to figure it out. Clicking through the first result reveals that there's apparently another "Capital City Boxing" club in Alabama.

Region tags improve search results by providing valuable information about website location right in the green URL line. Continuing our prior example, here's a screen shot of the new region tag (circled in red):

As you can see, the fourth result now includes the region name "Canada" after the green URL, so you can immediately tell that this result relates to the boxing club in Canada. With the new display, you no longer need to refine your search or click through the results to figure out which page is the one you're looking for. In general, our hope is that these region tags will help searchers more quickly identify which results are most relevant to their queries.

As a webmaster, you can control how this feature works by adjusting your Geographic Targeting settings. Log in to Webmaster Tools and choose Site configuration > Settings > Geographic Target. From here you can associate a particular country/region with your site. These settings will determine the name that appears as a region tag. You can learn more about using the Geographic Target tool in a prior blog post and in our Help Center.

We currently show region tags only for certain domains such as .com and .net where the location information would otherwise be unclear. We don't show region tags for results on domains like .br for Brazil, because the location is already implied by the green URL line in our default display. In addition, we only display region tags when the region supplied by the site owner is different from the domain where the search was entered. For example, if you do a search from the Singapore Google domain (google.com.sg), we won't show you region tags for all the websites webmasters have targeted to Singapore because we'd end up tagging too many results, and the tag is really most relevant for foreign regions. For the initial release, we anticipate roughly 1% of search results pages will include webpages with a region tag.

We hope you'll find this new feature useful, and we welcome your feedback.


Dj Matioka

New User Agent for News

Webmaster Level: Intermediate

Today we are announcing a new user agent for robots.txt called Googlebot-News that gives publishers even more control over their content. In case you haven't heard of robots.txt, it's a web-wide standard that has been in use since 1994 and which has support from all major search engines and well-behaved "robots" that process the web. When a search engine checks whether it has permission to crawl and index a web page, the "check if we're allowed to crawl this page" mechanism is robots.txt.

Publishers could easily contact us via a form if they didn't want to be included in Google News but did want to be in Google's web search index. Now, publishers can manage their content in Google News in an even more automated way. Site owners can just add Googlebot-News specific directives to their robots.txt file. Similar to the Googlebot and Googlebot-Image user agents, the new Googlebot-News user agent can be used to specify which pages of a website should be crawled and ultimately appear in Google News.

Here are a few examples for publishers:

Include pages in both Google web search and News:
User-agent: Googlebot
Disallow:

This is the easiest case. In fact, a robots.txt file is not even required for this case.

Include pages in Google web search, but not in News:
User-agent: Googlebot
Disallow:

User-agent: Googlebot-News
Disallow: /

This robots.txt file says that no files are disallowed from Google's general web crawler, called Googlebot, but the user agent "Googlebot-News" is blocked from all files on the website.

Include pages in Google News, but not Google web search:
User-agent: Googlebot
Disallow: /

User-agent: Googlebot-News
Disallow:

When parsing a robots.txt file, Google obeys the most specific directive. The first two lines tell us that Googlebot (the user agent for Google's web index) is blocked from crawling any pages from the site. The next directive, which applies to the more specific user agent for Google News, overrides the blocking of Googlebot and gives permission for Google News to crawl pages from the website.

Block different sets of pages from Google web search and Google News:
User-agent: Googlebot
Disallow: /latest_news

User-agent: Googlebot-News
Disallow: /archives

The pages blocked from Google web search and Google News can be controlled independently. This robots.txt file blocks recent news articles (URLs in the /latest_news folder) from Google web search, but allows them to appear on Google News. Conversely, it blocks premium content (URLs in the /archives folder) from Google News, but allows them to appear in Google web search.

Stop Google web search and Google News from crawling pages:
User-agent: Googlebot
Disallow: /

This robots.txt file tells Google that Googlebot, the user agent for our web search crawler, should not crawl any pages from the site. Because no specific directive for Googlebot-News is given, our News search will abide by the general guidance for Googlebot and will not crawl pages for Google News.

For some queries, we display results from Google News in a discrete box or section on the web search results page, along with our regular web search results. We sometimes do this for Images, Videos, Maps, and Products, too. This is known as Universal search results. Since Google News powers Universal "News" search results, if you block the Googlebot-News user agent then your site's news stories won't be included in Universal search results.

We are currently testing our support for the new user agent. If you see any problems please let us know. Note that it is possible for Google to return a link to a page in some situations even when we didn't crawl that page. If you'd like to read more about robots.txt, we provide additional documentation on our website. We hope webmasters will enjoy the flexibility and easier management that the Googlebot-News user agent provides.


Dj Matioka

How fast is your site?

We've just launched Site Performance, an experimental feature in Webmaster Tools that shows you information about the speed of your site and suggestions for making it faster.

This is a small step in our larger effort to make the web faster. Studies have repeatedly shown that speeding up your site leads to increased user retention and activity, higher revenue and lower costs. Towards the goal of making every webpage load as fast as flipping the pages of a magazine, we have provided articles on best practices, active discussion forums and many tools to diagnose and fix speed issues.

Now we bring data and statistics specifically applicable to your site. On Site Performance, you'll find how fast your pages load, how they've fared over time, how your site's load time compares to that of other sites, examples of specific pages and their actual page load times, and Page Speed suggestions that can help reduce user-perceived latency. Our goal is to bring you specific and actionable speed information backed by data, so stay tuned for more of this in the future.



The load time data is derived from aggregated information sent by users of your site who have installed the Google Toolbar and opted-in to its enhanced features. We only show the performance charts and tables when there's enough data, so not all of them may be shown if your site has little traffic. The data currently represents a global average; a specific user may experience your site faster or slower than the average depending on their location and network conditions.

This is a Labs product that is still in development. We hope you find it useful. Please let us know your feedback through the Webmaster Tools Forum.

Update on 12/04/2009: Our team just reconvened to provide you more information on this feature. Check out JohnMu's latest post on Site Performance!


Dj Matioka

Your site's performance in Webmaster Tools

Webmaster level: Intermediate

Let's take a quick look at the individual sections in the Google Webmaster Tools' Site Performance feature:

Performance overview



The performance overview shows a graph of the aggregated speed numbers for the website, based on the pages that were most frequently accessed by visitors who use the Google Toolbar with the PageRank feature activated. By using data from Google Toolbar users, you don't have to worry about us testing your site from a location that your users do not use. For example, if your site is in Germany and all your users are in Germany, the chart will reflect the load time as seen in Germany. Similarly, if your users mostly use dial-up connections (or high-speed broadband), that would be reflected in these numbers as well. If only a few visitors of your site use the Google Toolbar, we may not be able to show this data in Webmaster Tools.

The line between the red and the green sections on the chart is the 20th percentile — only 20% of the sites we check are faster than this. This website is pretty close to the 20% mark, which pages would we have to work on first?

Example pages with load times



In this section you can find some example pages along with the average, aggregated load times that users observed while they were on your website. These numbers may differ from what you see as they can come from a variety of different browsers, internet connections and locations. This list can help you to recognize pages which take longer than average to load — pages that slow your users down.

As the page load times are based on actual accesses made by your users, it's possible that it includes pages which are disallowed from crawling. While Googlebot will not be able to crawl disallowed pages, they may be a significant part of your site's user experience.

Keep in mind that you may see occasional spikes here, so it's recommended that you watch the load times over a short period to see what's stable. If you consistently see very large load times, that probably means that most of your users are seeing very slow page loads (whether due to slow connections or otherwise), so it's something you should take seriously.

Page Speed suggestions



These suggestions are based on the Page Speed Firefox / Firebug plugin. In order to find the details for these sample URLs, we fetch the page and all its embedded resources with Googlebot. If we are not able to fetch all of embedded content with Googlebot, we may not be able to provide a complete analysis. Similarly, if the servers return slightly modified content for Googlebot than they would for normal users, this may affect what is shown here. For example, some servers return uncompressed content for Googlebot, similar to what would be served to older browsers that do not support gzip-compressed embedded content (this is currently the case for Google Analytics' "ga.js").

When looking at flagged issues regarding common third-party code such as website analytics scripts, one factor that can also play a role is how wide-spread these scripts are on the web. If they are common across the web, chances are that the average user's browser will have already cached the DNS lookup and the content of the script. While these scripts will still be flagged as separate DNS lookups, in practice they might not play a strong role in the actual load time.

We offer these suggestions as a useful guideline regarding possible first performance improvement steps and recommend using the Page Speed plugin (or a similar tool) directly when working on your website. This allows you to better recognize the blocking issues and makes it easy to see how modifications on the server affect the total load time.


For questions about Webmaster Tools and this new feature, feel free to read the Help Center article, search and post in the Webmaster Help Forums or in the Page Speed discussion group. We hope this information helps you make your website even faster!


Dj Matioka

Handling legitimate cross-domain content duplication

Webmaster level: Intermediate

We've recently discussed several ways of handling duplicate content on a single website; today we'll look at ways of handling similar duplication across different websites, across different domains. For some sites, there are legitimate reasons to duplicate content across different websites — for instance, to migrate to a new domain name using a web server that cannot create server-side redirects. To help with issues that arise on such sites, we're announcing our support of the cross-domain rel="canonical" link element.



Ways of handling cross-domain content duplication:

• Choose your preferred domain
  When confronted with duplicate content, search engines will generally take one version and filter the others out. This can also happen when multiple domain names are involved, so while search engines are generally pretty good at choosing something reasonable, many webmasters prefer to make that decision themselves.

• Reduce in-site duplication
  Before starting on cross-site duplicate content questions, make sure to handle duplication within your site first.

• Enable crawling and use 301 (permanent) redirects where possible
  Where possible, the most important step is often to use appropriate 301 redirects. These redirects send visitors and search engine crawlers to your preferred domain and make it very clear which URL should be indexed. This is generally the preferred method as it gives clear guidance to everyone who accesses the content. Keep in mind that in order for search engine crawlers to discover these redirects, none of the URLs in the redirect chain can be disallowed via a robots.txt file. Don't forget to handle your www / non-www preference with appropriate redirects and in Webmaster Tools.

• Use the cross-domain rel="canonical" link element
  There are situations where it's not easily possible to set up redirects. This could be the case when you need to move your website from a server that does not feature server-side redirects. In a situation like this, you can use the rel="canonical" link element across domains to specify the exact URL of whichever domain is preferred for indexing. While the rel="canonical" link element is seen as a hint and not an absolute directive, we do try to follow it where possible.

Still have questions?

Q: Do the pages have to be identical?
A: No, but they should be similar. Slight differences are fine.

Q: For technical reasons I can't include a 1:1 mapping for the URLs on my sites. Can I just point the rel="canonical" at the homepage of my preferred site?
A: No; this could result in problems. A mapping from old URL to new URL for each URL on the old site is the best way to use rel="canonical".

Q: I'm offering my content / product descriptions for syndication. Do my publishers need to use rel="canonical"?
A: We leave this up to you and your publishers. If the content is similar enough, it might make sense to use rel="canonical", if both parties agree.

Q: My server can't do a 301 (permanent) redirect. Can I use rel="canonical" to move my site?
A: If it's at all possible, you should work with your webhost or web server to do a 301 redirect. Keep in mind that we treat rel="canonical" as a hint, and other search engines may handle it differently. But if a 301 redirect is impossible for some reason, then a rel="canonical" may work for you. For more information, see our guidelines on moving your site.

Q: Should I use a noindex robots meta tag on pages with a rel="canonical" link element?
A: No, since those pages would not be equivalent with regards to indexing - one would be allowed while the other would be blocked. Additionally, it's important that these pages are not disallowed from crawling through a robots.txt file, otherwise search engine crawlers will not be able to discover the rel="canonical" link element.

We hope this makes it easier for you to handle duplicate content in a user-friendly way. Are there still places where you feel that duplicate content is causing your sites problems? Let us know in the Webmaster Help Forum!



Dj Matioka

How did you do on the Webmaster Quiz?

Webmaster Level: All

Thanks to all of you who took our webmaster quiz and waited patiently to see how well you did! Today, we're pleased to present the Webmaster Quiz answers! We hope this quiz has provided some clarity on common issues users ask about in the Webmaster Help Forum. We'll go over a few of the questions and answers here, but if some of the answers lead you to ask more questions, we encourage you to continue the discussion in the forum!

1) You have moved your site to a new domain name. For users and search engines, the preferred way to permanently redirect traffic is:

Correct answer: a) 301 redirect
Explanation: A 301 redirect is preferred because it tells search engines, "Ok, this is the new domain I want you to show to users from now on," as opposed to something like a 302 redirect, which tells search engines, "Hey, this is only a temporary redirect--so, uh, I might change the URL soon, okay?" In addition to implementing a 301 redirect, the Change of Address feature in Webmaster Tools can help Google find your new site.

2) Your server is going to be moved and unavailable for a day. What should you do?

Correct answer: c) Return "Network Unavailable (503)" with a helpful message to all requests

Explanation: Maybe not as commonly known to webmasters, but very useful if your site is down! This tells crawlers to come back later, rather than crawling and indexing your "Down for maintenance" pages when you respond with 200 rather than 503. Check out the Help Center to learn more about HTTP status codes.

3) Your website is not in the index five days after you've put it online; what should you do?

Correct answer: b) Continue working on the site

Explanation: This one is a bit tricky. There could be a number of reasons why your site is not indexed. For example, a site's robots.txt file may contain a directive to inadvertently block crawlers from searching its contents. But the main take-away from this question is that if your site is pretty new, it may just be a matter of time before it gets indexed. You should continue to focus on improving your site for your users.

6) You need to remove 192 PDF files from the /private-files/ folder which have gotten indexed. What's the fastest way to do this?

Correct answer: d) Disallow the folder in robots.txt and request removal of the whole folder in Webmaster Tools.

Explanation: Before removing a directory that you don't want indexed, you need to include the Disallow directive in your robots.txt file to tell search bots not to crawl it anymore.

9) You have a country-coded domain name called example.es. To associate your site with Spain, you need to:

Correct answer: c) None of this is necessary. Google should already associate a domain ending in .es with Spain.

Explanation: Some country-coded domains may overlap with international ones, like .tv--which could also be a site from Tuvalu. But these sort of cases are rare and if they do arise, don't be shy to seek out help on the forum.

Great job to everyone who took the quiz and tested their know-how! And last but certainly not least, kudos to the top scorers! Congratulations on a quiz well done!

40/40:

• ChrisRaimondi
• theopeek
• beussery

39/40:

• Petro
• pornel
• Ian Macfarlane
• g1smd
• Mattman
• thinkpragmatic
• GLV
• GoalGorilla
• rssmarketer

38/40:

• BartVB
• Kim Minh Kaplan
• Ippi
• Erik Dafforn
• scole01
• Konstantin
• John
• fer.vazquez
• eMBe
• Todd Nemet
• p.jaroszynski
• ph0b

Dj Matioka

Is your site hacked? New Message Center notifications for hacking and abuse

Webmaster Level: All

As we crawl the web, we see bad content inserted on to thousands of hacked sites each day. The number of sites attacked is staggering and the problem is only getting worse. Hackers and spammers target and successfully compromise any sites they can - small personal sites, schools and universities, even multinational corporations. Spam attacks against forums and user content sections of sites, though not as shocking, are even more widespread.

You may have read in an earlier post that we've begun notifying webmasters about new software versions via Webmaster Tools to help protect their sites. Continuing with our effort to provide more useful information to webmasters, we're happy to announce that we'll soon be sending even more notifications to the Message Center.

Starting this month, we will notify more webmasters of more potential issues we've detected on their websites, including:

• Spammy or abused user-generated content
• Abused forum pages or egregious amounts of comment spam
• Suspected hacking

These notifications are meant to alert webmasters of potential issues and provide next steps on how to get their sites fixed and back into Google's search results. If it pertains to a hacking or abuse issue, the notification will point to example URLs exhibiting this type of behavior. These notifications will run in parallel with our existing malware notifications.

A notice of suspected hacking, for example, will look like this:

We've been notifying webmasters of suspected hacking for years, but a recent upgrade to our systems will allow us to notify many more site owners that have been hacked. We hope webmasters will find these notifications useful in making sure their sites are clean and secure, ultimately providing a better user experience for their visitors. In the future, we may extend this effort even further to include other types of vulnerabilities or abuse issues.

Just as before, webmasters who have not already signed up for Webmaster Tools may still do so and retrieve previously sent messages within one year of their send date. And if you don't want to miss out on any important messages, remember to use the email forwarding feature to receive these alerts in your inbox.

If you have any questions, please feel free to ask in our Webmaster Help Forum or leave your comments below.

Dj Matioka

Google's SEO Report Card

Webmaster Level: All

How many of Google's web pages use a descriptive title tag? Do we use description meta tags? Heading tags? While we always try to focus on the user, could our products use an SEO tune up? These are just some of the questions we set out to answer with Google's SEO Report Card.

Google's SEO Report Card is an effort to provide Google's product teams with ideas on how they can improve their products' pages using simple and accepted optimizations. These optimizations are intended to not only help search engines understand the content of our pages better, but also to improve our users' experience when visiting our sites. Simple steps such as fixing 404s and broken links, simplifying URL choice, and providing easier-to-understand titles and snippets for our pages can benefit both users and search engines. From the start of the project we also wanted to release the report card publicly so other companies and webmasters could learn from the report, which is filled with dozens of examples taken straight from our products' pages.

The project looked at the main pages of 100 different Google products, measuring them across a dozen common optimization categories. Future iterations of the project might look at deeper Google product web pages as well as international ones. We released the report card within Google last month and since then a good number of teams have taken action on it or plan to.

We hope you find our SEO Report Card useful and we'd love to hear your feedback in the comments below or in the Webmaster Central Help Forum. And if you'd like to do your own SEO tune up, a good place to start is by reading our free SEO Beginner's Guide.


Dj Matioka

A word on site clinics

Webmaster Level: All

We try to communicate with webmasters in lots of different places. For example, when we send representatives to conferences we’re happy to participate in public site clinics where we share best practices on how to improve the crawlability and site architecture of websites suggested by the audience.

However, we also want to help users who can’t or don’t want to attend search conferences. To reach more people, we started doing free virtual site clinics in languages other than English. These site clinics help site owners make websites in such a way that they are more easily crawled, indexed, and returned by search engine crawlers, which in turn helps webmasters gain more visibility on the web.

We did a series of free virtual site clinics in Spanish last year which spanned 5 blog posts. The clinics covered real problems on real sites, and we posted the results on the Spanish Webmaster Central blog. If you read Spanish, I recommend you go read the different posts covering everything from issues with framed sites, to more technical domain setup.

In some countries we don’t have dedicated webmaster-focused blogs, but we still want to help webmasters in those countries. That means that you might occasionally see site clinic or webmaster-related posts on AdWords blogs such as the forthcoming ones on the Nordic AdWords blogs (which cover Danish, Finnish, Norwegian and Swedish). Recently when we posted some advice for webmasters on one of our AdWords blogs, we received questions about the relationship between Google’s search and advertising programs. We wanted to again reassure our users that the ranking of Google’s organic search results is entirely separate from our advertising programs. Furthermore, we do not give any preference to AdWords customers in our site clinics - everybody is welcome to participate. We’re simply posting this on local “AdWords” blogs because it’s the best way for us to reach webmasters in those communities and languages.


Dj Matioka

Sharing the verification love

Webmaster Level: All

Everything is more fun with a friend! We've just added a feature to Webmaster Tools Site Verification to make it easier to share verified ownership of your websites.

In the past, if more than one person needed to be a verified owner of a website, they each had to go through the meta tag or HTML file verification process. That works fine for some situations, but for others it can be challenging. For example, what if you have twenty people who need to be verified owners of your site? Adding twenty meta tags or HTML files could be pretty time consuming. Our new verification delegation feature makes adding new verified owners a snap.



Once you're a verified owner of a website, you can view the Verification Details page (linked from Webmaster Tools or the Verification home page). That page will show you information about the site as well as a list of any other verified owners. At the bottom of the list of owners, you'll now see a button labeled "Add a user...". Click that, enter the user's email address, and that person will instantly become a verified owner for the site! You can remove that ownership at any time by clicking the "Unverify" link next to the person's email address on the Details page.

There are a few important things to keep in mind as you use this feature. First, each site must always have at least one owner who has verified directly (via meta tag or HTML file). If all of the directly verified owners become unverified, the delegated owners may also become unverified. Second, you can only delegate ownership to people with Google Accounts. Finally, remember that anyone you delegate ownership to will have exactly the same access you have. They can delegate to more people, submit URL Removal requests and manage Sitelinks in Webmaster Tools, etc. Only delegate ownership to people you trust!

We hope this makes things a little easier for those of you who need more than one person to be a verified owner of your site. As always, please visit the Webmaster Help Forum if you have any questions.



Dj Matioka

Fetch as Googlebot Mobile and Claim your Sidewiki comment - added to Webmaster Tools Labs!

Webmaster Level: All

Last October, we launched Webmaster Tools Labs and it has been a huge success. Malware Details have helped thousands of users identify pages on their site that may be infected with malicious code, and Fetch as Googlebot has given users more insight into our crawler.

Today, we're happy to announce two additional Labs features:

• Fetch as Googlebot-Mobile
• Create your Sidewiki page owner entry

Fetch as Gooblebot Mobile (developed by Ryoichi Imaizumi)

After we launched Fetch as Googlebot, many users with mobile-specific sites asked if we could provide the ability to fetch their pages as Googlebot-Mobile. We thought it was a great idea, and added it as an option to our Fetch as Googlebot feature. We have two mobile options: cHTML (primarily used for Japanese sites), and XHTML/WML.





Create your Sidewiki page owner entry (developed by Derek Prothro)

Sidewiki allows users to contribute helpful information to any webpage using a sidebar in Google Toolbar or a Chrome extension. Webmasters can create a special entry, called a page owner entry, that appears above all entries written by users.



After Sidewiki launched webmasters kept asking, "How can I put a Sidewiki page owner entry on all pages of my site quickly?" With the feature that we're introducing today, you can now create these page owner entries directly within Webmaster Tools for any site you own.



We're really happy about these new features, and hope you enjoy them as much as we do. Let us know what you think!



Dj Matioka

Microdata support for Rich Snippets

Webmaster Level: All

HTML5 is the fifth major revision of HTML, the core language of the World Wide Web. The HTML5 specification includes a description of microdata, a new markup standard for specifying structured information within web pages.

Today, we’re happy to announce support for microdata for use in rich snippets in addition to our existing support for microformats and RDFa. By using microdata markup in your web pages, you can specify reviews, people profiles, or events information on your web pages that Google may use to improve the presentation of your pages in Google search results.

Here is a simple HTML block showing a section of a review of “L’Amourita Pizza”:

Here is the same HTML with microdata added to specify the restaurant being reviewed, the author and date of the review, and the rating:

Microdata has the nice property of balancing richness with simplicity. As you can see, it’s easy to add markup to your pages using a few HTML attributes like itemscope (to define a new item), itemtype (to specify the type of item being described), and itemprop (to specify a property of that item). Once you’ve added markup to a page, you can test it using the rich snippets testing tool to make sure that Google can parse the data on your page.

As with microformats and RDFa, the vocabulary that we support -- including which item types and item properties are understood by Google -- is specified in our rich snippets documentation as well as on data-vocabulary.org. Marking up your content does not guarantee that rich snippets will show for your site; Google will expand the use of microdata markup gradually to ensure a great user experience.

To get started, here are some helpful links:

• Rich snippets documentation
• Overview of microdata
• Official microdata specification
• Rich snippets testing tool

Dj Matioka

Working with multi-regional websites

Webmaster Level: Intermediate

Did you know that a majority of users surveyed feel that having information in their own language was more important than a low price? Living in a non-English-speaking country, I've seen friends and family members explicitly look for and use local and localized websites—properly localized sites definitely have an advantage with users. Google works hard to show users the best possible search results. Many times those are going to be pages that are localized, for the user's location and/or in the user's language.

If you're planning to take the time to create and maintain a localized version of your website, making it easy to recognize and find is a logical part of that process. In this blog post series, we'll take a look at what is involved with multi-regional and multi-lingual websites from a search engine point of view. A multi-regional website is one that explicitly targets users in various regions (generally different countries); we call it multilingual when it is available in multiple languages, and sometimes, the website targets both multiple regions and is in multiple languages. Let's start with some general preparations and then look at websites that target multiple regions.

Preparing for global websites

Expanding a website to cover multiple regions and/or languages can be challenging. By creating multiple versions of your website, any issues with the base version will be multiplied; make sure that you have everything working properly before you start. Given that this generally means you'll suddenly be working with a multiplied number of URLs, don't forget that you'll need appropriate infrastructure to support the website.

Planning multi-regional websites

When planning sites for multiple regions (usually countries), don't forget to research legal or administrative requirements that might come into play first. These requirements may determine how you proceed, for instance whether or not you would be eligible to use a country-specific domain name.

All websites start with domain names; when it comes to domain names, Google differentiates between two types of domain names:

• ccTLDs (country-code top level domain names): These are tied to a specific country (for example .de for Germany, .cn for China). Users and search engines use this as a strong sign that your website is explicitly for a certain country.
• gTLDs (generic top level domain names): These are not tied to a specific country. Examples of gTLds are .com, .net, .org, .museum. Google sees regional top level domain names such as .eu and .asia as gTLDs, since they cannot be tied to a specific country. We also treat some vanity ccTLDs (such as .tv, .me, etc.) as gTLDs as we've found that users and webmasters frequently see these as being more generic than country-targeted (we don't have a complete list of such vanity ccTLDs that we treat as gTLDs as it may change over time). You can set geotargeting for websites with gTLDs using the Webmaster Tools Geographic Target setting.

Geotargeting factors

Google generally uses the following elements to determine the geotargeting of a website (or a part of a website):

1. Use of a ccTLD is generally a strong signal for users since it explicitly specifies a single country in an unmistakable way.
   or
   Webmaster Tools' manual geotargeting for gTLDs (this can be on a domain, subdomain or subdirectory level); more information on this can be found in our blog post and in the Help Center. With region tags from geotargeting being shown in search results, this method is also very clear to users. Please keep in mind that it generally does not make sense to set a geographic target if the same pages on your site target more than a single country (say, all German-speaking countries) — just write in that language and do not use the geotargeting setting (more on writing in other languages will follow soon!).
2. Server location (through the IP address of the server) is frequently near your users. However, some websites use distributed content delivery networks (CDNs) or are hosted in a country with better webserver infrastructure, so we try not to rely on the server location alone.
3. Other signals can give us hints. This could be from local addresses & phone numbers on the pages, use of local language and currency, links from other local sites, and/or the use of Google's Local Business Center (where available).

Note that we do not use locational meta tags (like "geo.position" or "distribution") or HTML attributes for geotargeting. While these may be useful in other regards, we've found that they are generally not reliable enough to use for geotargeting.

URL structures

The first three elements used for geotargeting are strongly tied to the server and to the URLs used. It's difficult to determine geotargeting on a page by page basis, so it makes sense to consider using a URL structure that makes it easy to segment parts of the website for geotargeting. Here are some of the possible URL structures with pros and cons with regards to geotargeting:

ccTLDs eg: example.de, example.fr	Subdomains with gTLDs eg: de.site.com, fr.site.com, etc.	Subdirectories with gTLDs eg: site.com/de/, site.com/fr/, etc.	URL parameters eg: site.com?loc=de, ?country=france, etc.
pros (+) - clear geotargeting - server location is irrelevant - easy separation of sites - legal requirements (sometimes)	pros (+) - easy to set up - can use Webmaster Tools geotargeting - allows different server locations - easy separation of sites	pros (+) - easy to set up - can use Webmaster Tools geotargeting - low maintenance (same host)	pros (+) (not recommended)
cons (-) - expensive (+ availability) - more infrastructure - ccTLD requirements (sometimes)	cons (-) - users might not recognize geotargeting from the URL alone (is "de" the language or country?)	cons (-) - users might not recognize geotargeting from the URL alone - single server location - separation of sites harder	cons (-) - segmentation based on the URL is difficult - users might not recognize geotargeting from the URL alone - geotargeting in Webmaster Tools is not possible

As you can see, geotargeting is not an exact science (even sites using country-code top level domain names can be global in nature), so it's important that you plan for the users from the "wrong" location. One way to do this could be to show links on all pages for users to select their region and language of choice. We'll look at some other possible solutions further on in this blog post series.

Dealing with duplicate content on global websites

Websites that provide content for different regions and in different languages sometimes create content that is the same or similar but available on different URLs. This is generally not a problem as long as the content is for different users in different countries. While we strongly recommend that you provide unique content for each different group of users, we understand that this may not always be possible for all pages and variations from the start. There is generally no need to "hide" the duplicates by disallowing crawling in a robots.txt file or by using a "noindex" robots meta tag. However, if you're providing the same content to the same users on different URLs (for instance, if both "example.de/" and "example.com/de/" show German language content for users in Germany), it would make sense to choose a preferred version and to redirect (or use the "rel=canonical" link element) appropriately.

Do you already have a website that targets multiple regions or do you have questions about the process of planning one? Come to the Help Forum and join the discussion. In following posts, we'll take a look at multi-lingual websites and then look at some special situations that can arise with global websites. Bis bald!



Dj Matioka