Google SEO resources for beginners

Webmaster Level: Beginner

Want to eat healthier and exercise more in 2010? That's tough! Want to learn about search engine optimization (SEO) so you can disregard the rumors and know what's important? That's easy! Here's how to gain SEO knowledge as you go about your new start to 2010:

Step 1: Absorb the basics

• If you like to learn by reading, download our SEO Starter Guide for reading while you're on an exercise bike, training for Ironman.
• Or, if you're more a video watcher, try listening to my "Search Friendly Development" session while you're cleaning your house. Keep in mind that some parts of the presentation are a little more technical.

• For good measure, and because at some point you'll hear references to them, check out our webmaster guidelines for yourself.

Step 2: Explore details that pique your interest

Are you done with the basics but now you have some questions? Good for you! Try researching a particular topic in our Webmaster Help Center. For example, do you want more information about crawling and indexing or understanding what links are all about?

Step 3: Verify ownership of your site in Webmaster Tools

It takes a little bit of skill, but we have tons of help for verification. Once you verify ownership of your site (i.e., signal to Google that you're the owner), you can:

• See more confidential information about your site, like crawl errors or messages from Google about your site
• Run through the Webmaster Tools checklist and learn new ways to monitor and manage your site
• Forward your Google messages to your personal email address

A sample message regarding the crawlability of your site

Step 4: Research before you do anything drastic

Usually the basics (e.g., good content/service and a crawlable site with indexable information) are the necessities for SEO. You may hear or read differently, but before you do anything drastic on your site such as robots.txt disallow'ing all of your directories or revamping your entire site architecture, please try:

• Researching material in our Help Center
• Checking for related posts in our Webmaster Forum
• Asking for feedback from the webmaster community -- we've got a super skilled group of Google employees, members, and Bionic Posters.
• 
  
• Dj Matioka

'New software version' notifications for your site

Webmaster level: All

One of the great things about working at Google is that we get to take advantage of an enormous amount of computing power to do some really cool things. One idea we tried out was to let webmasters know about their potentially hackable websites. The initial effort was successful enough that we thought we would take it one step further by expanding our efforts to cover other types of web applications—for example, more content management systems (CMSs), forum/bulletin-board applications, stat-trackers, and so on.

This time, however, our goal is not just to isolate vulnerable or hackable software packages, but to also notify webmasters about newer versions of the software packages or plugins they're running on their website. For example, there might be a Drupal module or Joomla extension update available but some folks might not have upgraded. There are a few reasons a webmaster might not upgrade to the newer version and one of the reasons could be that they just don't know a new version exists. This is where we think we can help. We hope to let webmasters know about new versions of their software by sending them a message via Webmaster Tools. This way they can make an informed decision about whether or not they would like to upgrade.

One of the ways we identify sites to notify is by parsing source code of web pages that we crawl. For example, WordPress and other CMS applications include a generator meta tag that specifies the version number. This has proven to be tremendously helpful in our efforts to notify webmasters. So if you're a software developer, and would like us to help you notify your users about newer versions of your software, a great way to start would be to include a generator meta tag that tells the version number of your software. If you're a plugin or a widget developer, including a version number in the source you provide to your users is a great way to help too.

We've seen divided opinions over time about whether it's a good security practice to include a version number in source code, because it lets hackers or worm writers know that the website might be vulnerable to a particular type of exploit. But as Matt Mullenweg pointed out, "Where [a worm writer's] 1.0 might have checked for version numbers, 2.0 just tests [a website's] capabilities...". Meanwhile, the advantage of a version number is that it can help alert site owners when they need to update their site. In the end, we tend to think that including a version number can do more good than harm.

We plan to begin sending out the first of these messages soon and hope that webmasters find them useful! If you have any questions or feedback, feel free to comment here.


Dj Matioka

Region Tags in Google Search Results

Webmaster Level: All

Country-code top-level domains (or ccTLDs) can provide people with a quick and valuable clue about the location of a website—for example, ".fr" for France or ".co.jp" for Japan. However, for certain top level domains like .com, .info and .org, it's not as easy to figure out the location. That's why today we're adding region information supplied by webmasters to the green address line on some Google search results.

This feature is easiest to explain through an example. Let's say you've heard about a boxing club in Canada called "Capital City Boxing." You try a search for [capital city boxing] to find out more, but it's hard to tell which result is the one you're looking for. Here's a screen shot:

None of the results provide any location information in the title or snippet, nor do they have a regional TLD (such as .ca for Canada). The only way to find the result you're looking for is to refine your search ([capital city boxing canada] works) or click through the various links to figure it out. Clicking through the first result reveals that there's apparently another "Capital City Boxing" club in Alabama.

Region tags improve search results by providing valuable information about website location right in the green URL line. Continuing our prior example, here's a screen shot of the new region tag (circled in red):

As you can see, the fourth result now includes the region name "Canada" after the green URL, so you can immediately tell that this result relates to the boxing club in Canada. With the new display, you no longer need to refine your search or click through the results to figure out which page is the one you're looking for. In general, our hope is that these region tags will help searchers more quickly identify which results are most relevant to their queries.

As a webmaster, you can control how this feature works by adjusting your Geographic Targeting settings. Log in to Webmaster Tools and choose Site configuration > Settings > Geographic Target. From here you can associate a particular country/region with your site. These settings will determine the name that appears as a region tag. You can learn more about using the Geographic Target tool in a prior blog post and in our Help Center.

We currently show region tags only for certain domains such as .com and .net where the location information would otherwise be unclear. We don't show region tags for results on domains like .br for Brazil, because the location is already implied by the green URL line in our default display. In addition, we only display region tags when the region supplied by the site owner is different from the domain where the search was entered. For example, if you do a search from the Singapore Google domain (google.com.sg), we won't show you region tags for all the websites webmasters have targeted to Singapore because we'd end up tagging too many results, and the tag is really most relevant for foreign regions. For the initial release, we anticipate roughly 1% of search results pages will include webpages with a region tag.

We hope you'll find this new feature useful, and we welcome your feedback.


Dj Matioka

How fast is your site?

We've just launched Site Performance, an experimental feature in Webmaster Tools that shows you information about the speed of your site and suggestions for making it faster.

This is a small step in our larger effort to make the web faster. Studies have repeatedly shown that speeding up your site leads to increased user retention and activity, higher revenue and lower costs. Towards the goal of making every webpage load as fast as flipping the pages of a magazine, we have provided articles on best practices, active discussion forums and many tools to diagnose and fix speed issues.

Now we bring data and statistics specifically applicable to your site. On Site Performance, you'll find how fast your pages load, how they've fared over time, how your site's load time compares to that of other sites, examples of specific pages and their actual page load times, and Page Speed suggestions that can help reduce user-perceived latency. Our goal is to bring you specific and actionable speed information backed by data, so stay tuned for more of this in the future.



The load time data is derived from aggregated information sent by users of your site who have installed the Google Toolbar and opted-in to its enhanced features. We only show the performance charts and tables when there's enough data, so not all of them may be shown if your site has little traffic. The data currently represents a global average; a specific user may experience your site faster or slower than the average depending on their location and network conditions.

This is a Labs product that is still in development. We hope you find it useful. Please let us know your feedback through the Webmaster Tools Forum.

Update on 12/04/2009: Our team just reconvened to provide you more information on this feature. Check out JohnMu's latest post on Site Performance!


Dj Matioka

Your site's performance in Webmaster Tools

Webmaster level: Intermediate

Let's take a quick look at the individual sections in the Google Webmaster Tools' Site Performance feature:

Performance overview



The performance overview shows a graph of the aggregated speed numbers for the website, based on the pages that were most frequently accessed by visitors who use the Google Toolbar with the PageRank feature activated. By using data from Google Toolbar users, you don't have to worry about us testing your site from a location that your users do not use. For example, if your site is in Germany and all your users are in Germany, the chart will reflect the load time as seen in Germany. Similarly, if your users mostly use dial-up connections (or high-speed broadband), that would be reflected in these numbers as well. If only a few visitors of your site use the Google Toolbar, we may not be able to show this data in Webmaster Tools.

The line between the red and the green sections on the chart is the 20th percentile — only 20% of the sites we check are faster than this. This website is pretty close to the 20% mark, which pages would we have to work on first?

Example pages with load times



In this section you can find some example pages along with the average, aggregated load times that users observed while they were on your website. These numbers may differ from what you see as they can come from a variety of different browsers, internet connections and locations. This list can help you to recognize pages which take longer than average to load — pages that slow your users down.

As the page load times are based on actual accesses made by your users, it's possible that it includes pages which are disallowed from crawling. While Googlebot will not be able to crawl disallowed pages, they may be a significant part of your site's user experience.

Keep in mind that you may see occasional spikes here, so it's recommended that you watch the load times over a short period to see what's stable. If you consistently see very large load times, that probably means that most of your users are seeing very slow page loads (whether due to slow connections or otherwise), so it's something you should take seriously.

Page Speed suggestions



These suggestions are based on the Page Speed Firefox / Firebug plugin. In order to find the details for these sample URLs, we fetch the page and all its embedded resources with Googlebot. If we are not able to fetch all of embedded content with Googlebot, we may not be able to provide a complete analysis. Similarly, if the servers return slightly modified content for Googlebot than they would for normal users, this may affect what is shown here. For example, some servers return uncompressed content for Googlebot, similar to what would be served to older browsers that do not support gzip-compressed embedded content (this is currently the case for Google Analytics' "ga.js").

When looking at flagged issues regarding common third-party code such as website analytics scripts, one factor that can also play a role is how wide-spread these scripts are on the web. If they are common across the web, chances are that the average user's browser will have already cached the DNS lookup and the content of the script. While these scripts will still be flagged as separate DNS lookups, in practice they might not play a strong role in the actual load time.

We offer these suggestions as a useful guideline regarding possible first performance improvement steps and recommend using the Page Speed plugin (or a similar tool) directly when working on your website. This allows you to better recognize the blocking issues and makes it easy to see how modifications on the server affect the total load time.


For questions about Webmaster Tools and this new feature, feel free to read the Help Center article, search and post in the Webmaster Help Forums or in the Page Speed discussion group. We hope this information helps you make your website even faster!


Dj Matioka

Sharing the verification love

Webmaster Level: All

Everything is more fun with a friend! We've just added a feature to Webmaster Tools Site Verification to make it easier to share verified ownership of your websites.

In the past, if more than one person needed to be a verified owner of a website, they each had to go through the meta tag or HTML file verification process. That works fine for some situations, but for others it can be challenging. For example, what if you have twenty people who need to be verified owners of your site? Adding twenty meta tags or HTML files could be pretty time consuming. Our new verification delegation feature makes adding new verified owners a snap.



Once you're a verified owner of a website, you can view the Verification Details page (linked from Webmaster Tools or the Verification home page). That page will show you information about the site as well as a list of any other verified owners. At the bottom of the list of owners, you'll now see a button labeled "Add a user...". Click that, enter the user's email address, and that person will instantly become a verified owner for the site! You can remove that ownership at any time by clicking the "Unverify" link next to the person's email address on the Details page.

There are a few important things to keep in mind as you use this feature. First, each site must always have at least one owner who has verified directly (via meta tag or HTML file). If all of the directly verified owners become unverified, the delegated owners may also become unverified. Second, you can only delegate ownership to people with Google Accounts. Finally, remember that anyone you delegate ownership to will have exactly the same access you have. They can delegate to more people, submit URL Removal requests and manage Sitelinks in Webmaster Tools, etc. Only delegate ownership to people you trust!

We hope this makes things a little easier for those of you who need more than one person to be a verified owner of your site. As always, please visit the Webmaster Help Forum if you have any questions.



Dj Matioka

Fetch as Googlebot Mobile and Claim your Sidewiki comment - added to Webmaster Tools Labs!

Webmaster Level: All

Last October, we launched Webmaster Tools Labs and it has been a huge success. Malware Details have helped thousands of users identify pages on their site that may be infected with malicious code, and Fetch as Googlebot has given users more insight into our crawler.

Today, we're happy to announce two additional Labs features:

• Fetch as Googlebot-Mobile
• Create your Sidewiki page owner entry

Fetch as Gooblebot Mobile (developed by Ryoichi Imaizumi)

After we launched Fetch as Googlebot, many users with mobile-specific sites asked if we could provide the ability to fetch their pages as Googlebot-Mobile. We thought it was a great idea, and added it as an option to our Fetch as Googlebot feature. We have two mobile options: cHTML (primarily used for Japanese sites), and XHTML/WML.





Create your Sidewiki page owner entry (developed by Derek Prothro)

Sidewiki allows users to contribute helpful information to any webpage using a sidebar in Google Toolbar or a Chrome extension. Webmasters can create a special entry, called a page owner entry, that appears above all entries written by users.



After Sidewiki launched webmasters kept asking, "How can I put a Sidewiki page owner entry on all pages of my site quickly?" With the feature that we're introducing today, you can now create these page owner entries directly within Webmaster Tools for any site you own.



We're really happy about these new features, and hope you enjoy them as much as we do. Let us know what you think!



Dj Matioka

URL removal explained, Part I: URLs & directories

Webmaster level: All

There's a lot of content on the Internet these days. At some point, something may turn up online that you would rather not have out there—anything from an inflammatory blog post you regret publishing, to confidential data that accidentally got exposed. In most cases, deleting or restricting access to this content will cause it to naturally drop out of search results after a while. However, if you urgently need to remove unwanted content that has gotten indexed by Google and you can't wait for it to naturally disappear, you can use our URL removal tool to expedite the removal of content from our search results as long as it meets certain criteria (which we'll discuss below).

We've got a series of blog posts lined up for you explaining how to successfully remove various types of content, and common mistakes to avoid. In this first post, I'm going to cover a few basic scenarios: removing a single URL, removing an entire directory or site, and reincluding removed content. I also strongly recommend our previous post on managing what information is available about you online.

Removing a single URL

In general, in order for your removal requests to be successful, the owner of the URL(s) in question—whether that's you, or someone else—must have indicated that it's okay to remove that content. For an individual URL, this can be indicated in any of three ways:

• block the page from crawling via a robots.txt file
• block the page from indexing via a noindex meta tag
• indicate that the page no longer exists by returning a 404 or 410 status code

Before submitting a removal request, you can check whether the URL is correctly blocked:

• robots.txt: You can check whether the URL is correctly disallowed using either the Fetch as Googlebot or Test robots.txt features in Webmaster Tools.
• noindex meta tag: You can use Fetch as Googlebot to make sure the meta tag appears somewhere between the <head> and </head> tags. If you want to check a page you can't verify in Webmaster Tools, you can open the URL in a browser, go to View > Page source, and make sure you see the meta tag between the <head> and </head> tags.
• 404 / 410 status code: You can use Fetch as Googlebot, or tools like Live HTTP Headers or web-sniffer.net to verify whether the URL is actually returning the correct code. Sometimes "deleted" pages may say "404" or "Not found" on the page, but actually return a 200 status code in the page header; so it's good to use a proper header-checking tool to double-check.

If unwanted content has been removed from a page but the page hasn't been blocked in any of the above ways, you will not be able to completely remove that URL from our search results. This is most common when you don't own the site that's hosting that content. We cover what to do in this situation in a subsequent post. in Part II of our removals series.

If a URL meets one of the above criteria, you can remove it by going to http://www.google.com/webmasters/tools/removals, entering the URL that you want to remove, and selecting the "Webmaster has already blocked the page" option. Note that you should enter the URL where the content was hosted, not the URL of the Google search where it's appearing. For example, enter
   http://www.example.com/embarrassing-stuff.html
not
   http://www.google.com/search?q=embarrassing+stuff

This article has more details about making sure you're entering the proper URL. Remember that if you don't tell us the exact URL that's troubling you, we won't be able to remove the content you had in mind.

Removing an entire directory or site

In order for a directory or site-wide removal to be successful, the directory or site must be disallowed in the site's robots.txt file. For example, in order to remove the http://www.example.com/secret/ directory, your robots.txt file would need to include:
   User-agent: *
   Disallow: /secret/
It isn't enough for the root of the directory to return a 404 status code, because it's possible for a directory to return a 404 but still serve out files underneath it. Using robots.txt to block a directory (or an entire site) ensures that all the URLs under that directory (or site) are blocked as well. You can test whether a directory has been blocked correctly using either the Fetch as Googlebot or Test robots.txt features in Webmaster Tools.

Only verified owners of a site can request removal of an entire site or directory in Webmaster Tools. To request removal of a directory or site, click on the site in question, then go to Site configuration > Crawler access > Remove URL. If you enter the root of your site as the URL you want to remove, you'll be asked to confirm that you want to remove the entire site. If you enter a subdirectory, select the "Remove directory" option from the drop-down menu.

Reincluding content

You can cancel removal requests for any site you own at any time, including those submitted by other people. In order to do so, you must be a verified owner of this site in Webmaster Tools. Once you've verified ownership, you can go to Site configuration > Crawler access > Remove URL > Removed URLs (or > Made by others) and click "Cancel" next to any requests you wish to cancel.

Still have questions? Stay tuned for the rest of our series on removing content from Google's search results. If you can't wait, much has already been written about URL removals, and troubleshooting individual cases, in our Help Forum. If you still have questions after reading others' experiences, feel free to ask. Note that, in most cases, it's hard to give relevant advice about a particular removal without knowing the site or URL in question. We recommend sharing your URL by using a URL shortening service so that the URL you're concerned about doesn't get indexed as part of your post; some shortening services will even let you disable the shortcut later on, once your question has been resolved.

Edit: Read the rest of this series:
Part II: Removing & updating cached content
Part III: Removing content you don't own
Part IV: Tracking requests, what not to remove

Companion post: Managing what information is available about you online

Dj Matioka

When and why was my site flagged for malware? Learn in near real-time!

Webmaster Level: All

We’ve been hearing this question for many years from webmasters. That’s why we built features such as the Safe Browsing API, the malware review form, and our Malware details Labs feature.

As of today, once we notice your site is infected, we’ll do our best to send an e-mail to the address you have associated with your account in Webmaster Tools. We believe malware is such an important issue for site owners that being quickly informed is beneficial to you and your website’s visitors.

In addition, we’ve promoted our Malware details feature out of Labs and placed it under Diagnostics. The malware data is now updated four times faster than before, we’ve updated our algorithms for identifying injected content, and we’re now able to identify exploits which we were unable to catch earlier.



We hope this allows you to stay up-to-date with any malware issues we detect on your site, and to fix them quickly.

As always, please let us know if you have any feedback or questions about how to fix malware-related issues in our Webmaster Help Forum.




Dj Matioka

More data and charts in Top Search Queries

Webmaster Level: All

We've got good news for site owners who are frequent users of the Top search queries feature in Webmaster Tools: we’re now providing more detailed data for each individual search query. We previously just reported the average position at which your site’s pages appeared in the search results for a particular query. Now you can click on a given search query in the Top search queries report to see a breakdown of the number of impressions and the amount of clickthrough for each position that your site’s pages appeared at in the search results associated with that query. Impressions are the number of times that your site’s pages appeared in the search results for the query. Clickthrough is the number of times searchers clicked on that query’s search results to visit a page from your site. In addition to impressions and clickthrough numbers, you’ll also see a list of your site's pages that were linked to from the search results for that search query. As we went about increasing the amount of data available, we also implemented measures to increase the detail of the data overall.



It used to be that you could only see Top search queries data for your site's top 100 queries. We’ve significantly increased the number of queries we show. Now if your site ranks for more than 100 queries, you’ll see new pagination buttons at the bottom of the Top Search Queries table allowing you to page through a much larger sampling of the queries that return your site in search results.



Previously, if you wanted to visualize your Top search queries data you could download your site's data and generate your own charts. To save you some time and effort, we're now generating a chart for you, and displaying it right within the page.



The Top search queries chart includes a date range selector similar to what Google Analytics offers. So now if you really want to see what your site's top search queries were for a particular week in the past, you can see the data for just that slice in time.



Finally, for sites that have numerous keywords that change frequently, we’ve added the ability to search through your site’s top search queries so that you can filter the data to exactly what you’re looking for in your query haystack.



We hope you enjoy these updates to the Top search queries feature and that it's even more useful for understanding how your site appears and performs in our search results. If you've got feedback or questions about the new Top search queries, please share your thoughts in our Webmaster Help Forum.



Dj Matioka

Updated malware feature in Webmaster Tools

Webmaster Level: All

A little over six months ago we released a new malware diagnostic tool in Webmaster Tools with the help of Lucas Ballard from the anti-malware team. This feature has been a great success; many of you were interested to know if Google had detected malicious software in your site, and you used the tool's information to find and remove that malware and to fix the vulnerabilities in your servers.

Well, a few days ago we promoted the malware diagnostics tool from Labs to a full Webmaster Tools feature. You can now find it under the Diagnostics menu. Not only that, we also added support for malware notifications. As you may already know, if your site has malware we may show a warning message in our search results indicating that the site is potentially harmful. If this is the case, you should remove any dangerous content as soon as possible and patch the vulnerabilities in your server. After you've done that, you can request a malware review in order to have the warning for your site removed. What's new in our latest release is that the form to request a review is now right there with the rest of the malware data:

We've also made several other improvements under the covers. Now the data is updated almost four times faster than before. And we've improved our algorithms for identifying injected content and can pinpoint exploits that were difficult to catch when the feature first launched.

On the Webmaster Tools dashboard you'll still see a warning message when you have malware on one of your sites. This message has a link that will take you directly to the malware tool. Here at Google we take malware very seriously, and we're working on several improvements to this feature so that we can tell you ASAP if we detect that your site is potentially infected. Stay tuned!

For more details, check out the Malware & Hacked sites help forum.

Dj Matioka

URL removal explained, Part IV: Tracking your requests & what not to remove

Webmaster Level: All

In this final installation in our URL removal series, let's talk about following up on your removal requests, as well as when not to use Google's URL removal tool. If you haven't already, I recommend reading the previous posts in this series:
Part I: Removing URLs & directories
Part II: Removing & updating cached content
Part III: Removing content you don't own
Companion post: Managing what information is available about you online

Understanding the status of your requests

Once you've submitted a removal request, it will appear in your list of requests. You can check the status of your requests at any time to see whether the content has been removed, or whether the request is still or pending or was denied.



If a request was denied, you should see a "Learn more" link next to it explaining why that particular request was denied. Since different types of removals have different requirements, the reason why a particular request was denied can vary. The "Learn more" link should help you figure out what you need to change in order to make your request successful. For example, you may need to change the URL in question so that it meets the requirements for the type of removal you requested; or, if you can't do that, you may need to request a different type of removal (one whose requirements your URL currently meets).

If a request has been marked "Removed" but you still see that content in search results, check the following:

• Is the URL that's appearing in search results the exact same URL that you submitted for removal? It's fairly common for the same, or similar, content to appear on multiple URLs on a site. You may have successfully removed one URL, but still see others containing that same content.
     Solution: Request removal of the other URL(s) in question. See this article for help.


• Keep in mind that URLs are case sensitive, so requesting removal of http://www.example.com/embarrassingstuff.html is not the same as requesting removal of http://www.example.com/EmbarrassingStuff.html
     Solution: Request removal of the exact URL(s) that appear in search results, including the same capitalization. See this article for help.


• When a request is marked "Removed," that can mean different things depending on what type of request you submitted. If you requested removal of an entire URL, then "Removed" should mean that that entire URL no longer appears in our search results. If you requested removal of the cached copy of a URL, "Removed" means that the cached copy has been removed and will no longer appear in search results; but the URL itself may still appear.
     Solution: Double-check what type of removal you requested by looking at the "Removal Type" column. If you requested a cache removal but you want the entire URL gone, make sure the URL meets the requirements for complete removal and then file a new request for complete removal of the URL.

When not to use the URL removal tool

• To clean up cruft, like old pages that 404.
  The tool is intended for URLs that urgently need to be removed, such as confidential data that was accidentally exposed. If you recently made changes to your site and just have some outdated URLs in the index, Google's crawlers will see this as we recrawl your URLs, and those pages will naturally drop out of our search results over time. There's no need to request an urgent removal through this tool.


• To remove crawl errors from your Webmaster Tools account.
  The removal tool removes URLs from Google's search results, not from your Webmaster Tools account. There's currently no way for you to manually remove URLs from this report; they will drop out naturally over time as we stop crawling URLs that repeatedly 404.


• To "start from scratch" with your site.
  If you're worried that your site may have a penalty, or you want to "start from scratch" after purchasing a domain from someone else, we don't recommend trying to use the URL removal tool to remove your entire site and then "start over." Search engines gather a lot of information from other sites (such as who links to you, or what words they use to describe your site) and use this to help understand your site. Even if we could remove everything we currently know about your site, a lot of it would come back exactly the same once we'd recrawled all the other sites that help us understand your site and put it in context. If you're worried that your domain has some bad history, we recommend filing a reconsideration request letting us know what you're worried about and what has changed (such as that you've acquired the domain from someone else, or that you've changed certain aspects of your site).


• To take your site "offline" after hacking.
  If your site was hacked and you want to get rid of bad URLs that got indexed, you can use the URL removal tool to remove any new URLs that the hacker created, e.g., http://www.example.com/buy-cheap-cialis-skq3w598.html. But we don't recommend removing your entire site, or removing URLs that you'll eventually want indexed; instead, simply clean up the hacking and let us recrawl your site so that we can reindex the new, cleaned-up content as soon as possible. This article contains more details on how to deal with hacking.


• To get the right "version" of your site indexed.
  When a request to remove https://www.example.com/tattoo.html is accepted, http://www.example.com/tattoo.html is also removed. The same is true of the www and non-www versions of your URL or site. This is because the same content is often available at each of these URLs and we realize that most webmasters and searchers don't want these duplicates appearing in search results. In short, the URL removal tool should not be used as a canonicalization tool. It won't keep your favorite version, it'll remove all versions (http/https and www/non-www) of a URL.

We hope this series has answered your questions about removing content from Google's search results, and helped you troubleshoot any issues that may arise. Join us in our Help Forum if you still have questions.



Dj Matioka

Top Search Queries is now Search Queries with Average Position and Stars

Webmaster Level: All

Since we released the latest version of Top Search Queries in Webmaster Tools we've gotten a bunch of feedback, most of which was overwhelmingly positive. Today, we're happy to bring even more improvements to Top Search Queries that we've implemented as a direct result of your feedback. First of all we've shortened "Top Search Queries" to be just "Search Queries" to better reflect all of the data provided by this feature. In addition to the name change you'll notice that Search Queries has several new updates. As requested by many of you, we're now showing an "Average position" column right on the main Search Queries page. This provides a quick at-a-glance way to see where your site is showing in the search results for specific queries. The other change you'll notice is that we're showing a "Displaying" number for Impressions and Clicks. This number represents a total count of the data displayed in the Search Queries table. The number in bold appearing just above it is a total count of all queries including the "long tail" of queries which are not displayed in the Search Queries table. When the "Displaying" number is not visible, such as when you select a specific country from the "All countries" drop-down menu, then the bold number is the total count of the data displayed in the Search Queries table.



We've also added an Average position column to the Search Queries download.



The other addition we've made to Search Queries is a "Starred" tab. Next to each Query on the Search Queries page there is now a clickable star icon. You can click the star icon for all of the queries that are of most interest to you. All of the queries that you "star" will be consolidated under the Starred tab providing a super easy way to view just the queries you care about.



We hope that this update makes Search Queries even more useful. If you've got feedback or suggestions for Search Queries please let us know in the Webmaster Help Forum.



Dj Matioka

Crawl Errors now reports soft 404s

Webmaster Level: All

Today we’re releasing a feature to help you discover if your site serves undesirable "soft” or “crypto” 404s. A "soft 404" occurs when a webserver responds with a 200 OK HTTP response code for a page that doesn't exist rather than the appropriate 404 Not Found. Soft 404s can limit a site's crawl coverage by search engines because these duplicate URLs may be crawled instead of pages with unique content.

The web is infinite, but the time search engines spend crawling your site is limited. Properly reporting non-existent pages with a 404 or 410 response code can improve the crawl coverage of your site’s best content. Additionally, soft 404s can potentially be confusing for your site's visitors as described in our past blog post, Farewell to Soft 404s.

You can find the new soft 404s reporting feature under the Crawl errors section in Webmaster Tools.



Here’s a list of steps to correct soft 404s to help both Google and your users:

1. Check whether you have soft 404s listed in Webmaster Tools
2. For the soft 404s, determine whether the URL:
   1. Contains the correct content and properly returns a 200 response (not actually a soft 404)
   2. Should 301 redirect to a more accurate URL
   3. Doesn’t exist and should return a 404 or 410 response
3. Confirm that you’ve configured the proper HTTP Response by using Fetch as Googlebot in Webmaster Tools
4. If you now return 404s, you may want to customize your 404 page to aid your users. Our custom 404 widget can help.

We hope that you’re now better enabled to find and correct soft 404s on your site. If you have feedback or questions about the new "soft 404s" reporting feature or any other Webmaster Tools feature, please share your thoughts with us in the Webmaster Help Forum.



Dj Matioka

New Message Center notifications for detecting an increase in Crawl Errors

Webmaster Level: All

When Googlebot crawls your site, it’s expected that most URLs will return a 200 response code, some a 404 response, some will be disallowed by robots.txt, etc. Whenever we’re unable to reach your content, we show this information in the Crawl errors section of Webmaster Tools (even though it might be intentional and not actually an error). Continuing with our effort to provide useful and actionable information to webmasters, we're now sending SiteNotice messages when we detect a significant increase in the number of crawl errors impacting a specific site. These notifications are meant to alert you of potential crawl-related issues and provide a sample set of URLs for diagnosing and fixing them.

A SiteNotice for a spike in the number of unreachable URLs, for example, will look like this:


We hope you find SiteNotices helpful for discovering and dealing with issues that, if left unattended, could negatively affect your crawl coverage. You’ll only receive these notifications if you’ve verified your site in Webmaster Tools and we detect significant changes to the number of crawl errors we encounter on your site. And if you don't want to miss out on any these important messages, you can use the email forwarding feature to receive these alerts in your inbox.

If you have any questions, please post them in our Webmaster Help Forum or leave your comments below.

Dj Matioka

To err is human, Video Sitemap feedback is divine!

Webmaster Level: All

You can now check your Video Sitemap for even more errors right in Webmaster Tools! It’s a new Labs feature to signal issues in your Video Sitemap such as:

• URLs disallowed by robots.txt
• Thumbnail size errors (160x120px is ideal. Anything smaller than 90x50 will be rejected.)

Video Sitemaps help us to better crawl and extract information about your videos, so we can appropriately feature them in search results.

Totally new to Video Sitemaps? Check out the Video Sitemaps center for more information. Otherwise, take a look at this new Labs feature in Webmaster Tools.



Dj Matioka

Verification time savers — Analytics included!

Webmaster Level: All

Nobody likes to duplicate effort. Unfortunately, sometimes it's a fact of life. If you want to use Google Analytics, you need to add a JavaScript tracking code to your pages. When you're ready to verify ownership of your site in other Google products (such as Webmaster Tools), you have to add a meta tag, HTML file or DNS record to your site. They're very similar tasks, but also completely independent. Until today.

You can now use a Google Analytics JavaScript snippet to verify ownership of your website. If you already have Google Analytics set up, verifying ownership is as simple as clicking a button.


This only works with the newer asynchronous Analytics JavaScript, so if you haven't migrated yet, now is a great time. If you haven't set up Google Analytics or verified yet, go ahead and set up Google Analytics first, then come verify ownership of your site. It'll save you a little time — who doesn't like that? Just as with all of our other verification methods, the Google Analytics JavaScript needs to stay in place on your site, or your verification will expire. You also need to remain an administrator on the Google Analytics account associated with the JavaScript snippet.

Don't forget that once you've verified ownership, you can add other verified owners quickly and easily through the Verification Details page. There's no need for each owner to manually verify ownership. More effort and time saved!


We've also introduced an improved interface for verification. The new verification page gives you more information about each verification method. In some cases, we can now provide detailed instructions about how to complete verification with your specific domain registrar or provider. If your provider is included, there's no need to dig through their documentation to figure out how to add a verification DNS record — we'll walk you through it.


The time you save using these new verification features might not be enough to let you take up a new hobby, but we hope it makes the verification process a little bit more pleasant. As always, please visit the Webmaster Help Forum if you have any questions.



Dj Matioka

New ways to view Webmaster Tools messages

Webmaster Level: All

Now there’s a new way to see just the messages for a specific site. A new Messages feature will appear on all site pages. The feature is just like the Message Center on the home page, except it‘ll show only messages for the currently selected site. This gives you more freedom to choose how you want to view your messages: either for all your sites, or for just one site at a time.


Alerts (formally known as SiteNotice messages) will now be more prominent in the Message Center. These messages tell you about significant changes we’ve noticed related to your site which may indicate serious problems. For instance, alerts may warn you about an increase in crawl errors, an increase in 404 errors, or about possible outages. With their newfound prominence comes a new name: what used to be “SiteNotice messages” will now simply be known as “alerts.”

Messages containing alerts will be marked with an icon to make them quickly distinguishable from other messages. Each site’s Dashboard will display a notification whenever the site has unread alerts. The Dashboard notification will lead to the new site Message Center with a filter enabled to show only alerts for the current site.


You can also enable the alerts filter yourself. On the home page, enabling the alerts filter across all your sites is a great way to see alerts you may have missed and may help you find problems common across multiple sites. Even with these changes we recommend you use the email forwarding feature to receive these important alerts without having to visit Webmaster Tools.

We hope these new features make it easier to manage your messages. If you have any questions, please post them in our Webmaster Help Forum or leave your comments below.

Dj Matioka

Google Instant: Impact on Search queries

Webmaster Level: All

Webmasters, you may notice some changes in you Search queries data due to the launch of Google Instant. With Google Instant, the page updates dynamically to show results for the top completion of what the user has typed, so this means people could be seeing and visiting your website much faster than before, and often without clicking the search button or hitting “enter.”

While the presentation of the search results may change, our most important advice to webmasters remains the same: Users want to visit pages with compelling content and a great user experience.

With Google Instant, you may notice an increase in impressions because your site will appear in search results as users type.

Impressions are measured in three ways with Google Instant:

1. Your site is displayed in search results as a response to a user’s completed query (e.g. by pressing “enter” or selecting a term from autocomplete). This is the traditional model.
   
   With Google Instant, we also measure impressions in these new cases:


2. The user begins to type a term on Google and clicks on a link on the page, such as a search result, ad, or a related search.


3. The user stops typing, and the results are displayed for a minimum of 3 seconds.

To give an example, let’s say your site has lots of impressions for [hotels] and [hotels in santa cruz]. Now, because Instant is quickly fetching results as the user types, the user could see your site in the search results for [hotels] after typing only the partial query [hote]. If a user types the partial query [hote] and then clicks on any result on the page for [hotels], that counts as an impression for your site. That impression will appear in Webmaster Tools for the query [hotels]. The term 'hotels' would also be included in the HTTP referrer when the user clicks through to visit your website.

It’s likely that your site will still see impressions for queries like [hotels in santa cruz], but because Instant is helping the user find results faster, your site may see an increase in impressions for shorter terms as well.

Let us know if we can help you better understand how these changes impact Webmaster Tools, measured Search queries and impressions, CTR, or anything else. We’d love to hear from you in our blog comments or Webmaster Help Forum.

Dj Matioka

Webmaster Tools - Links to your site updated

Webmaster Level: All

The "Links to your site" feature in Webmaster Tools is now updated to show you which domains link the most to your site, in addition to other improvements. On the overview page you'll notice that there are three main sections: the domains linking most to your site, the pages on your site with the most links, and a sampling of the anchor text external sites are using when they link to your site.


Who links the most
Clicking the “More »” link under the “Who links the most” section will take you to a new view that shows a listing of all the domains that link to your site. Each domain in the list can be expanded to display a sample of pages from your site which are linked to by that domain.


The "More »" link under each specific domain lists all the pages linked to by that domain. At the top of the page there's a total count of links from that domain and a total count of your site's pages linked to from that domain.


Your most linked content
If you drill into the “Your most linked content” view from the overview page, you’ll see a listing of all your site’s most important linked pages. There's also a link count for each page as well as a count of domains linking to that page. Clicking any of the pages listed will expand the view to show you examples of the leading domains linking to that page and the number of links to the given page from each domain listed. The data used for link counts and throughout the "Links to your site" feature is more comprehensive now, including links redirected using 301 or 302 HTTP redirects.


Each page listed in the "All linked pages" view has an associated "More »" link which displays all the domains linking to that specific page on your site.


Each domain listed leads to a report of all the pages from that domain linking to your specific page.


We hope the updated “Links to your site” feature in Webmaster Tools will help you better understand where the links to your site are coming from and improve your ability to track changes to your site’s link profile. Please post any comments you have about this updated feature or post your questions in the Webmaster Help Forum. We appreciate your feedback since it helps us to continue to improve the functionality of Webmaster Tools.


Dj Matioka