How Google AdSense Crawler Works Plus Simple Setup Guideline

Just like how the Google AdSense Crawler works, there are many other forms of web crawlers with specific functions. A Web Crawler is a piece of software code used to collect the information on webpages available on the internet and index them in the database. Google has plenty of crawlers for collecting different types of information on the web.

Specifically, Google AdSense Crawler is one of Google’s web crawlers collection inventory lists — it’s used to collect the details of webpage content in order to serve relevant ads on that page. In this article, we will explain more detail about the Google AdSense crawler and how it works. Precisely, the crawler report is automatically performed and updated weekly.

Making sure your site is fully crawlable can help you earn more revenue from your content. If the AdSense ads crawler can’t access your content, you can refer to some crawler issues guide to help you fix the problem. For instance, if your website is behind a login, it’s possible that the ads crawler is being redirected to a login page, which means it can’t crawl your content.

What if your content is behind a restricted network or IP range? Have you put restrictions in place that limit the geographies or IP ranges that can access your content? If that’s the case, you should consider removing these restrictions or making your content publicly accessible, so that the ads crawler can crawl your URLs. These are some of the uses of ads crawler.

What Google AdSense Crawler Is All About

By definition, a crawler, also known as a spider or a bot, is the software Google uses to process and index the content of web pages. Google AdSense Crawler is a toolkit for indexing those pages that display AdSense advertisements. The AdSense crawler visits your site to determine its content in order to provide relevant ads. This helps to serve relevant ads to consumers.

And, it’s quite different from the web crawlers used for displaying search engine results. Besides the Google AdSense Crawler, below are some of the other main and notable crawlers (User-Agents) used by Google that you should also know about:

Crawler Name	The Main Purpose
Googlebot-Web	Used to index content for showing in Google web search results. This is also the same crawler used for smartphones.
Googlebot-Image	Used to index images for showing in Google image search results.
Googlebot-News	Used to collect news feeds for showing in Google news search results.
Googlebot-Video	Used to crawl videos on the web for showing in video search results.
Googlebot-Mobile	Used for Google mobile search on feature phones.
Mediapartners-Google	Used for indexing web page content for displaying relevant Google AdSense ads.

As we aforementioned, the Google AdSense Crawler is different from the Google Website Crawler. The two crawlers are separate, but they do share a cache. This is to avoid both crawlers requesting the same pages, thereby helping publishers conserve their bandwidth. Similarly, the Search Console crawler is separate. Also, the crawler indexes by URL.

Whereby, the Google crawler will access site.com and www.site.com separately. However, the crawler will not count site.com and site.com/#anchor separately. The crawler will attempt to access URLs only where the ad tags are implemented. Only pages displaying Google ads should be sending requests to Google systems and being crawled.

Similarly, the crawler will attempt to access pages that redirect. Meaning, that when you have “original pages” that redirect to other pages, the crawler must access the original pages to determine that a redirect is in place. Therefore, the crawler’s visit to the original pages will appear in your access logs.

It’s also, important to realize, that there is no control over how often the crawler will index your site content. At this time, Google has no control over re-crawling sites. Since all crawling is done automatically by its bots. If you make changes to a page, it may take up to 1 or 2 weeks before the changes are reflected in the Google index.

According to webnots, unlike Googlebot which crawls all the pages, there’s some distinct use of AdSense crawler.

Consider the following key uses:

AdSense crawlers will only attempt to crawl the pages on which ad codes are placed.
You can monitor AdSense crawler errors from your AdSense account under the “Crawler Access” section.
Remember correcting crawler errors for AdSense will not have any impact on the crawler errors shown in Google Search Console since they are from two different bots.
Google does not allow controlling the frequency of AdSense crawlers. The frequency is automatic and any changed content may take 1-2 weeks for getting indexed.
https://yoursite.com and https://www.yoursite.com are considered two different sites and the redirected URLs are also crawled.
AdSense crawler follows the robots.txt directives similar to all other Google bots.
You can also provide access to password-protected pages on your site for displaying ads. Learn more about this here.

Is your site up and running? Can visitors access your site on the web? Make sure your site is published and live on the web.

Access Issues:

the site is behind a login.
content is behind a restricted network or IP range.
you’ve blocked the ads crawler in your robots.txt file.

Website Issues:

can’t be found.
has freshly published content.
uses multiple redirects.
you’re using personalized page URLs.
you’re using POST data.

Hosting Issues:

There’s an issue with your site’s name server.
There’s an issue with your site’s server.

Resolving the issues listed on your Crawler access page will have no impact on your placement within Google search results. For more information on your site’s ranking on Google, review the entry on getting included in Google search results in detail.

Do visitors need login details to access your content? If your answer is yes, you’ll need to set up a crawler login to display ads on login-protected pages. You can read and learn more about how to solve and fix AdSense crawler issues in more detail.

Steps To Display Ads On Login-Protected Pages

After your AdSense account has been activated, you can display Google ads on pages behind a login by creating a crawler login. This will give the AdSense ads crawler access to visit your pages and deliver ads. To do so, you’ll need to provide login details for your website and verify your site using Search Console as well.

Please follow these steps.

Sign in to your AdSense account.
Click Account.
Click Access and authorization, then click Crawler access.
In the “Crawler access” section, click Add login.
In the “Login Information” section, provide the login details that our crawler will use to access your site:
- Restricted directory or URL: The URL that is blocked from our crawler’s access.
- Login URL: The URL that our crawler should visit to sign in.
- Login method: The method your site uses to authenticate users who log in. POST and GET are common authentication methods.
- Login parameters: The credentials that our crawler will use to sign in (for example, your site’s username and password). For GET and POST methods, please enter the URL parameter key-value pairs such that your server will return a cookie for logged-in access.
Click Add login.
Sign in to Search Console using your Google Account. You must be using your Google Account in order for any changes you make in Search Console to be reflected in your AdSense account.
Add your site to your Search Console account by following these instructions. Make sure you add the site that you want to allow our crawler to access.

And now, after you’ve verified your site, the login details you’ve provided should allow the AdSense crawler to reach your pages and provide ads. Google is also introducing Publishers on Air: A new video platform where AdSense and Ad Manager experts come to you. Stay up-to-date with new products, best practices, and expert advice via livestreams (click here).

Controlling Google AdSense Crawler With Robots.txt

Your site can’t be found. If the URL sent to Google points to a page that doesn’t exist (or no longer exists) on a website or results in a 404 (Not Found) error, the ads crawler won’t successfully crawl your content. Have you blocked the Google Ads Crawler in your robots.txt file? Is the Google AdSense Crawler disallowed in your robots.txt file?

If you’ve modified your site’s robots.txt file to disallow the AdSense crawler from indexing your pages, then Google is not able to serve ads on these pages. To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your robots.txt file.

Consider the following:

User-agent: Mediapartners-Google Disallow: /

This change will allow the ads crawler to index the content of your site and provide you with Google ads. Please note that any changes you make to your robots.txt file may not be reflected in the Google index until its crawlers attempt to visit your site again. So, in order for the ads to be served on your website, you should give access to the ads crawler in your robots.txt file.

On the contrary, just like in the above line of code, you can use the same directive in your robots.txt file to block the AdSense crawler. Both the Google and AdSense Mediapartners crawlers honor your robots.txt file. If your robot.txt file prohibits access to certain pages or directories, then they will not be crawled.

Note: If you’re serving ads on pages that are being roboted out with the line User-agent: *, then the AdSense crawler will still crawl these pages. To prevent the AdSense crawler from accessing your pages, you need to specify User-agent: Mediapartners-Google in your robots.txt file.

You can also block Googlebot to avoid pages showing on the search results, at the same time, allow AdSense crawler to show ads on the website. With that in mind, in order to avoid all this hustle, there’s an easier way out for webmasters. And, that’s none other than installing and activating the right tool for this work — Google Site Kit Plugin for WordPress websites.

The good news is that anyone with a WordPress website can now easily Install Site Kit by Google. May it be for a small or large business site, or even for a newbie to an advanced webmaster, Google Site Kit can significantly help. And now, in this guide, we are going to explain in detail how and why. At the end of October, Google announced the release of its new plugin.

Install And Setup Google Site Kit Plugin For Your WordPress Website

So, are you a WordPress user looking to monetize your site with AdSense? There are several ways to place AdSense ad code into your WordPress site. If you’re new to AdSense, one of your first tasks is to connect your site to AdSense. We highly recommend you use Site Kit for WordPress by Google. Site Kit can connect your WordPress website to your AdSense.

And then, it can now easily and quickly place the AdSense code on all your website pages for you — to start showing ads automatically across all your website blog posts and web pages. Site Kit is a free-of-charge, open-source plugin for WordPress that helps you access unified metrics and insights from all its other complimentary solution services for webmasters.

Including but not limited to Google AdSense, Google Analytics, Google Search Console, and PageSpeed Insights directly from your WordPress account. In order to successfully install and use Site Kit, your website must meet certain requirements. This is, especially, important for beginner webmasters to note so that they’ll know the reason why the plugin fails on their website.

Consider the following requirements:

WordPress version 4.7 or above plus PHP version 5.6+
A modern web browser like the Edge, Chrome, or Firefox – Internet Explorer is not currently supported
Is publicly accessible – it isn’t in maintenance mode, accessible only via password, or otherwise blocked
REST API is available – Site Kit must be able to communicate via REST API with Google services.
- To ensure that the REST API is available for your site, go to Tools > Site Health.

Currently, only users with admin-level access to WordPress can set up and use the Site Kit plugin. In that case, this guide explains the permission levels for Google tools when used with Site Kit. Such as Google Search Console, Google Analytics, Tag Manager, or even AdSense. To Install Site Kit By Google, firstly, you’ll need to log in to your WordPress website.

And then, secondly, make sure that your website meets all the mandatory requirements. Such as the website should be publicly accessible. Meaning, that it isn’t under any kind of maintenance mode or only accessible through a password. Similarly, you should also make sure that you’re using a staging environment in addition to your site production authoring.

Follow these simple setup guidelines:

Go to your WordPress dashboard, and click Add New under the “Plugins” menu.
Search for “Site Kit by Google”, find the plugin, and click Install Now.
After you’ve installed the plugin, click Activate.
Click Start Setup and then follow the steps to set up the Site Kit.
After you’ve completed the setup, click Connect Service under “AdSense”. You can find this on the Site Kit dashboard or “Settings” tab.
Follow the instructions in the plugin.

When you click on the Start Setup button in the previous step, a new tab opens, loading the Google Site Kit configuration page. Click on Sign in with Google to continue.

When your site is ready to show ads and has traffic, you’ll see your AdSense metrics in the “AdSense” tab of the plugin. Usually, we recommend that you allow Site Kit to place the AdSense code, which will place the AdSense code for non-AMP and AMP versions of your site.

Some Keynotes To Bear In Mind:

In order to ensure that AdSense ads serve your website right, make sure you’ve enabled Auto ads in your AdSense account. In particular, since Site Kit does not have permission to make changes to your AdSense account settings. And, when your website is ready to show ads and has traffic, you’ll see your AdSense metrics in the “AdSense” tab of the plugin.
It’s, important to realize, that if you have visitors from the European Economic Area (EEA) or the UK, you might want to uncheck this option to make sure that you’re complying with Google’s EU user consent policy. You can change your settings for serving ads to users in the EEA and the UK after your AdSense account is approved.

Sometimes, your AdSense ads may only display on the homepage and not show on other pages of your website. Especially, after you Install Site Kit Plugin By Google and then after activating it on your WordPress website. Of course, the toolkit is great and it blocks all recommended bad bots, right on.

The Main Benefits To Note Using The Google Site Kit Plugin

Site Kit is the official WordPress plugin from Google for insights about how people find and use your site (learn more). It’s the one-stop solution to deploy, manage, and get insights from critical Google tools to make the site successful on the web. It provides authoritative, up-to-date insights from multiple Google products directly on the WordPress dashboard.

That said, as you learn how to install Site Kit by Google in simple steps, you’ll notice that includes powerful features that make using Google products seamless and flexible. Whilst not intended for in-depth data reporting, it’s useful for providing developers with a quick and easy data-based website performance overview.

You’ll Get Data Access For:

Google Search Results Console
GA (Google Analytics) Property (V4)
Google Optimize Plus Tag Manager
PageSpeed Insights Toolkits
Google AdSense Platform

It’s also important to realize, there is even a complete guide on how to use Site Kit with a staging environment. But, it’s only for sites that have a staging environment and a production site. Whereby, the staging site displays the production site data. Some of its key features include easy-to-understand stats directly through your WordPress website admin dashboard.

Additionally, you’ll also have official stats from multiple Google tools, all in one dashboard, and a quick setup for various Google tools without having to edit the source code of your site. There are also metrics for your entire site and for individual posts. It also comes with an easy-to-manage, granular permissions across WordPress and different Google products, etc.

What’s more, with actionable tips from PageSpeed Insights, you’ll also be able to improve performance. As you also track how many people saw your site in search results. And even know what query they used to search for your website. In addition, when you Install Site Kit by Google, it’ll help you to show key metrics and insights from all other Google products.

Summary Thoughts:

WordPress’ ubiquity and market dominance have made goliaths like Google release official plugins for the CMS. Today we’re looking at a more open, more inclusive integration of a variety of third-party services within a WordPress website, which ultimately benefits the end user. Google’s Site Kit is a brilliant example of this inclusive integration tool.

Going forward, it’d be great to see deeper integrations with YouTube Analytics and Google Ads, since a lot of online businesses spend considerable amounts of money on ads for paid content marketing and promotion. In other words, the Google Site Kit promises to make it easier for website owners to access and manage key services of their websites.

They also conduct analytics to increase the success of their online properties. So, this plugin can be particularly useful when you’re a WordPress developer. Having said that, you can learn more about Google tools that are right for you from its webmaster central blog. Sometimes, you may also decide to make your staging site live (move your site to production).

So, be that as it may, if you go along with your business and you get stuck, you can either Uninstall the Helper Plugin, try to Reset the Site Kit plugin, or even Uninstall Site Kit Plugin from your WordPress website altogether — that’s if you no longer wish to continue using it on your website. But, you can always Consult Us for more support before you can do that.