google-crawling-indexing

Files (3)

• SKILL.md
• google-search-central-summaries.md · references/
• google-search-central-toc.md · references/

SKILL.md (first lines)

# Google Crawling and Indexing

## Description

This skill covers the *Crawling and indexing* section of Google Search Central. It explains how Google discovers, crawls, renders, and indexes web content, and it gives you practical controls you can use to guide each stage of that process. The guidance here is built directly on the official English Google Search Central documentation, so the focus is on actions that work: building and submitting sitemaps, writing a correct `robots.txt` file, using `meta` robots tags and canonical links, implementing server-side redirects, optimizing JavaScript sites, managing crawl budget, and removing content safely.

> The supporting English summaries for this skill are in [`references/google-search-central-summaries.md`](references/google-search-central-summaries.md), and a structured index of every page is in [`references/google-search-central-toc.md`](references/google-search-central-toc.md).

## Usage

### When to invoke

- The user asks how Google crawls, renders, and indexes pages.
- The user needs to create, validate, or submit a sitemap.
- The user wants to configure `robots.txt` or `meta` robots tags.
- The user has duplicate content and needs canonical URLs.
- The user is migrating a site or renaming URLs and needs redirects.
- The user needs SEO guidance for a JavaScript site, single-page app, or lazy-loaded content.
- The user wants to remove or block content from Google.
- The user needs to request a recrawl, manage crawl budget, or fix crawl errors.

### Input

- A specific topic or problem from the crawling and indexing section.
- Optional: a site URL, a `robots.txt` file, a sitemap, or page markup to review.

### Output

- A structured answer with direct links to the relevant Google Search Central pages.
- Step-by-step actions and concrete examples.
- Common pitfalls to avoid and how to verify the fix.

---

## Core concepts

### 1. Crawling and indexing overview

Google Search has three overlapping stages: crawling, indexing, and serving search results. Understanding the difference between them is the foundation of every technical SEO decision.

**Crawling** is the automated discovery process. Googlebot, which uses a smartphone user agent for mobile-first indexing, starts from a list of known URLs and follows links and sitemaps to find new pages. It also handles rendering: Googlebot fetches the HTML, then runs JavaScript with an evergreen version of Chromium so that client-side content can be analyzed. Crawling is governed primarily by `robots.txt`, by the availability and crawlability of links, and by the site's crawl budget.

**Indexing** is the analysis and storage stage. After a page is crawled and rendered, Google processes the content, evaluates quality and relevance, and may add it to the index. A page that is crawled is not necessarily indexed, and a page that is indexed is not necessarily shown for any given query. To prevent a page from being indexed, use a `noindex` rule or password protection; `robots.txt` alone does not prevent indexing.

**Serving** is the result-selection stage. When a user searches, Google retrieves relevant pages from the index and ranks them. Ranking is influenced by relevance, authority, structured data, page experience, and many other signals, but those are outside the scope of this skill. What matters here is that only publicly accessible, properly canonicalized, and technically sound pages can compete.

Key principles to keep in mind:

1. Google only indexes public, accessible pages. If a page requires authentication, Googlebot generally cannot see it.
2. `robots.txt` controls which URLs Googlebot may crawl, but it does not prevent a URL from appearing in search results if other sites link to it.
3. `noindex` prevents a page from being shown in search results, but for it to work, the page must be crawlable so that Google can see the directive.
4. Duplicate or near-duplicate content should be canonicalized so that signals such as links are consolidated on the preferred URL.
5. Google uses the mobile version of a page's content for indexing, so desktop and mobile users should see the same core content and metadata.

For a high-level explanation of the whole system, see [How Google Search works](https://developers.google.com/search/docs/fundamentals/how-search-works?hl=en). For the crawling and indexing landing page, see [Overview of crawling and indexing topics](https://developers.google.com/search/docs/crawling-indexing?hl=en).

### 2. Sitemaps

A sitemap is a file that tells Google which pages, videos, images, and other files on your site are important, and it can provide metadata such as the last modification date, change frequency, and alternate language versions. Sitemaps are not a requirement for being indexed, but they are especially valuable when:

- Your site is large, making it hard to ensure every page has internal links.
- Your site is new and has few external links.
- You publish rich media, news, or localized content and want to give Google extra metadata.
- You have isolated or deep pages that are not well linked from the rest of the site.

Google supports three sitemap formats, and none is universally preferred; choose the one that fits your publishing workflow:

- **XML sitemaps** are the most versatile. They support extensions for images, video, news, and localized URLs. Most content management systems and plugins generate them automatically. The trade-off is that they can be more complex to maintain on very large or frequently changing sites.
- **RSS, mRSS, and Atom 1.0** feeds are convenient because many CMSes create them automatically. They are useful for frequently updated content and can include video metadata, but they cannot include image or news sitemap data.
- **Plain text sitemaps** are the simplest format: one URL per line. They are easy to maintain but limited to indexable textual content and provide no metadata.

A single sitemap file must be no larger than 50 MB uncompressed and must contain no more than 50,000 URLs. If you exceed either limit, split the file into multiple sitemaps and use a sitemap index file to list them. The sitemap file must be UTF-8 encoded and should use fully qualified, absolute URLs. A sitemap placed at the site root, such as `https://www.example.com/sitemap.xml`, can affect all files on the host, while a sitemap in a subdirectory can only affect descendants of that directory unless it is explicitly submitted through Search Console.

A minimal XML sitemap looks like this:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/foo.html</loc>
    <lastmod>2022-06-04</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/bar.html</loc>
    <lastmod>2022-06-04</lastmod>
  </url>
</urlset>
```

Sitemap extensions let you give Google additional metadata beyond the page URL. An [image sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/image-sitemaps?hl=en) can list images that might not otherwise be discovered, such as those loaded by JavaScript. A [video sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/video-sitemaps?hl=en) can specify the title, description, running time, and age-appropriateness of videos. A [News sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/news-sitemap?hl=en) is for articles approved for Google News and should contain only news articles published in the last two days. You can also list [localized versions of a page](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap?hl=en) using `xhtml:link rel="alternate"` entries inside each URL. If you have many sitemaps, create a [sitemap index file](https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps?hl=en) that lists each individual sitemap and submit only the index.

Sitemap best practices include:

1. Include only canonical URLs that you want to see in search results.
2. Use image, video, or news extensions when you have that content.
3. Submit the sitemap through the [Sitemaps report in Search Console](https://search.google.com/search-console?hl=en), or reference it in `robots.txt` with a `Sitemap:` directive.
4. Keep sitemaps up to date; remove URLs that no longer exist and use accurate `<lastmod>` values.

Common mistakes with sitemaps include submitting non-canonical URLs, including pages that are blocked by `robots.txt` or marked `noindex`, and allowing the file to grow beyond the size limits. See the official guidance on [Learn about sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview?hl=en) and [Build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap?hl=en).

### 3. robots.txt

A `robots.txt` file tells search engine crawlers which URLs they can access. Its primary purpose is to avoid overloading your site with crawler requests; it is **not** a security mechanism and it **does not** keep a page out of Google Search results.

The file must be placed in the top-level directory of the site, on a supported protocol. For Google Search, the supported protocols are HTTP, HTTPS, and FTP. A `robots.txt` file on `https://www.example.com/robots.txt` controls crawling for `https://www.example.com/` only; it does not apply to subdomains, other protocols, or other port numbers. Each rule group begins with a `User-agent` line, followed by `Disallow` and `Allow` lines. The most specific matching rule applies, and groups are processed from top to bottom.

A typical `robots.txt` file looks like this:

```
User-agent: *
Disallow: /includes/

User-agent: Googlebot
Allow: /includes/

Sitemap: https://www.example.com/sitemap.xml
```

In this example, all crawlers are blocked from the `/includes/` directory, but Googlebot is allowed because the more specific group overrides the general one. The `Sitemap:` directive points Google to the sitemap file.

From MathiasPaulenko/ai-toolkit, MIT.

More from MathiasPaulenko/ai-toolkit