Respect the file limits
A single sitemap has documented ceilings of 50000 URLs and 50 MB uncompressed: both limits apply. Split larger sets across sitemaps, optionally referenced by an index. Use UTF-8 and fully qualified URLs.
| Check | Benchmark | Detail |
|---|---|---|
| URLs per sitemap | 50000 | Ceiling for one file. |
| Uncompressed size | 50 MB | This ceiling also applies. |
| Encoding | UTF-8 | With correctly escaped XML values. |
| priority and changefreq | Ignored by Google | Do not control its crawling. |
Describe the pages actually intended
- List preferred URLs consistent with canonicals, accessible and intended for indexing. Identify redirects, errors and excluded pages before submission.
- Escape XML values correctly and inspect the served file, status and child sitemaps. Compression does not remove the uncompressed-size limit.
- Use lastmod for a significant, verifiable change. Do not refresh every date daily to simulate updates.
Separate accepted files from indexed pages
Monitor parsing diagnostics and URL states separately. A valid sitemap can include a page not selected for other reasons. Google ignores priority and changefreq; these fields cannot buy priority or set crawl frequency.
Primary references
Put intentional URLs in a sitemap
A sitemap is a discovery inventory for URLs the site intends crawlers to visit. Redirects, duplicates and noindex pages weaken coherence, yet even a valid entry does not guarantee indexing.
- Compare each entry with its final response and canonical.
- Update lastmod only after a real change.
- Check language alternates and internal links for key pages.
Primary reference: Google Search Central — Crawling and indexing. Related method: read the practical guide.
Do not use sitemaps to compensate for broken navigation
Sitemaps can signal records missing from pagination links but cannot make the reader journey usable. Keep canonical URLs, lists and sitemaps consistent, then check discovery beyond page one.
- Include relevant canonical URLs according to actual content.
- Test continuation links and records on each family’s final page.
- Keep modification dates meaningful rather than changing them whenever a sitemap rebuilds.
Acceptance case to reproduce
Fictional example: these inputs describe no customer or observed result.
View case inputs
{
"page_2": "/catalog?page=2",
"canonical": "/catalog",
"next_link": "<button>More</button>"
}Expected decision
Page 2 needs its own canonical and crawlable href links to subsequent pages.
Verification procedure
- Record each page’s URL, canonical and links.
- Reach subsequent content without a JavaScript button.
- Keep crawlable href links and the self-canonical. Open each URL directly and compare displayed items; a different address repeating the first page does not validate continuation.
Google Search Central — Pagination and incremental page loading
Primary reference: Google Search Central — Pagination and incremental page loading. Related method: read the practical guide.
Sitemaps: verify URLs, shards and lastmod dates
Check the file offered to crawlers beyond simple XML syntax validity.
- Canonical, language and published state
- Uncompressed bytes and count
- Change source and check
Acceptance case to reproduce
Fictional example: these inputs describe no customer or observed result.
View case inputs
{
"shard_urls": 50001,
"uncompressed_bytes": 20000000,
"lastmod_source": "generation_time"
}Expected decision
50,001 URLs exceed the per-file limit. Split the sitemap and derive lastmod from meaningful changes. In this exercise, files containing 50,000 and 1 URL meet the count limit; also check the 50 MB limit per uncompressed file. Gzip does not raise these limits. The generation date does not replace each page’s modification date.
Verification procedure
- Record URL count and uncompressed size.
- Validate shards, canonicals and meaningful lastmod dates.
- Keep exclusions and shard-to-index mappings.
Primary reference: Sitemaps.org — Protocol. Related method: read the practical guide.
Robots.txt: crawling, indexing and evidence
A sitemap proposes URLs; it does not decide the index
For Google, sitemap inclusion is a weak canonical signal. A proposed URL still needs to be crawled, processed and potentially selected.
- Publish useful URLs consistent with canonical signals and internal links.
- Check robots.txt and each page’s directives separately.
- Retain evidence of actual indexing; submitting a sitemap is not proof of an outcome.
Review the twelve robots.txt cases →
References reviewed