Robots.txt: separate crawling, indexing and access control

Published · Updated · SEO, SXO and visibility · 3 min read · By Cyber Intelligence Embassy

At a glance

Robots.txt gives crawl rules to crawlers that follow them. It does not restrict human access or guarantee removal of an address from Google.

Empty digital forensics bench with storage devices and precision tools Illustration · fictional scene
In this article

Read the rule against the actual URL

Here, the general group disallows /internal-search/ but permits /internal-search/help/. The paths are illustrative. Sitemap supplies a discovery address rather than an indexing instruction. Check whether a more specific crawler group applies before using the general rules.

User-agent: *
Disallow: /internal-search/
Allow: /internal-search/help/
Sitemap: https://example.com/sitemap.xml

Choose the control for the goal

  • Private resources need server-side authentication and authorization. Listing a path in robots.txt does not make it private.
  • To exclude a public page from indexing, provide a readable noindex directive. Blocking its crawl can prevent the crawler from seeing that instruction.
  • To reduce unnecessary public-page crawling, target the required paths and keep resources needed for rendering accessible.

Check the response actually served

Inspect the file at the root of the relevant protocol, host and port, then test exact URLs, case and applicable groups. Use the robots.txt report and URL Inspection in Search Console for available observations. A change may need another crawl before a visible search effect appears.

Primary references

Test the access path before changing robots.txt

A robots.txt rule governs crawling, not a guaranteed removal from the index. Blocking a page that must be fetched so a noindex directive can be read can create a contradictory signal.

  • Test the exact URL and applicable crawler group.
  • Compare robots rules, HTTP response and page directives.
  • Replay discovery through HTML links after each change.

Primary reference: Google Search Central — Crawling and indexing. Related method: read the practical guide.

Separate HTTP response from rendered content

Compare text, links and directives in initial HTML and rendering. Record dependencies on clicks, sessions or APIs. Google can execute JavaScript, but successful rendering does not guarantee indexing or visibility in other assistants.

  • Type, status and capture date
  • Text and links in initial response and rendering
  • Scripts, APIs, sessions and required interactions

Acceptance case to reproduce

Fictional example: these inputs describe no customer or observed result.

View case inputs
{
    "response_status": 200,
    "html_link": "<span onclick=\"navigate()\">Item</span>",
    "rendered_text": true,
    "script_failure_test": true
}

Expected decision

A clickable span is not an href link. Check discovery and content when scripts fail.

Verification procedure

  1. Capture received HTML, rendered content and href links.
  2. Disable scripts and test essential navigation.
  3. Keep text, links, canonical and robots in both states.

Google Search Central — JavaScript SEO basics

Does Google execute JavaScript?

Yes, with limits and a rendering phase. Check observed outcomes on your URLs rather than assuming everything is processed.

Primary reference: Google Search Central — JavaScript SEO basics. Related method: read the practical guide.

URL migration: prepare mappings and test redirects

Change domain or URL structure while preserving useful destinations for old pages and their language editions.

  • Old URL and language
  • Retention, merger or justified removal
  • Final target, status, canonical and alternatives

Primary reference: Google Search Central — Site moves with URL changes. Related method: read the practical guide.

Robots.txt: crawling, indexing and evidence

Read groups before selecting the longest rule

Groups for the same token combine; * is a fallback and is not added to a specific group. Line order is not a general priority rule.

  • Identify applicable groups before matching rules against the path.
  • Use the most specific match; for equivalent rules the RFC recommends Allow.
  • Do not infer deindexing or access protection from Disallow alone.

Review the twelve robots.txt cases →

References reviewed