Resources · 80

AI search: verify visibility and publication controls

Separate crawling, indexing, previews and AI service usage before changing site directives.

· 4 min

Global network map in an empty intelligence briefing room AI-generated editorial illustration

What this guide helps achieve

  • Separate uses
  • Inspect effective settings
  • Follow current requirements
  • Check distribution

Quick check

  • Can one file control all AI visibility?
  • Must useful FAQs disappear after rich-result retirement?
  • Will a correctly configured page be cited?

Step-by-step method

  1. 01

    Separate uses

    Define what should be allowed: human access, crawling, indexing, snippets, citations or other documented service usage. One crawler rule is not a universal policy for every assistant. Identify useful public content and data requiring access control.

    Deliverable: service, use and decision matrix.

  2. 02

    Inspect effective settings

    Record HTTP status, robots.txt, HTML and header directives, canonical URLs and infrastructure restrictions. Directives must be observable to the relevant crawler. Distinguish crawl blocking from index removal; protect private content with authentication and authorization.

    Deliverable: captures of actually served responses.

  3. 03

    Follow current requirements

    Consult service documentation and its changes. Google states that llms.txt is not used for Search ranking and special AI markup is unnecessary. Check applicable Search Console settings too. Do not promise citations based on a file or valid schema.

    Deliverable: dated references and checked configuration.

  4. 04

    Check distribution

    After changes inspect the crawler-visible response and allow reprocessing time. Keep query, language, date, URL and observed excerpt. Separate occasional citations, visits and conversions. Check actually published translations and copies; changed directives do not correct third-party content.

    Deliverable: observations, limits and next review.

Acceptance case to reproduce

Fictional example: these inputs describe no customer or observed result.

View case inputs
{
    "robots_meta": "nosnippet",
    "google_indexed": true,
    "ai_citations_observed": 0,
    "observation_period": "one-test"
}

Expected decision

nosnippet controls the documented Google uses. Zero observed citations does not establish removal from the index.

Fictional variations to test

The page has noindex, but robots.txt prevents crawling it.

The crawler cannot read the directive. To remove a public page from the index, allow it to read noindex and verify processing; protect confidential content separately.

HTML declares index, but the HTTP header declares noindex.

Inspect the HTTP response and rendered metadata together. Remove the conflicting rule at its source, then verify the response actually served after clearing the cache.

The document is confidential, but only nosnippet protects it.

Restrict access on the server. A snippet rule does not block directly opening the document; also test its URL without an authorized session.

Evidence to retain

  • URL, robots.txt rules and directive the crawler can actually read.
  • HTTP headers, HTML metadata and checked cache version.
  • Direct-access result with and without authorization.

Additional reference : Google Search Central — Block indexing with noindex [EN]

Additional reference : Google Search Central — Control what you share [EN]

Verification procedure

  1. Record directives, agent and service documentation.
  2. Check indexing and snippet controls separately.
  3. Keep dated observations without generalizing one citation.

Google Search Central — Generative AI guidance

Reusable worksheet

Complete with your authorised observations. These fields are a working template, not observed results.

FieldInformation to record
ServiceCrawler, intended use and dated documentation
ConfigurationHTTP response, directives and settings
ObservationQuery, language, date, cited source and outcome
ReviewOwner, dependencies and next verification

Worked example

Illustrative situation

Fictional example: a team adds llms.txt and treats one assistant mention as a proven SEO improvement.

Decision and expected evidence

It separates services, inspects actual settings, checks the cited excerpt and follows user outcomes over a documented period.

Distinguish the mechanisms

MechanismPurposeCheck or limitation
robots.txtControl relevant crawler accessDoes not provide access control
noindexRequest exclusion from a supporting indexThe engine must observe the directive
Preview controlsLimit excerpts according to serviceCheck scope and reprocessing
llms.txtDocument for systems that read itDoes not improve Google ranking

Management indicators

IndicatorWhat it measuresFirst action
Verified controlsObserved settings and documented testsResolve configuration discrepancies
Observed citationsSources actually cited under the protocolDo not confuse with conversions

Common pitfalls

  • Does not provide access control
  • The engine must observe the directive
  • Check scope and reprocessing
  • Does not improve Google ranking

Frequently asked questions

Can one file control all AI visibility?

No. Services have distinct mechanisms and policies. Inspect documentation and effective settings.

Must useful FAQs disappear after rich-result retirement?

No. Google retired FAQ rich results from 7 May 2026; questions can remain useful to readers.

Will a correctly configured page be cited?

That is not guaranteed. Technical access, useful content and availability guarantee neither indexing nor selection.

Publication and visibility in AI search

Check a setting’s scope before publishing

A crawl refusal, indexing exclusion, snippet restriction and Search AI participation choice operate at different levels.

  • Read HTML and X-Robots-Tag from the served response alongside the applicable robots group.
  • Check the property’s explicit or inherited Search generative AI setting.
  • Separate Search AI participation, training and third-party service policy; retain dated sources.

Compare AI controls and report data →

References reviewed

Official references

References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.