Resources · 80
AI search: verify visibility and publication controls
Separate crawling, indexing, previews and AI service usage before changing site directives.
· 4 min
What this guide helps achieve
- Separate uses
- Inspect effective settings
- Follow current requirements
- Check distribution
Quick check
- Can one file control all AI visibility?
- Must useful FAQs disappear after rich-result retirement?
- Will a correctly configured page be cited?
Step-by-step method
- 01
Separate uses
Define what should be allowed: human access, crawling, indexing, snippets, citations or other documented service usage. One crawler rule is not a universal policy for every assistant. Identify useful public content and data requiring access control.
Deliverable: service, use and decision matrix.
- 02
Inspect effective settings
Record HTTP status, robots.txt, HTML and header directives, canonical URLs and infrastructure restrictions. Directives must be observable to the relevant crawler. Distinguish crawl blocking from index removal; protect private content with authentication and authorization.
Deliverable: captures of actually served responses.
- 03
Follow current requirements
Consult service documentation and its changes. Google states that llms.txt is not used for Search ranking and special AI markup is unnecessary. Check applicable Search Console settings too. Do not promise citations based on a file or valid schema.
Deliverable: dated references and checked configuration.
- 04
Check distribution
After changes inspect the crawler-visible response and allow reprocessing time. Keep query, language, date, URL and observed excerpt. Separate occasional citations, visits and conversions. Check actually published translations and copies; changed directives do not correct third-party content.
Deliverable: observations, limits and next review.
Acceptance case to reproduce
Fictional example: these inputs describe no customer or observed result.
View case inputs
{
"robots_meta": "nosnippet",
"google_indexed": true,
"ai_citations_observed": 0,
"observation_period": "one-test"
}Expected decision
nosnippet controls the documented Google uses. Zero observed citations does not establish removal from the index.
Fictional variations to test
The page has noindex, but robots.txt prevents crawling it.
The crawler cannot read the directive. To remove a public page from the index, allow it to read noindex and verify processing; protect confidential content separately.
HTML declares index, but the HTTP header declares noindex.
Inspect the HTTP response and rendered metadata together. Remove the conflicting rule at its source, then verify the response actually served after clearing the cache.
The document is confidential, but only nosnippet protects it.
Restrict access on the server. A snippet rule does not block directly opening the document; also test its URL without an authorized session.
Evidence to retain
- URL, robots.txt rules and directive the crawler can actually read.
- HTTP headers, HTML metadata and checked cache version.
- Direct-access result with and without authorization.
Additional reference : Google Search Central — Block indexing with noindex [EN]
Additional reference : Google Search Central — Control what you share [EN]
Verification procedure
- Record directives, agent and service documentation.
- Check indexing and snippet controls separately.
- Keep dated observations without generalizing one citation.
Reusable worksheet
Complete with your authorised observations. These fields are a working template, not observed results.
| Field | Information to record |
|---|---|
| Service | Crawler, intended use and dated documentation |
| Configuration | HTTP response, directives and settings |
| Observation | Query, language, date, cited source and outcome |
| Review | Owner, dependencies and next verification |
Worked example
Illustrative situation
Fictional example: a team adds llms.txt and treats one assistant mention as a proven SEO improvement.
Decision and expected evidence
It separates services, inspects actual settings, checks the cited excerpt and follows user outcomes over a documented period.
Distinguish the mechanisms
| Mechanism | Purpose | Check or limitation |
|---|---|---|
| robots.txt | Control relevant crawler access | Does not provide access control |
| noindex | Request exclusion from a supporting index | The engine must observe the directive |
| Preview controls | Limit excerpts according to service | Check scope and reprocessing |
| llms.txt | Document for systems that read it | Does not improve Google ranking |
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Verified controls | Observed settings and documented tests | Resolve configuration discrepancies |
| Observed citations | Sources actually cited under the protocol | Do not confuse with conversions |
Common pitfalls
- Does not provide access control
- The engine must observe the directive
- Check scope and reprocessing
- Does not improve Google ranking
Frequently asked questions
Can one file control all AI visibility?
No. Services have distinct mechanisms and policies. Inspect documentation and effective settings.
Must useful FAQs disappear after rich-result retirement?
No. Google retired FAQ rich results from 7 May 2026; questions can remain useful to readers.
Will a correctly configured page be cited?
That is not guaranteed. Technical access, useful content and availability guarantee neither indexing nor selection.
Publication and visibility in AI search
Check a setting’s scope before publishing
A crawl refusal, indexing exclusion, snippet restriction and Search AI participation choice operate at different levels.
- Read HTML and X-Robots-Tag from the served response alongside the applicable robots group.
- Check the property’s explicit or inherited Search generative AI setting.
- Separate Search AI participation, training and third-party service policy; retain dated sources.
Compare AI controls and report data →
References reviewed
Official references
References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.
