Resources · 86
API pagination: complete exports without missing or duplicated records
Define ordering, tokens, authorization and recovery when a collection changes during traversal.
· 3 min
What this guide helps achieve
- Define ordering and scope
- Treat tokens as continuation
- Detect completion from the contract
- Recover after failures
- Reconcile the result
Quick check
- Does an empty page mean completion?
- Can a cursor provide authorization?
- Does deduplication prevent missing records?
Step-by-step method
- 01
Define ordering and scope
Document filters, ordering and a tie-breaking key. A timestamp alone may be insufficient when several objects share it. Decide whether the collection stays live or the export uses a snapshot. Do not promise a frozen view when the provider offers no such guarantee.
Deliverable: ordering and consistency contract.
- 02
Treat tokens as continuation
AIP-158 specifies opaque tokens and separates pagination from authorization: a token does not confer access rights. Check identity and permissions on every request. Keep reading parameters consistent with the contract and handle expiry or invalidation explicitly.
Deliverable: valid, expired and unauthorized token cases.
- 03
Detect completion from the contract
Do not assume completion because a page is shorter than requested. An API may return fewer items while providing continuation. Under AIP-158 an empty next_page_token indicates completion; adapt clients to the provider’s actual contract. Also detect repeated tokens without progress.
Deliverable: completion and loop tests.
- 04
Recover after failures
Save continuation after durable storage of the processed batch. Deduplicate using stable identifiers and keep recovery state separate from displayed totals. If a snapshot expires, restart or reconcile under an explicit policy; do not silently concatenate two different collection states.
Deliverable: checkpoint and failure scenario.
- 05
Reconcile the result
Compare unique identifiers, exclusions, states and any reference total at a common cutoff. Test creation, changes and deletion during reading. A correct count does not prove the right objects were exported. Protect files and traces according to their data.
Deliverable: export manifest and documented differences.
A paginated read that can recover
Under an AIP-158-style contract, continuation determines the next step. A short page does not establish completion.
Define the read
Retain filter, order, parameters and authorized identity.
Store the batch
Durably save objects and their identifiers.
Follow continuation
A present token leads to the next request with bound parameters.
Reconcile at completion
When the final token is empty, check exclusions, reference state and coverage.
Fictional example: the connection fails after batch storage. Recovery uses the saved checkpoint and checks identifiers already processed.
Reusable worksheet
Complete with your authorised observations. These fields are a working template, not observed results.
| Field | Information to record |
|---|---|
| Collection | Filters, order, tie-breaker and guarantees |
| Continuation | Token, linked parameters, expiry and completion |
| Recovery | Durable batch, unique identifiers and saved checkpoint |
| Result | Manifest, exclusions, cutoff and reconciliation differences |
Worked example
Illustrative situation
Fictional example: products are added while a catalog is exported using offsets, moving a row between pages.
Decision and expected evidence
Testing finds a repeat and an absent product; the team chooses an available API snapshot or documents live reading with reconciliation.
Distinguish the mechanisms
| Mechanism | Purpose | Check or limitation |
|---|---|---|
| Offset | Access a numeric position | Insertions can shift positions |
| Cursor | Continue from a reading marker | Does not itself guarantee a snapshot |
| Snapshot | Fix a reference state | Check validity period and cost |
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Unique identifiers | Distinct objects retained in the export | Compare with an equivalent reference scope |
| Progress | Tokens and batches genuinely advancing | Stop loops and investigate |
| Verified coverage | Objects reconciled with a reference state | Do not equate volume with completeness |
Common pitfalls
- Insertions can shift positions
- Does not itself guarantee a snapshot
- Check validity period and cost
Frequently asked questions
Does an empty page mean completion?
Check the contract and continuation token. Page size alone should not determine completion.
Can a cursor provide authorization?
No. The server must check request permissions independently of the pagination token.
Does deduplication prevent missing records?
No. It removes repeats without recovering unread objects. Test ordering stability and collection changes.
Official references
References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.
Continue exploring
Related analysis
