Match the signal to the question
OpenTelemetry distinguishes signals including metrics, logs and traces. They complement one another without guaranteeing complete visibility. Document measurement windows, instrumentation coverage and exclusions; an average alone may hide the slowest requests.
| Question | Useful signal | Known limitation |
|---|---|---|
| How many requests fail? | Aggregated metrics | Define failures, scope and time window. |
| Where was time spent? | Trace and step durations | Sampling and uninstrumented dependencies. |
| What event happened? | Structured log | No secrets or excessive data. |
| Does the journey work? | Functional probe | One scenario does not cover every use. |
Make diagnosis actionable
- Measure throughput, errors and duration distributions by normalized operation. Avoid user identifiers and variable URLs as metric labels.
- Link events to a correlation identifier and service version. Check context propagation and sampling policy before investigating a missing trace.
- Give every alert an impact, owner and first action. Use dedicated accounts for probes and avoid irreversible operations.
Protect data and check outcomes
Keep passwords, tokens and sensitive bodies out of logs. Restrict access and retention. A successful HTTP response can still contain an incorrect business result: test expected invariants and permissions in a controlled environment too.
Primary references
Further reading
Connect telemetry to a user task
Metrics, logs and traces answer different questions. High uptime can conceal a failing checkout or sign-in journey. Define the outcome users expect, then instrument the calls that deliver it.
- Choose a journey indicator for success, latency and user-visible failure.
- Propagate correlation across services while limiting sensitive data in traces.
- Rehearse a dependency failure to see whether alerts lead to action and an understandable degraded mode.
Primary reference: OpenTelemetry — Observability primer. Related method: read the practical guide.
Collection outages: reconcile logs and missing events
Separate produced, received and usable events to explain an outage without treating silence as absence of incidents.
- Map application, collector and storage with owners. Prepare a finite fictional event sequence with stable identifiers and no secrets or customer data. Retain event and observation times: collection delay does not automatically mean loss.
- In testing, interrupt the destination and restore it. Observe queue filling, retries, rejections and restart. Persistent queues can limit some losses but remain bounded and depend on storage; define full-queue behaviour. Check that collection does not silently block the service.
- Reconcile the reference with events actually readable at the same observation cutoff. Count repeats, delays and absences separately; network acknowledgement does not prove final usability. Without a reliable reference, state unknown coverage rather than zero lost events or zero incidents.
Acceptance case to reproduce
Fictional example: these inputs describe no customer or observed result.
View case inputs
{
"expected_ids": [
"E1",
"E2",
"E3"
],
"received_ids": [
"E1",
"E1",
"E3"
],
"received_rows": 3,
"unique_received": 2,
"missing_ids": [
"E2"
],
"observation_cutoff": "same_for_both_sets"
}Expected decision
Fictional example: E1, E2 and E3 are expected, but E1, E1 and E3 arrive. Three rows do not establish completeness: E1 repeats and E2 is missing at the observation cutoff.
Do three received rows prove three events were retained?
No. Two rows may repeat an identifier and hide an absence. Compare identifiers, the observation window and pipeline stages.
Primary reference: OpenTelemetry — Collector resiliency. Related method: read the practical guide.
Dangling subdomains: retire a CNAME without leaving a claimable dependency
Tie each DNS name to a controlled resource and make DNS cleanup part of service retirement.
- Record public name, CNAME chain, provider resource and owner. Reconcile the DNS zone with the authorized account inventory. An HTTP error, NXDOMAIN or absent target is a finding to investigate, not proof of takeover eligibility; provider protections differ.
- For a retiring service, remove the mapping or move it to a destination you control before releasing the resource. Account for previous TTLs and caches; check authoritative DNS and several resolvers. Preserve provider ownership validations that are still needed instead of deleting all TXT records.
- Check links, redirects, OAuth callbacks, webhooks and cookie domains still using the name. If the target was already released, correct the mapping and investigate possible exposure through the incident process. Reconcile DNS and resource inventories after every deletion.
Acceptance case to reproduce
Fictional example: these inputs describe no customer or observed result.
View case inputs
{
"name": "portal.example",
"cname_present": true,
"resource_deleted": true,
"provider_claimability": "not_verified",
"takeover_observed": false,
"expected": "remove_or_replace_mapping_and_review_exposure"
}Expected decision
Fictional example: portal.example still points to a deleted resource. Remove the CNAME or replace it with a controlled target; an absent resource alone does not establish that a third party took over the subdomain.
Primary reference: Microsoft — Prevent dangling DNS entries. Related method: read the practical guide.
API retries: count the end-to-end budget
In monitoring, separate individual call duration, retry waits and total client journey time. Fast average calls can coexist with a slow journey when errors trigger successive waits.
Fictional teaching case — figures describe neither CIE nor a client.
This fictional scenario has four total attempts, each limited to 250 ms, with deterministic waits of 100, 200 and 400 ms. The modelled budget is 1,700 ms: 1,000 ms of calls and 700 ms waiting. It exceeds a 1,500 ms deadline before other journey costs are counted.
- Total attempts
- 4One initial attempt and three retries
- Accumulated waits
- 700 ms100 + 200 + 400
- Modelled budget
- 1,700 ms4 × 250 + 700
| Step | Maximum call time | Wait before next call |
|---|---|---|
| Attempt 1 | 250 ms | 100 ms |
| Attempt 2 | 250 ms | 200 ms |
| Attempt 3 | 250 ms | 400 ms |
| Attempt 4 | 250 ms | None |
| Total | 1,000 ms | 700 ms |
- Calls1,000 ms
- Waits700 ms
- Entered deadline1,500 ms
Checks for interpreting the case
- Propagate a global deadline and bound each call by the remaining time. Avoid stacking independent retry policies across layers.
- Verify the operation can be repeated without duplicate effects. A timeout does not establish that the remote server stopped processing.
- This model excludes jitter, caps, Retry-After, DNS resolution, connection, queues and local validation. Measure these costs before configuring a service.
When retries make an outage worse
A duration budget is only one condition. An operation may be fast enough yet unsafe to repeat or may amplify load on an unavailable provider.
| Signal to examine | Concrete check | Effect on the decision |
|---|---|---|
| Non-idempotent action | Check the key, its scope and server-side result retention. | Reconcile state before repeating a creation or payment. |
| Retries at multiple layers | Count attempts from the browser, gateway and service. | Assign retry ownership and an end-to-end budget. |
| Sustained outage | Check deadline, cancellation and the journey without the provider. | Provide explicit degradation and controlled recovery. |
From a number to a verification
Exercise the complete failure scenario
- Test to reproduce
- Test unavailability, slow responses and response loss after processing. Check idempotency before retrying a write and count retries in intermediate layers too.
- Stop signal
- Retrying a non-idempotent operation without knowing whether the first attempt took effect.
Change an assumption
Reduce retries without duplicating the business action
The preceding example allows four attempts and reaches 1,700 ms. Here is what one fewer attempt changes, keeping modelled budget separate from observed duration.
- Three attempts
3 × 250 ms = up to 750 ms of calls in the model.
- Two waits
100 + 200 ms = 300 ms; modelled total: 1,050 ms.
- Deadline
That leaves 450 ms out of 1,500 ms for other costs, without guaranteeing they fit.
Reading exercise: the variations describe fictional situations.
What if the first attempt already created the order?
- New evidence
- The server processed the action, but its response was lost.
- What changes
- Another attempt may repeat the business effect. Verify idempotency or retrieve operation state before resending the action. Timing arithmetic cannot solve this problem.
What if three layers each allow three attempts?
- New evidence
- In a simplified maximum case, three attempts at each of three levels can produce 27 downstream calls.
- What changes
- Identify the layer responsible for retries and the overall budget. The product 3 × 3 × 3 describes this scenario, not a fixed production volume.
Why count attempts rather than retries?
One initial attempt plus three retries makes four calls. Distinct names avoid an off-by-one error in configuration and calculation.
Try a retry budget
n × timeout + wait × (2^(n−1) − 1). Doubling waits, without jitter or caps.
Related method: read the practical guide.
References: data, uses and limitations
Documentation reviewed on .
| Source and purpose | Verified data or benchmark | How to use it | Limitation to retain |
|---|---|---|---|
| NIST · SP 800-61 Rev. 3 Prepare incident response within cybersecurity risk management. |
Revision 3 was published in April 2025 and supersedes Revision 2. It is a CSF 2.0 Community Profile. | Rehearse containment decisions, evidence preservation, communication and recovery. Compare planned actions with evidence produced during the exercise. | Reading the guide does not validate a local plan. Contacts, emergency access and dependencies need local testing. |
| Google · Core Web Vitals Distinguish loading, responsiveness and visual stability. |
Good thresholds are LCP ≤ 2.5 s, INP ≤ 200 ms and CLS ≤ 0.1 at the 75th percentile, segmented by mobile and desktop. | Separate field measurement from laboratory tests. Record period, URL, device, population and sample size before comparing changes. | These thresholds are not measured results for this site. A good lab score alone does not establish visitor experience. |
API deprecation and retirement
Announced retirement and continuity to verify
A Sunset field signals intent; it establishes neither actual shutdown nor guaranteed continuity until then. Prepare dependencies, data and access to move.
- Identify the announcement’s exact scope and owner.
- Keep dates and updates with receipt evidence.
- Test exit, in-flight operations and removal of obsolete access.
Compare deprecation and retirement announcements →
IETF · RFC 9745 · Deprecation · Standards Track · March 2025 · §2–5 · consulted 2026-10-23
IETF · RFC 8594 · Sunset · Informational · May 2019 · §3–5, §8 · consulted 2026-10-23