Observability for systems analyst interviews

Why observability shows up in SA interviews

If a hiring manager at Stripe, Uber, or DoorDash asks a systems analyst about observability, they are not testing whether you can recite "logs, metrics, traces". They want to see whether you understand the difference between monitoring (known unknowns — predictable failure modes you already dashboarded) and observability (unknown unknowns — the new bug at 2am no one has seen before). The first protects yesterday's incident. The second lets you debug tomorrow's.

In a typical 45-minute SA loop, observability comes up inside system design rounds — usually right after the candidate sketches a microservice architecture and the interviewer asks: "a customer says checkout is slow, walk me through how you'd find the root cause." A weak answer says "check the dashboard". A strong answer reaches for traces first, drills into the slow span, correlates with structured logs keyed by trace_id, then confirms the regression on a p99 latency metric broken down by route.

The three pillars in one frame

The canonical model from the OpenTelemetry community is the three pillars: logs, metrics, and traces. Each answers a different question, costs a different amount, and ages differently in storage. The mistake most candidates make is treating them as interchangeable. They are not.

A request to /checkout generates one trace, ten spans, fifty structured log lines, and roughly a hundred metric data points across the path. The volume gap is real — that is why teams sample traces aggressively, aggregate metrics by minute, and rotate logs after 30 days.

Logs without correlation IDs are a liability, not an asset — they fill disk and answer no question.

Logs, metrics, traces in depth

Logs — structured, correlated, sampled

A modern log line is JSON, not free-form text. The minimum useful schema:

{
  "ts": "2026-05-25T12:00:00Z",
  "level": "INFO",
  "service": "checkout-api",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "user_id": "u_42",
  "event": "order_created",
  "order_id": "o_991",
  "amount_usd": 129.00
}

Strengths: narrative detail, audit-grade, easy to grep once indexed. Weaknesses: volume explodes (a busy service emits 100k+ lines per minute), and unindexed search is slow and expensive. The fix at scale is structured logging plus a log pipeline (Fluent Bit → Loki/Elasticsearch) with field-level indexing only on the fields you query — trace_id, service, level. Indexing every field doubles your bill for no win.

Metrics — cheap, aggregatable, alertable

Metrics are pre-aggregated numerics over time. The Prometheus model dominates: counters, gauges, histograms, summaries. A single time series — http_requests_total{method="POST",route="/checkout",status="200"} — costs almost nothing to store at minute granularity, even kept for a year.

http_requests_total{method="POST",route="/checkout",status="200"} 12345
http_request_duration_seconds{route="/checkout",quantile="0.99"} 0.250

The trap is high-cardinality labels. Adding user_id to a metric label explodes the time-series count from hundreds to millions, and Prometheus eats RAM at roughly 3KB per active series. With one million series that's 3GB just for the index. Rule of thumb: if a label value can be more than a few hundred distinct strings, it does not belong as a metric label — push it into logs or traces.

Traces — the latency microscope

A distributed trace is a tree of spans, each representing a unit of work — an HTTP call, a DB query, a cache lookup. The trace tells you where the time went:

[Client request] ──── 850ms total ────────────────
  └─ [API gateway]                    20ms
      └─ [checkout-service]          780ms
          ├─ [auth-service]           45ms
          ├─ [inventory-service]      30ms
          └─ [payments-service]      690ms   ← suspect
              └─ [stripe-api]        680ms

In that example the p99 latency lives inside the payments call to Stripe, not in your own code. Without traces, you'd spend two hours blaming the database. With them, the answer is visible in thirty seconds. The standard now is OpenTelemetry (OTel) — vendor-neutral SDKs that emit traces, metrics, and logs over OTLP to any backend (Tempo, Jaeger, Honeycomb, Datadog).

Tooling landscape

The tooling space has consolidated into two camps: all-in-one SaaS (Datadog, New Relic, Honeycomb) and open-source stack (Prometheus + Loki + Tempo + Grafana, often called the LGTM stack). Both ship to OpenTelemetry-compatible ingestion.

At early-stage startups (under 30 engineers) Datadog wins on time-to-value — you instrument once and the dashboards just appear. Past about $300k/year in observability spend, every team I've seen has migrated at least the metrics layer to Prometheus or VictoriaMetrics to halve the bill. Honeycomb is the niche pick for teams that lean hard on traces and want to query by arbitrary attribute without paying cardinality tax.

Dashboards, alerts, and SLOs

Observability without SLOs is just expensive nostalgia. The Google SRE playbook breaks it into three terms that every SA candidate should be able to define cold:

• SLI (indicator) — the actual measurement, e.g. fraction of /checkout requests under 500ms.
• SLO (objective) — the target, e.g. 99.9% of /checkout requests under 500ms over a rolling 28 days.
• SLA (agreement) — the contractual commitment to a customer, usually weaker than the internal SLO with money attached.

The error budget falls out of the SLO: 99.9% over 28 days allows roughly 40 minutes of downtime per month. When the budget is exhausted, teams pause feature work and ship reliability fixes. This is the mechanism that prevents the eternal "we'll fix it next quarter" loop.

For dashboards, the standard layout is the RED method (Rate, Errors, Duration) for request-driven services, and USE (Utilization, Saturation, Errors) for resource-driven systems like databases or queues. A good interview answer mentions both and explains when to use which: RED for APIs, USE for the Kafka cluster underneath them.

Common pitfalls

The most expensive mistake teams make with logs is logging everything at INFO. Every service emits debug-grade detail by default, the bill triples, and the actually-important WARN/ERROR lines drown in noise. The fix is a log level policy enforced in code review: INFO for state changes (order created, payment captured), DEBUG behind a flag for dev, WARN for recoverable degradation, ERROR for things a human needs to see. Pair this with sampling at the SDK level for high-volume INFO events — keep 1 in 100, full-fidelity for ERROR.

A second trap is metric cardinality explosion from labelling. A junior engineer adds user_id as a Prometheus label "just to be able to filter", and the next morning the Prometheus pod OOMs. The same anti-pattern shows up with request_id, order_id, and any monotonically growing string. The fix is mechanical: metrics get low-cardinality labels (service, route, status code, region), and the question "which user?" is answered from logs or traces using the trace_id as join key.

The third pitfall is trace sampling without thought. Naive head-based sampling at 1% means the rare 0.1% of failing requests are almost never captured — you only see traces of successful requests, which is the opposite of what you need. The fix is tail-based sampling: collect every span, decide after the trace is complete whether to keep it (always keep errors and slow traces, sample the rest). Tools like OpenTelemetry Collector or Honeycomb's Refinery implement this natively.

Fourth, teams often alert on causes instead of symptoms. Alerting on "CPU > 80%" wakes engineers up for noise; alerting on "p99 latency > 1s for 5 minutes" wakes them up when customers actually feel pain. The Google SRE rule of thumb is alert on the SLI, page on user-visible symptoms only. Anything else goes to a ticket queue, not PagerDuty.

Finally, dashboards rot. A dashboard built for the launch of a service in 2023 still shows that service's first endpoint and nothing added since. The pragmatic fix is treating dashboards as code (Grafana JSON in Git, reviewed in PRs) and a quarterly audit where the on-call lead deletes anything that hasn't been viewed in 60 days.

Related reading

• Logging strategy for systems analyst interviews
• Distributed tracing for systems analyst interviews
• Health checks for systems analyst interviews
• Incident response for systems analyst interviews
• Latency budget for systems analyst interviews

If you want to drill systems analyst interview questions like this one every day, NAILDD is launching with structured prep across architecture, observability, and design patterns.

FAQ

What's the difference between monitoring and observability?

Monitoring answers known questions with predefined dashboards and alerts — you set up a CPU graph because you've been burned by CPU before. Observability is the broader capability of answering arbitrary new questions about a running system from its outputs, including questions you didn't anticipate when you built it. A system can be heavily monitored and barely observable (only canned dashboards), or lightly monitored and highly observable (rich structured events you can slice any way at query time). Modern best practice aims for the second.

How are logs, metrics, and traces actually correlated in practice?

The glue is the trace context — a trace_id and span_id propagated through every service via HTTP headers (traceparent in the W3C standard) or message attributes for async work. Every log line a service emits during the handling of that request includes those IDs. Every metric exemplar can attach the trace ID of a representative slow request. The result is a graph: you start at a metric (p99 spike), click an exemplar, land on the trace, then pivot to the logs of the slow span. OpenTelemetry standardised this; almost all modern backends honour it.

How much should observability cost as a fraction of infra spend?

Industry benchmarks land around 5–15% of total infrastructure spend for healthy teams. Above 20%, you are paying for noise — usually high log retention, unsampled traces, or high-cardinality metrics. Below 3%, you are likely under-instrumented and paying for it in incident MTTR instead. The right answer in an interview is "it depends, but I'd target ~10% with regular audits of the top-10 most expensive log sources and metric series".

Do I need OpenTelemetry if I'm already on Datadog (or any other vendor)?

Yes — even when you stay on Datadog, instrumenting with OTel SDKs and exporting via the OTel Collector buys you vendor portability. If pricing or strategy changes, you re-point the collector at a different backend instead of rewriting every service. The Datadog Agent already accepts OTLP natively, so there is no functional downside. Most large orgs that started on a single vendor have made this switch in the last two years specifically to avoid lock-in.

When does an SA candidate get asked observability questions vs an SRE candidate?

SRE rounds go deep on implementation — how Prometheus federation works, how to design a Thanos deployment, error budget burn-rate alerts with multi-window multi-burn-rate math. SA rounds stay at the design level — when to add a new SLI, how to scope what to instrument, how to argue for an observability budget. A systems analyst should be able to draw the three-pillar diagram, name the tools in each layer, and explain SLI/SLO/SLA cold. Going deeper is bonus; not being able to do those three is the disqualifier.

Is structured logging worth the migration cost on a legacy codebase?

For services that are actively developed and on-call, yes — the payback is usually inside one incident where you save hours of grepping. For services in maintenance mode that nobody pages on, no — the migration cost rarely pays back. A pragmatic middle path is to introduce structured logging only at the HTTP middleware layer (one request → one structured access log line with trace context) and leave deeper code untouched. That alone gives you 80% of the debugging value for 10% of the work.