Observability Interviews: 12 Questions on OTel, Traces, SLOs

You have three years of Datadog dashboards on your resume and a panel interview in two hours. The recruiter said platform/SRE role, heavy observability. That phrase means something specific now — not can you read a graph, but do you understand OpenTelemetry semantics, trace context propagation, and how error budgets actually get calculated.

Interview panels in 2025 and 2026 have converged on a shorter, sharper set of questions because OTel became the de facto standard and SLO math stopped being optional. If you can answer these twelve cleanly, you will out-perform most candidates who only know vendor UI clicks.

Why the Bar Moved: OTel Became the Baseline

Three years ago, observability interviews were tool-specific: Splunk syntax, New Relic dashboards, Datadog tags. Now most platform teams have standardized on the OpenTelemetry Collector for ingestion, regardless of backend. Interviewers assume you know the vendor-neutral layer first, then ask how you would wire it to whatever backend the team runs.

That shift means questions about spans, context propagation, and semantic conventions show up before any tool-specific question does.

The Core Six: Tracing and OpenTelemetry Questions

These test whether you understand the mechanics, not just the marketing.

1. Explain the difference between a trace, a span, and a span context. A trace is the end-to-end record of a request; a span is one unit of work inside it; span context is the trace ID, span ID, and trace flags that get propagated so downstream services attach to the same trace.
2. How does trace context propagate across an HTTP call? Via the W3C Trace Context header — traceparent and optionally tracestate — carried on the outbound request and read by the next service's instrumentation.
3. What is an exemplar and why does it matter? An exemplar links a specific metric data point — say, a P99 latency bucket — back to an actual trace ID that produced it. It is the bridge that lets you go from a spike on a graph straight to the request that caused it, without guessing.
4. What is the OpenTelemetry Collector and why put it between your app and your backend? It decouples instrumentation from the backend. You can switch from Jaeger to Tempo to a SaaS vendor by changing an exporter config, not re-instrumenting every service.
5. What is a sampling strategy, and how would you choose one for a high-traffic service? Head-based sampling decides at the start of a trace, cheap but can miss rare errors. Tail-based sampling decides after the trace completes, so it can keep all error traces and a percentage of normal ones — more useful, more expensive to run.
6. What is semantic convention, and why does OTel enforce it? Standard attribute names — http.method, db.system, service.name — so traces from different teams and languages are queryable the same way instead of every team inventing its own tag scheme.

RED vs USE: The Question Everyone Half-Answers

Almost every candidate has heard both acronyms. Few can say cleanly when to use which. This is the fastest way to separate someone who read a blog post from someone who has run an on-call rotation.

The interview trap: they will ask you to monitor a message queue consumer. RED alone misses queue backlog. USE alone misses whether the consumer is actually processing successfully. The correct answer combines both — RED for the service's request handling, USE for the infrastructure it depends on.

SLOs and Error Budgets: The Math They Will Actually Ask

This is where candidates freeze because they know the vocabulary but never did the arithmetic. Be ready for a whiteboard version of this:

• SLI — the measured indicator, e.g., percentage of requests under 300ms.
• SLO — the target, e.g., 99.9% of requests under 300ms over 30 days.
• Error budget — the allowed failure margin. At 99.9% over 30 days, that is roughly 43 minutes of budget to spend on deploys, incidents, or risk.

Expect a follow-up: "Your error budget is 80% consumed by day 12 of a 30-day window. What do you do?" The expected answer is not panic — it is policy. Most SRE teams freeze non-critical deploys, prioritize the burn-rate alert investigation, and only unfreeze once the budget trend stabilizes. If you can describe a burn-rate alert — fast burn over 1 hour vs slow burn over 6 hours, per the Google SRE workbook — you signal real production experience.

Cardinality Traps: The Question That Filters Seniors From Juniors

This is the one that trips people who only worked with small services. Ask yourself: what happens if you tag every metric with user_id?

The answer they want: cardinality explosion. Each unique label value creates a new time series. A metric tagged by user_id on a service with a million users can generate a million time series from one metric name — and most time series databases degrade sharply, or the bill spikes, well before that.

Good candidates name the fix directly:

• Keep high-cardinality identifiers (user_id, request_id, session_id) in traces and logs, not in metric labels.
• Use metrics for aggregate rates and traces/exemplars for the individual-request drill-down.
• Set label allowlists in the Collector's processor pipeline to catch this before it reaches the backend.

If you can walk through that trade-off unprompted, you have just answered the hardest question on most observability interview loops without being asked directly.

Quick Reference: The 12 Questions

• Trace vs span vs span context
• Trace context propagation (W3C traceparent)
• Exemplars and their purpose
• Role of the OpenTelemetry Collector
• Head-based vs tail-based sampling
• Why semantic conventions matter
• When to use RED vs USE
• How to monitor a queue consumer (both frameworks together)
• Define SLI, SLO, error budget with numbers
• What to do when an error budget burns fast
• What causes cardinality explosion
• How to prevent cardinality explosion in practice

The Josh Pros LLC team places contract platform and SRE consultants into observability-heavy roles across the country, and we hear which of these questions come up most often, client by client. If you want a read on what a specific role is really testing for, email contact@joshpros.com or visit https://joshpros.com.

#Observability #OpenTelemetry #DistributedTracing #SRE #PlatformEngineering #SLO #ErrorBudget #ITContracting #TechInterviews #CloudCareers #DevOps #ContractConsultants

Back to all insights