If you've worked an observability contract in the last year, you've probably noticed the RFPs changed. Clients aren't asking for 'Datadog experience' anymore. They're asking for OpenTelemetry instrumentation, vendor-neutral pipelines, and someone who can explain why the log bill tripled.
That shift is not cosmetic. It reflects a real re-platforming happening inside enterprise observability teams, and it's creating a specific, well-paid lane for consultants who understand the pipeline mechanics, not just the dashboards on top of it.
This piece maps what actually changed, why it's happening now, and which skills make you useful on day one of a migration instead of day one of a tutorial.
What changed: OTel stopped being optional
OpenTelemetry's traces, metrics, and logs specifications have matured enough that most major observability vendors now ingest OTLP natively. Semantic conventions — the standardized attribute names for things like http.method, db.system, or k8s.pod.name — are stable enough that teams can finally write instrumentation once and route it anywhere.
That one fact collapses a lot of vendor lock-in. A client can instrument an application with the OTel SDK, send data through an OTel Collector, and decide later whether it lands in Grafana, Splunk, Datadog, or an internal data lake. The agent is no longer the product. The pipeline is.
For consultants, this means the valuable skill isn't '[Vendor X] certified.' It's understanding the collector architecture that sits upstream of whatever UI the client happens to be paying for this quarter.
Why clients are re-platforming right now
Three pressures are driving this, and you'll hear all three in discovery calls:
- Cost. Log and trace volume grew faster than budgets. Teams that once shipped everything to one vendor are now asking what they can drop, sample, or route to cheaper cold storage.
- Lock-in fatigue. Procurement teams got burned by multi-year vendor contracts tied to proprietary agents. A vendor-neutral collector layer gives them renegotiating leverage.
- Multi-cloud and multi-tool reality. Most enterprises run a mix of Kubernetes, serverless, and legacy VMs, often split across AWS, Azure, and GCP. A single proprietary agent rarely covers all of it cleanly; a collector-based pipeline does.
None of this means Datadog, Grafana, or Splunk are going away. It means the contract work has moved one layer down — into the collector configs, the processors, and the routing logic that decides what those tools ever see.
The pipeline skills that matter mid-migration
Tutorials teach you to stand up a collector and auto-instrument a sample app. That gets you in the door. It does not get you through week three of a real migration, when the client's actual problems look like this:
- Collector topology design. Deciding between agent-per-host, gateway collectors, or a hybrid model, and understanding the latency and reliability trade-offs of each.
- Processor configuration. Batch processors, memory limiters, attribute processors for scrubbing PII, and tail-sampling processors that run at the collector rather than the SDK.
- Semantic convention governance. Enforcing consistent attribute naming across teams so a trace from the Java service and a trace from the Go service can actually be correlated downstream.
- Cardinality control. Catching high-cardinality labels (user IDs, request IDs) before they hit a metrics backend and blow up the bill or crash the query engine.
- Dual-pipeline cutover planning. Running legacy agents and OTel in parallel during migration without double-billing or losing data continuity, then safely decommissioning the old path.
Notice what's missing from that list: a specific vendor's query language. That's deliberate. Clients can train you on their dashboard syntax in a day. They cannot quickly train someone to reason about collector pipeline failure modes under load.
Trace sampling and cost control: the conversation every client is having
Sampling strategy is where the cost conversation actually lives, and it's a frequent interview topic right now.
| Strategy | How it works | Where consultants add value |
|---|---|---|
| Head-based sampling | Decision made at trace start, before outcome is known | Simple to implement; consultants tune sample rate per service tier |
| Tail-based sampling | Decision made after the full trace completes, using full context (errors, latency) | Requires collector-level buffering and policy design; higher-value skill |
| Adaptive/dynamic sampling | Sample rate adjusts automatically based on traffic volume or error rate | Requires monitoring the sampler itself and tuning thresholds over time |
Tail-based sampling at the collector is the pattern clients increasingly want, because it lets them keep every error trace and every slow trace while discarding most of the boring, successful ones. Implementing that correctly — without losing causally related spans or overwhelming collector memory — is advanced, specific, billable work. It's also exactly the kind of skill that separates a consultant who read the docs from one who has shipped it.
If you want to verify current defaults and configuration options, the OpenTelemetry Collector documentation and the CNCF OTel project repo are the primary sources — check them directly rather than relying on secondhand summaries, since processor options evolve quickly.
What to do in the next 90 days
You don't need to become a maintainer. You need working fluency with the pieces clients are actually touching:
- Stand up an OTel Collector locally and configure at least one receiver, one processor chain (batch plus tail-sampling), and two exporters pointing at different backends.
- Practice writing semantic-convention-compliant instrumentation for one language SDK you already use professionally.
- Read through one real cardinality incident postmortem (several vendors publish these) to understand how label explosions actually happen in production.
- Be ready to explain, in plain language, the cost trade-off between head-based and tail-based sampling to a non-technical stakeholder — that explanation is often the actual interview question.
None of this requires a certification exam. It requires hands-on reps with the collector config file, because that's the artifact every migration project revolves around.
If you're weighing which observability skills are worth your next few weeks of study, or you want a sense of what's actually showing up in current contract requirements, the Josh Pros LLC team tracks this across our active roles. Reach out at contact@joshpros.com or visit https://joshpros.com — no pressure, just a conversation about where the market is heading.
#OpenTelemetry #ObservabilityEngineering #TraceSampling #ContractTechConsultants #SRE #CloudSkills #DevOps #PlatformEngineering #ITStaffing #TechContracting #OTelCollector #CostOptimization
Talk to a real recruiter, not a bot.
We'll tell you the rate, the client, and the terms before you interview. And if we're not the right fit, we'll say so.