A service that ramps frontend traffic to a new version in steps, checks error rate, latency, and conversion metrics at each step, and rolls back automatically if they regress.
A progressive rollout service for frontend deploys at Credit Karma, used across the frontend org of roughly 1,000 engineers. The system automates traffic ramp-ups, validates health via real metrics, and rolls back when signals degrade. It applies canary-style deploy patterns (common on the backend) to web frontends.
It borrows the concepts from Argo Rollouts and closely mirrors its model, but it's custom-built for Credit Karma's frontend deploy system, which isn't Kubernetes-based.
Context
Frontend releases here leaned on manual checks and someone's judgment about whether a release looked healthy — inconsistent, and slow across a lot of teams. Canary-style rollouts were normal on the backend but hadn't made it to the web frontends. This gave them a standardized, automated path to production.
What I built
- A standalone service coordinating progressive rollouts using a declarative
rollout.ymlper project, mirroring Argo Rollouts standards - A state machine and worker queue that executes rollout steps and evaluates health checks
- Pluggable "weight providers" to integrate with existing deploy infrastructure, which is not Kubernetes-based
- Health analysis powered by queryable telemetry (e.g., error rates, latency, conversion proxies)
- First-class observability and audit trails for every rollout step
How it works
- Teams define rollout plans in code: traffic weights, pauses, and analysis checks
- The service applies each step, shifting traffic gradually to the canary version
- For every step, it runs metric queries; on failure, it halts and rolls back automatically
- On success, it promotes to 100% and records the outcome
Outcomes
- Cut the time a bad release stays live by triggering rollback automatically on metric regression rather than waiting for a manual call.
- Standardized rollout steps across teams instead of per-team manual checks.
- Clear accountability with dashboards and auditable histories
- Adopted by multiple product teams through the shared rollout.yml format.
Tech notes
- Service runtime in Node.js with durable state and background workers
- Configuration-as-code via YAML to keep rollout logic versioned with the project
- Adapters to connect the service to Credit Karma's non-Kubernetes deploy targets and telemetry stack.
Rolling it out
Adoption was incremental, and deliberately so. Each team started with a small set of metrics, wired up the telemetry queries, and only added rollback gates as they came to trust the signals — nobody wants an automatic rollback firing on a metric they don't believe yet.