Progressive Schema Rollout With Percentage-Based Flags

You flipped the new schema path to 100% in one move, and within ninety seconds p99 latency doubled, the error dashboard lit up, and the only lever you had was a full application redeploy that would take four minutes to roll out. The point of a percentage-based flag is to never be in that position: instead of a binary on/off, the flag exposes the new path to a fraction of traffic — 1%, then 5%, then 25% — so a regression shows up as a rounding error on one bucket, not a fleet-wide outage. This page is the runbook for ramping a schema-backed path by percentage, wiring a metrics gate to each bump so the ramp pauses itself when a signal degrades, and snapping every request back to the legacy path with a single config write. It assumes you have already gated the read and write paths correctly, which the parent Feature Flag Rollouts topic covers; here the concern is the ramp curve and the gates around it. Examples target PostgreSQL 11+ and MySQL 8.0.

Symptom / Error Signatures

You are on the right page if a schema rollout jumped straight to full traffic and you are now watching one or more of these signals, or if you tried to ramp but the percentage never actually changed what traffic did:

• PostgreSQL ERROR: canceling statement due to statement timeout appearing only on the new query path as its share of traffic rose.
• MySQL ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction correlated with the exact minute the rollout percentage was bumped.
• p95/p99 latency on gated endpoints stepping up in lockstep with each percentage increase, while the legacy baseline stays flat.
• The rollout percentage was set to 25 in config but the new-path request metric still reads ~1% — a sign the bucket key is unstable or the SDK cache never refreshed.
• flag evaluation returned default (service unreachable) in application logs, meaning a fraction of traffic silently fell to the fallback path mid-ramp and your percentage is no longer what the dashboard claims.

The fastest confirmation is to compare the configured percentage against the observed new-path share. If a 25% rollout is not producing roughly a quarter of requests on the new path, the ramp is not behaving as a percentage at all, and no further bump is safe until that gap is explained.

-- PostgreSQL · monitoring role · read-only · is the new path's share tracking the configured percentage?
-- Run during a ramp; the ratio should approximate the rollout fraction.
SELECT
  count(*) FILTER (WHERE query ILIKE '%order_status%') AS new_path,
  count(*) FILTER (WHERE query NOT ILIKE '%order_status%'
                     AND query ILIKE '%orders%')       AS legacy_path
FROM pg_stat_activity
WHERE state = 'active';

Root Cause Analysis

A percentage rollout fails for one of two reasons, and they demand opposite fixes, so name which one you have before touching anything. The first is a real regression: the new schema path is genuinely slower or more error-prone, and raising its traffic share simply surfaced load that the low percentage was hiding. The right response is to let the gate stop the ramp and hold — the pattern working as designed. The second is a rollout mechanics bug: the percentage is not being applied faithfully, so the numbers on your dashboard are fiction. This happens when the bucket key is unstable (a user flips buckets between requests, smearing traffic across both paths), when the flag SDK polls on a long interval so a bump takes minutes to propagate, or when an unreachable flag service quietly serves the default to a slice of the fleet.

The distinction matters because a metrics gate can only protect you against the first failure. If the mechanics are wrong, the gate is reading a percentage that does not exist and will either hold forever or wave through a ramp that is really running at a different fraction. The two most common divergences by flag-delivery model:

The idiomatic ramp couples every percentage bump to a metrics gate on the new path specifically, not the aggregate — because at 5% a doubling of new-path error rate is invisible in a blended number. Tie that gate to the same signals your pipeline already tracks in schema migration metrics and SLOs, so the rollout gate and the deploy gate speak the same language.

Immediate Mitigation

When a ramp is actively degrading, the goal is to shed traffic off the new path faster than the incident widens. Every step below is a config action or a read-only query — none of it touches DDL, because the schema is additive and both paths remain valid.

1. Drop the percentage to 0, do not delete the flag. Setting the fraction to 0 returns all traffic to the legacy path on the next request; deleting the flag risks resolving to an undefined default. Keep the flag so you can resume the ramp after triage.

   # Operator action · flag API · takes effect on the next evaluated request, no deploy, no DDL
   # Safe unconditionally while dual-write kept the legacy column current (see parent topic).
   curl -X PATCH https://flags.internal/api/flags/schema_v2 \
        -d '{"rollout": {"percentage": 0}}'

2. Confirm the drop actually propagated. With a poll-based SDK the change is not instant; verify the new-path share is falling before you assume the incident is contained.

   -- PostgreSQL · monitoring role · read-only · new-path active queries should trend to zero
   SELECT count(*) FROM pg_stat_activity
   WHERE state = 'active' AND query ILIKE '%order_status%';

3. Kill any query the new path left blocking. If a slow new-path statement is holding locks, terminate the specific backend — never an OS SIGKILL.

   -- PostgreSQL · requires pg_signal_backend or superuser
   SELECT pg_terminate_backend(<blocker_pid>);
   
   -- MySQL 8.0 · requires CONNECTION_ADMIN or SUPER
   KILL <blocking_thread_id>;

4. Pin background contexts to legacy. Crons, queue consumers, and batch jobs evaluate with no user key; set them explicitly to the legacy path so they do not keep exercising the failing route while you investigate.

5. Freeze the ramp automation. If a scheduler or pipeline was auto-advancing the percentage, pause it so it cannot re-raise the fraction under you. This is the same interlock that auto-reverting migrations on health-check failure applies to deploys, pointed at the flag instead of the migration.

Permanent Fix / Long-Term Pattern

The durable fix is to treat the ramp as a gated state machine, not a series of manual guesses. Each percentage is a checkpoint; advancing requires the metrics gate to pass over an observation window; failing the gate holds or steps back automatically. This is the read-path application of the broader Feature Flag Rollouts discipline, and it depends on the schema being additive so that any step-down is lossless.

1. Define the ramp curve and the gate up front. Fix the steps and the SLO budget before the rollout starts, so no one negotiates thresholds mid-incident. Encode the fallback so a flag-service outage fails to the legacy path.

   # Flag config (rollout service) · applied via API, no deploy · default-off on eval failure
   schema_v2:
     default: false
     rollout:
       by: user_id            # stable bucket key — a user stays on one path across requests
       percentage: 1          # ramp starts here
     steps: [1, 5, 25, 50, 100]
     gate:
       window_seconds: 600    # observe each step for 10 minutes before advancing
       max_error_rate: 0.01   # new-path error budget
       max_p99_ms: 250        # new-path latency ceiling
     contexts:
       batch_processor: false # background jobs stay legacy until 100% holds

2. Emit a per-path metric on every gated request. Tag each request with the resolved flag value so the gate reads the new path in isolation, not a blended average that hides a small-bucket regression.

3. Advance only when the gate passes over the full window. After each bump, hold for window_seconds, then compare new-path error rate and p99 against the budget. Pass advances to the next step; fail holds or steps down one level.

   # Ramp controller (Python) · runs in the deploy pipeline · reads metrics, writes flag config
   # Never advances on a partial window; a breached budget steps back, it does not pause blindly.
   def advance_ramp(flag, metrics, steps):
       cur = flag.percentage
       nxt = next((s for s in steps if s > cur), cur)
       if metrics.new_path_error_rate() > flag.gate.max_error_rate \
          or metrics.new_path_p99_ms() > flag.gate.max_p99_ms:
           flag.set_percentage(prev_step(steps, cur))   # step down, keep serving
           return "held"
       flag.set_percentage(nxt)
       return "advanced"

4. Verify the database tolerates the higher share at each step. Rising traffic on the new index or column can surface lock contention that was dormant at 1%.

   -- MySQL 8.0 · user with PROCESS privilege · read-only · watch new-path lock contention as share rises
   SELECT * FROM performance_schema.data_locks WHERE OBJECT_NAME = 'orders';

5. Hold at 100% through a full business cycle, then retire the flag with the legacy path. Only after the top step is stable do you tear down the branch, the sequence covered in coupling schema changes to feature flags and removing both.

Verification Checklist

• The schema migration is additive and already applied, so every step of the ramp — including 0% — runs against a live, valid database.
• Traffic is bucketed on a stable key (user_id or tenant_id) and the observed new-path share matches the configured percentage within tolerance.
• A per-path metric is emitted on every gated request, so the gate reads new-path error rate and p99 in isolation, not a blended average.
• Each percentage bump holds for the full observation window and advances only when the new-path SLO budget passes.
• A budget breach automatically steps the percentage down (or to 0) with no DDL, and background contexts stay pinned to legacy until 100% holds.
• The flag resolves to the legacy path on a flag-service outage, so an eval failure never strands traffic on the new path mid-ramp.

Up one level: Feature Flag Rollouts.

Related

• Using Feature Flags to Toggle Schema Changes Safely
• Coupling Schema Changes to Feature Flags and Removing Both
• Tracking Schema Migration Metrics and SLOs

Frequently Asked Questions

Why ramp by percentage instead of just toggling the flag on? A binary toggle exposes 100% of traffic to any regression the instant you flip it, so your blast radius is the whole fleet and your only recovery is a fast rollback after the damage is done. A percentage ramp exposes the new path to a small, controllable slice first, so a latency or error regression shows up on 1–5% of traffic where a metrics gate can catch it and hold the ramp before most users ever touch the new path.

How do I stop a bad ramp from advancing automatically? Tie every percentage bump to a metrics gate that reads the new path in isolation over a fixed observation window, and make the controller step the percentage down on a budget breach rather than merely pausing. Because the schema is additive, stepping down — even all the way to 0% — issues no DDL and is safe on the next request, so the ramp fails closed toward the known-good legacy path.

Why does my configured percentage not match the traffic I observe on the new path? Almost always an unstable bucket key or a stale SDK cache. If you bucket on something that changes per request (a request ID or random value) a user smears across both paths, so the split never settles at the configured fraction; bucket on a stable user_id or tenant_id instead. If the numbers lag the config, a poll-based SDK has not refreshed yet — wait one poll interval or switch to a streaming SDK before advancing.