Canarying Schema Changes on a Subset of Tenants

The migration added a column and an index to every tenant’s invoices table and passed every test. It ran across all 3,000 tenants in forty minutes. The next morning, the 60 tenants on the legacy billing plan could not issue invoices, because their data had a pattern the new constraint rejected — something no test fixture contained. Fleets have a structural advantage single databases lack: you can try a change on a few tenants and watch before touching the rest. A canary wave turns “tested on fixtures” into “proven on real tenants with real data and real traffic”, and a health gate between waves turns a fleet-wide incident into a small, reversible one. This guide designs canary groups, defines the gate, and connects it to the orchestrator. It belongs to Migrating Multi-Tenant Databases.

Waves With Health Gates Rollout pipeline. Wave 0: internal and test tenants. Gate: no errors. Wave 1: canary of 1 percent representative tenants including a large one. Gate: canary error rate and latency within bounds of control group for a soak period. Wave 2: 10 percent. Gate. Wave 3: remainder. Waves With Health Gates Wave 0 internal tenants clean no errors? Wave 1 canary 1%, representative health ≈ control? Waves 2–3 10%, then rest stop, fix hold fleet fail
Each gate compares migrated tenants with not-yet-migrated ones, so the signal is the migration's effect, not the day's traffic.

Symptom / Error Signatures

You need canaries if fleet migrations have ever caused:

  • A problem that affected every tenant at once, discovered only after the rollout completed.
  • Failures limited to a subset of tenants with unusual data, a legacy feature, or a very large dataset — invisible in staging.
  • Performance regressions on tenants of a particular size, for example a new index that is harmless on small tenants and slow to build or to maintain on large ones.

A canary gate that works reports something like: wave 1 (32 tenants): error rate 0.41% vs control 0.39%, p99 212 ms vs 205 ms — PASS, or FAIL: invoice creation errors 4.8% vs 0.2% on 3 canary tenants (plan: legacy_billing).

Root Cause Analysis

Tests and staging use fixtures and a handful of synthetic tenants. Production tenants differ in size, feature usage, data history and traffic, and migrations interact with all four: a backfill’s duration depends on data volume, a constraint’s validation depends on historical data quality, a new index’s cost depends on write rate. The only reliable way to learn how a migration behaves across that diversity is to apply it to a sample of real tenants and observe.

Two design choices make the observation meaningful. The canary group must be representative: stratified across plans, sizes, regions and feature flags, including at least one of the largest tenants. And the gate must compare the canary with a control group of not-yet-migrated tenants over the same period, so ordinary fluctuations (a traffic spike, an unrelated incident) affect both and cancel out.

Canary selection What it catches What it misses
internal/test tenants only syntax, obvious errors real data, scale
random 1% common data patterns rare plans, largest tenants
stratified 1% + largest tenants most data patterns, scale effects extremely rare edge cases
customer-opted “early adopters” real usage with tolerance may not be representative
A Stratified Canary Group (Example) Matrix showing how a 32-tenant canary is drawn from a 3,000-tenant fleet across strata: plan, size band and special features, with counts for each. A Stratified Canary Group (Example) Stratum Fleet share Canary tenants standard plan, small 62% 10 standard plan, large 18% 6 enterprise 9% 5 (incl. the largest tenant) legacy billing plan 2% 3 (over-sampled) other regions 9% 8
Stratifying the canary makes sure every plan, size band and legacy feature is represented before the fleet is exposed.

Immediate Mitigation

If a fleet migration without canaries is causing problems now:

1. Stop the rollout. Mark remaining tenants hold in the orchestrator’s status table so no further tenants are migrated.

-- PostgreSQL · registry · pause the fleet
UPDATE tenant_migrations SET status = 'hold' WHERE status = 'pending';

2. Find the pattern among affected tenants. Join error data to tenant attributes (plan, size, features, region) to see what the failing tenants have in common.

3. Fix forward or roll back only the affected tenants, then resume the rollout with a canary wave that deliberately includes tenants with the problematic attribute.

Permanent Fix / Long-Term Pattern

Make waves and gates part of the orchestrator, not a manual habit.

1. Assign waves automatically from the tenant registry: wave 0 for internal tenants, wave 1 for a stratified 1–2% sample plus the largest tenants, wave 2 for 10–20%, wave 3 for the rest. Re-draw the canary for each rollout so the same customers are not always first.

-- PostgreSQL · registry · stratified canary: ~1% per plan/size stratum plus the 3 largest tenants
WITH ranked AS (
  SELECT tenant_id, plan, size_band,
         row_number() OVER (PARTITION BY plan, size_band ORDER BY random()) AS rn,
         count(*)     OVER (PARTITION BY plan, size_band) AS n
  FROM tenants WHERE internal = false
)
UPDATE tenant_migrations tm SET wave = 1
FROM ranked r
WHERE tm.tenant_id = r.tenant_id
  AND (r.rn <= greatest(1, ceil(r.n * 0.01))
       OR r.tenant_id IN (SELECT tenant_id FROM tenants ORDER BY data_bytes DESC LIMIT 3));

2. Define the gate as a query over metrics you already collect: error rate for the key operations, p95/p99 latency, database load attributable to the canary tenants — each compared with the control group over a soak period long enough to include the tenants’ normal activity (often a few hours; a full business day for billing-type features).

3. Hold on failure, automatically. A failed gate sets remaining waves to hold and pages the owner; a pass promotes the next wave. Record gate results with the rollout for later review.

Canarying works only if the application tolerates a mixed fleet, which requires backward-compatible migrations — the discipline in Expand and Contract Methodology. The same wave structure is useful for feature enablement after the schema is in place, combining with progressive schema rollout with percentage flags.

Canary vs Control Error Rate During a Soak Line chart of error rate over six hours after migrating the canary wave. Control group stays around 0.3 percent. A healthy canary tracks it closely. A faulty canary rises to about 4.8 percent within two hours, triggering the gate. Canary vs Control Error Rate During a Soak 0 2 4 0 h 1 h 2 h 3 h 4 h 5 h 6 h hours since canary wave error rate (%) gate threshold control healthy canary faulty canary
Comparing against a control group over the same hours separates the migration's effect from normal daily variation.

Verification Checklist

Frequently Asked Questions

How large should the canary be? Large enough to be representative — usually 1–2% of tenants, stratified by plan, size and features, plus the largest tenants — and small enough that a failure affects few customers.

How long should the canary soak? Long enough to exercise the code paths the migration affects. For migrations touching frequently used tables, a few hours is often enough; for monthly or daily processes such as billing, soak through at least one run of those processes.

Should the largest tenants go first or last? Include at least one in the canary to surface scale problems early, then decide per migration: many teams run the remaining largest tenants in a dedicated wave with extra monitoring.

Can canaries work for shared-table designs? Not for the DDL itself, which applies to all tenants at once. They still work for behaviour: gate the new code path by tenant (a feature flag keyed by tenant) and roll it out in waves after the backward-compatible schema change.

Related Articles