Handling Partial Failures in Fleet-Wide Migrations

The fleet rollout finished with 2,951 tenants migrated and 49 failed. Eleven failed on lock timeouts, twenty-three on a unique violation during an index build, nine because their schema was missing a column an older migration should have added, and six with connection errors during a network blip. The next deploy is in two days and depends on the new column existing everywhere. In a single-database world, a failed migration is binary; in a fleet, partial failure is the normal outcome of any large rollout, and the job is to converge the stragglers without blocking everyone else — and without letting the application assume a version the fleet has not reached. This guide classifies failures, retries the transient ones, quarantines the rest for individual repair, and keeps code compatible with version skew until the fleet converges. It belongs to Migrating Multi-Tenant Databases.

Triage for a Failed Tenant Decision tree. If the error is transient, such as a lock timeout or connection error, retry automatically with backoff. If it is a data problem such as a constraint or unique violation, quarantine the tenant and repair its data. If it is schema drift such as a missing object, quarantine and bring the tenant's schema up to the expected baseline, then retry. Triage for a Failed Tenant Transient (lock timeout, connection)? yes no Auto-retry with backoff Data problem (constraint, duplicate)? yes no Quarantine; repair data; retry Quarantine; fix schema drift; retry
Retry what time will fix; quarantine and repair what it will not.

Symptom / Error Signatures

Partial failures show up in the orchestrator’s status table and in per-tenant logs:

tenant_0917  failed  ERROR: canceling statement due to lock timeout                              -- transient
tenant_1204  failed  ERROR: could not create unique index "invoices_number_key" ... is duplicated -- data
tenant_2230  failed  ERROR: column "region" of relation "invoices" does not exist                 -- drift
tenant_3011  failed  FATAL: terminating connection due to administrator command                  -- transient

The downstream symptom, if nothing is done, is application errors on straggler tenants after the next release assumes the new schema: column "tax_region" does not exist on exactly the 49 tenants that failed.

Root Cause Analysis

Across thousands of tenants, rare conditions become certainties: some tenant is busy when its migration runs, some tenant has duplicate data a new constraint rejects, some tenant missed an earlier migration because of an old incident, some connection drops. A fleet orchestrator should expect a small failure rate and treat each class differently:

Failure class Examples Automatic action Human action
transient lock timeout, deadlock, connection reset retry with backoff, up to N attempts none unless persistent
data unique violation, check violation, cast failure quarantine clean or migrate data, then retry
drift missing column, unexpected object, wrong version history quarantine reconcile schema to baseline, then retry
migration bug fails on every tenant with a feature stop the wave fix the migration

The application side matters as much. While stragglers remain, the fleet spans versions, so code must work on the oldest version still present. The safest rule is to derive the fleet’s minimum version from the status table and gate any code that needs a newer schema on either that minimum or on the individual tenant’s version.

Outcome of a 3,000-Tenant Rollout Stacked bar of tenant outcomes. First pass: 2,951 succeeded, 17 transient failures, 23 data failures, 9 drift failures. After automatic retries: 2,968 succeeded, 23 data, 9 drift. After repairs: 3,000 succeeded. Outcome of a 3,000-Tenant Rollout first pass 2951 after retries 2968 after repairs 3000 succeeded transient data problem drift
Retries clear the transient failures; the remaining few need individual repair before the fleet converges.

Immediate Mitigation

1. Classify the failures. Group failed tenants by error pattern from the status table.

-- PostgreSQL · registry · read-only · failures grouped by error signature
SELECT regexp_replace(last_error, '"[^"]*"', '"…"', 'g') AS error_pattern, count(*)
FROM tenant_migrations
WHERE status = 'failed'
GROUP BY 1 ORDER BY 2 DESC;

2. Retry the transient ones with backoff; they usually succeed.

-- PostgreSQL · registry · requeue transient failures
UPDATE tenant_migrations SET status = 'pending'
WHERE status = 'failed'
  AND (last_error ILIKE '%lock timeout%' OR last_error ILIKE '%deadlock%' OR last_error ILIKE '%connection%');

3. Quarantine the rest so the orchestrator stops retrying them, and create a ticket per class with the affected tenants.

-- PostgreSQL · registry
UPDATE tenant_migrations SET status = 'quarantined' WHERE status = 'failed';

4. Protect the next release. Before deploying code that needs the new schema, check the fleet minimum; if stragglers remain, keep the code path gated.

-- PostgreSQL · registry · lowest version any active tenant is on
SELECT min(current_version) AS fleet_min FROM tenants WHERE active;

Permanent Fix / Long-Term Pattern

Build failure handling into the orchestrator. Every tenant attempt records the error; transient errors trigger automatic retries with exponential backoff and a cap; other errors move the tenant to quarantine with a notification. A daily report lists quarantined tenants and their age, so stragglers cannot be forgotten. Deploy tooling refuses to ship code that requires a schema version above the fleet minimum unless the code is gated per tenant.

Reduce each class at the source. Transient failures shrink with short lock timeouts plus retries, and with migrations scheduled per tenant outside their busy hours. Data failures shrink with a pre-flight check that runs the migration’s preconditions (duplicate checks, constraint checks) across the fleet before the rollout, turning a failure into a to-do list. Drift failures shrink with regular reconciliation of every tenant’s schema against the expected baseline, as in tracking schema versions across thousands of tenants. Make every tenant migration idempotent so retries are always safe, per Idempotent Script Design.

-- PostgreSQL · pre-flight precondition check run on every tenant before the rollout
-- tenants returning rows need data cleanup before the unique index can be built
SELECT number, count(*) FROM invoices GROUP BY number HAVING count(*) > 1 LIMIT 5;
Converging a Fleet After Partial Failure Five steps. Classify failures by error pattern; auto-retry transient failures with backoff; quarantine data and drift failures with tickets; repair each quarantined tenant and requeue it; confirm the fleet minimum version reaches the target before dependent code ships. Converging a Fleet After Partial Failure STEP 1 Classify by error pattern STEP 2 Auto-retry transient only STEP 3 Quarantine data + drift STEP 4 Repair + requeue per tenant STEP 5 Fleet min = target then ship dependent code
The rollout is finished when the fleet minimum reaches the target — not when the last wave ends.

Communicate with affected customers when a failure is theirs to see. A quarantined tenant usually experiences nothing — it simply stays on the previous version, which the application still supports — but if the fix requires cleaning their data (merging duplicates, correcting invalid values), the change may be visible to them. Agree a process with support for tenant-specific data repairs, including a record of what was changed and why, so a migration’s data cleanup never surprises a customer.

Verification Checklist

Frequently Asked Questions

Should one failed tenant stop the whole rollout? No, unless failures indicate a bug in the migration itself — for example, every tenant with a certain feature failing. Isolated failures should be retried or quarantined while the rest of the fleet continues.

How do I know when the rollout is really done? When the minimum schema version across all active tenants equals the target. Track it from the status table and verify it against the tenants’ own history tables.

Is it safe to retry a failed tenant migration? If the migration is transactional (PostgreSQL DDL) or idempotent, yes. Non-transactional steps, such as concurrent index builds, need idempotent guards and invalid-index cleanup before a retry.

How many retries are reasonable for transient failures? Three to five, with exponential backoff spanning minutes. A tenant that keeps timing out on locks is usually busy at that time of day; reschedule it for its quiet hours rather than retrying immediately.

What causes schema drift between tenants? Past partial failures that were never repaired, manual fixes on individual tenants, tenants restored from old backups, or tenants created from an outdated template. Regular reconciliation against the expected baseline catches them before the next rollout.

Related Articles