Running Blue-Green Deploys with a Shared Database

The platform supports blue-green deploys: the green environment boots the new release, smoke tests run against it, the load balancer flips, and blue stays warm for two hours as the instant rollback. The team loved it until the first release with a schema change. The migration ran before green booted — and blue, still serving all production traffic, immediately started failing because a column it reads had been renamed. The next attempt ran the migration after the flip; green failed on startup because the column it needed did not exist yet. Both environments share one database, so the database must serve both versions at the same time for as long as both might receive traffic — including the two hours blue is kept as a rollback target. This guide defines that compatibility window, orders migrations around the switch, and keeps rollback safe. It belongs to Blue-Green Deployments for Databases.

Symptom / Error Signatures

Shared-database blue-green failures come in two shapes, depending on when the migration runs:

• Migration before the switch, not backward compatible: blue — still serving everyone — errors immediately: column "customer_ref" does not exist, null value in column ... violates not-null constraint, or ORM mapping errors for a changed type.
• Migration after the switch: green fails health checks or errors on first requests: relation "loyalty_tiers" does not exist, because it booted against the old schema.
• Rollback breaks: traffic returns to blue, but a contract migration already ran for green, so blue fails exactly as in the first case.

Root Cause Analysis

Blue-green makes two properties explicit that rolling deploys also rely on. First, both versions run against the database at the same time — during green’s smoke tests, and for the whole time blue is kept as a rollback target. Second, rollback is a traffic switch, not a redeploy, so it happens in seconds and gives no time to reverse a migration. Together they mean the schema at any moment must satisfy both vN (blue) and vN+1 (green).

The rule that follows is the classic N-1 compatibility contract: a release may only apply migrations that the previous release can run against, and may only remove structure that the previous release no longer uses. In practice:

Immediate Mitigation

1. If blue is failing after a pre-switch migration, restore compatibility rather than reverting blindly: re-add a dropped column as nullable, recreate a renamed table’s old name as a view (see renaming a table with an updatable view), or restore a database default. These are instant catalog changes.

-- PostgreSQL · migration role · restores a column blue still selects
SET lock_timeout = '3s';
ALTER TABLE orders ADD COLUMN IF NOT EXISTS customer_id bigint;
-- ROLLBACK PATH: drop it again only after blue is retired.

2. If green fails on boot because the schema is missing, run the expand migration now — it should be backward compatible by design — then restart green’s health checks. Do not switch traffic until green is healthy.

3. Before any rollback switch, confirm that no contract migration has run since blue was last serving. If one has, restore compatibility first (step 1), then switch.

Permanent Fix / Long-Term Pattern

Build the deploy pipeline around the N-1 contract:

1. Expand migrations for vN+1 run first, while blue serves all traffic. Linting and review confirm each is backward compatible, as in enforcing backward compatibility checks in pull requests.
2. Green boots and is tested against the expanded schema.
3. Traffic switches to green; blue stays warm for the rollback window.
4. Blue is retired at the end of the window.
5. Contract migrations for anything vN+1 made obsolete ship with vN+2, after step 4 — never with vN+1.

# YAML · deploy pipeline sketch (generic CI) · stage order enforces N-1 compatibility
# WARNING: the contract job belongs to the NEXT release's pipeline, gated on blue being retired.
stages:
  - name: expand-migrations        # backward-compatible DDL only; lint-gated
    run: ./migrate --phase expand
  - name: deploy-green
    run: ./deploy --color green --release "$RELEASE"
  - name: smoke-test-green
    run: ./smoke --target green
  - name: switch-traffic
    run: ./lb switch --to green
  - name: retire-blue
    when: after 2h without rollback
    run: ./deploy --color blue --scale 0

Tag each migration file with its phase (expand or contract) and let the pipeline refuse to run a contract migration while the previous color is still available for rollback. Feature flags help decouple enabling green’s new behaviour from the traffic switch, as in using feature flags to toggle schema changes safely. When the change is to the database itself — a major version, a new instance — the problem becomes database blue-green, covered in using logical replication for blue-green database cutover.

Verification Checklist

• Every migration in a release is tagged expand or contract, and contract migrations never run while the previous color can receive traffic.
• Expand migrations run before green boots, and blue’s error rate is monitored while they apply.
• Green passes smoke tests against the shared database before any traffic switch.
• A rollback switch was rehearsed in staging with the new schema in place.
• Blue is retired before the next release’s contract migrations run.

Frequently Asked Questions

Should migrations run before or after the traffic switch? Expand migrations run before green boots, because green needs them and blue tolerates them. Contract migrations run after blue is retired, in a later release. Nothing destructive runs around the switch itself.

How long is the compatibility window? From the expand migration until blue is retired — including the entire rollback window. If blue is kept warm for a day, the schema must support both versions for a day.

Is this different from a rolling deploy? The rules are the same, but blue-green makes the overlap longer and the rollback faster, so violations are more likely to be exercised. A schema that is safe for blue-green is safe for rolling deploys.

What about background workers and scheduled jobs? They are part of blue and green too. Make sure blue’s workers are stopped or switched along with web traffic, and that jobs scheduled by blue do not run against structures green has changed.

Can two databases avoid the compatibility problem? Not for writes: if blue and green used separate databases, writes made to one during the window would be missing from the other. Shared-database blue-green with N-1 compatibility is simpler and safer for application releases.