Rolling Forward Instead of Rolling Back

The release added a status_v2 column, backfilled it, and switched reads to it. Two hours later a reporting bug appeared, and the on-call engineer ran the down migration to “roll back”. It dropped status_v2 — along with two hours of status changes that had been written only there. The report bug was fixed an hour later; the lost status changes took two days to reconstruct from logs. Down migrations are exact inverses of schema changes and blind to data: once real writes land in new structure, reversing the schema destroys them. For databases, the safer default after deploy is usually to roll forward — ship a new, small, corrective change — and to design releases so that rolling back the application never requires rolling back the schema. This guide sets out when each is appropriate and how to make fixing forward fast. It belongs to Rollback Automation.

Roll Back or Roll Forward? Decision tree after a problematic release. If the application can be rolled back without touching the schema, roll back the application only. Otherwise, if the migration is additive and nothing has written to the new structure yet, the down migration is safe. Otherwise roll forward with a corrective migration. Roll Back or Roll Forward? Can the app roll back without schema changes? yes no Roll back the application only Additive and nothing written yet? yes no Down migration is safe Roll forward: corrective migration
Rolling back code is almost always safe; rolling back schema is safe only before data lands in it.

Symptom / Error Signatures

Teams discover the need for a roll-forward policy after incidents such as:

  • A down migration that dropped a column or table containing writes made since deploy.
  • A down migration that failed because data written after deploy violates the old schema (for example, values longer than the old column limit, or NULLs where the old schema had NOT NULL).
  • A rollback that took longer than the original problem, because reversing a large backfill or rewrite is itself a long migration.
  • IrreversibleMigration or missing down methods discovered during the incident.

Root Cause Analysis

A migration changes two things: structure and, often implicitly, data. Down migrations reverse the structure mechanically — add becomes drop, rename becomes rename back — but they cannot reverse what happened to data in between. Once the application has written to new structure, the new structure holds information the old one never had, and removing it loses that information. And some changes are not reversible at all without data work: narrowing a type, restoring a dropped column’s contents, undoing a backfill that overwrote values.

Change Down migration after writes Better response
add nullable column drops column and its new data keep column; roll back app only
add table drops table and its rows keep table; roll back app only
backfill / data transform reverse transform may be lossy corrective forward migration
widen type narrowing may fail on new values keep wider type
drop column (contract) column comes back empty restore from backup (see recovery guide)
add constraint drop constraint — safe drop constraint forward or backward

The strategy that makes rollback cheap is designing releases so the application can be rolled back without touching the schema: every migration is backward compatible with the previous release, destructive steps wait for a later release, and new behaviour is behind flags. Then an application rollback is a redeploy, and schema problems are fixed forward calmly.

Rolling Back Schema vs Rolling Forward Two panels. Rolling back schema after writes: drops new structure and the data written to it, may fail on data the old schema cannot hold, and can take as long as the original migration. Rolling forward: application rolled back or flag turned off immediately, then a small corrective migration shipped through the normal pipeline; no data lost. Rolling Back Schema vs Rolling Forward Roll back the schema down migration drops new structure data written since deploy lost may fail on new data can be as slow as the original risky after writes Roll forward roll back app / disable flag now keep the schema ship corrective migration normal review + pipeline no data loss
Rolling forward separates "stop the bleeding" (app rollback or flag) from "fix the schema" (a reviewed corrective change).

Immediate Mitigation

When a release with a migration is misbehaving:

1. Stop the user-facing damage without touching the schema. Roll back the application to the previous release, or turn off the feature flag guarding the new behaviour. Because migrations are backward compatible, the previous release runs correctly against the new schema.

2. Assess whether a schema change is needed at all. Many incidents are code bugs that only look like schema problems. If the schema is fine, fix the code and redeploy.

3. If the schema is wrong, write a corrective forward migration. For example, a mistakenly strict constraint is relaxed; a wrong default is corrected; a bad backfill is re-run correctly — each as a normal, reviewed migration.

-- PostgreSQL · corrective forward migration · replaces a constraint that rejected valid data
-- WARNING: runs through the normal pipeline with lock_timeout; no data is dropped.
SET lock_timeout = '3s';
ALTER TABLE orders DROP CONSTRAINT IF EXISTS orders_status_v2_check;
ALTER TABLE orders ADD CONSTRAINT orders_status_v2_check
  CHECK (status_v2 IN ('pending', 'paid', 'shipped', 'refunded', 'partially_refunded')) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_status_v2_check;
-- ROLLBACK PATH: this is itself a forward fix; the previous constraint definition is in migration V71.

4. Use a down migration only when it is provably harmless — the migration was additive and nothing has written to the new structure (check with a count), or the reverse is a pure constraint or index removal.

Permanent Fix / Long-Term Pattern

Adopt roll-forward as the default and make it fast. The ingredients:

  • Backward-compatible releases. Every migration works with the previous application version, so an application rollback never needs a schema rollback — the rule in running blue-green deploys with a shared database.
  • Flags around new behaviour. Turning off a flag is faster than any deploy, as in using feature flags to toggle schema changes safely.
  • A fast corrective path. The pipeline can ship a small migration within minutes: pre-approved templates for common fixes (relax a constraint, drop a new index, add a default), an expedited review for high-severity incidents, and the usual gates still running.
  • Down migrations kept honest. Keep them for development and CI, test them, and mark genuinely irreversible migrations explicitly — the guidance in writing safe down migrations for automated rollback. Production runbooks should say when they may be used.

For the rare case where data really was destroyed by a migration, recovery is a separate procedure, covered in recovering data after an irreversible migration.

Incident Response With Roll-Forward Five steps. Detect the problem; roll back the application or disable the flag within minutes; diagnose whether schema or code is at fault; ship a corrective forward migration or code fix through the expedited pipeline; review the incident and adjust templates or checks. Incident Response With Roll-Forward STEP 1 Detect alerts, errors STEP 2 App rollback / flag off minutes, no schema change STEP 3 Diagnose schema or code? STEP 4 Corrective change expedited pipeline STEP 5 Review templates, checks
Minutes to stop the damage, then a normal (but expedited) change to fix it — never an improvised schema reversal.

Verification Checklist

Frequently Asked Questions

Are down migrations useless then? No. They are valuable in development, in CI round-trip tests, and in production for additive changes that nothing has written to yet. They are dangerous as a reflex after real traffic has written to new structure.

What if the previous application version cannot run against the new schema? Then the release broke the backward-compatibility rule, and the immediate fix is to restore compatibility forward — re-add a dropped column, recreate a renamed object as a view, restore a default — rather than reverse everything.

How fast can a corrective migration realistically ship? With templates for common fixes and an expedited review, minutes to tens of minutes. The key is that the pipeline’s automated checks are fast and remain in place; skipping them turns a fix into a second incident.

Does rolling forward mean never reverting a migration? It means reverting only when reversal is provably harmless. Dropping an unused new index or a constraint is a reversal and is fine; dropping a column full of new data is not.

How do I know nothing has written to new structure? Count it: rows in a new table, non-null values in a new column, or entries in a new index’s table. If the count is zero and the flag was never enabled, the down migration loses nothing.

Related Articles