Treat every data migration like it will go wrong

On the last day of January 2017, an engineer at GitLab was fighting a database problem late at night. In the middle of it, they ran a deletion against what they believed was a secondary server. It was the primary. Around three hundred gigabytes of production data vanished in seconds. The team then went looking for the backups and discovered, one after another, that several separate recovery mechanisms had silently not been working for some time. The company recovered, and earned a lot of respect for live-streaming the whole ordeal, but the lesson stuck with a generation of engineers. The dangerous part was not the deletion. It was that nobody had checked the safety nets were real.

I think about that incident every time I open a terminal to run a one-off data script. We have learned to be careful with application code. We write tests, we review pull requests, we run continuous integration. Then someone needs to fix some data, writes a quick loop that updates a few thousand rows, and runs it straight against production with no test, no review, and no second pair of eyes. The script with the most direct power to corrupt your source of truth is usually the least scrutinised thing in the building.

Why data scripts fail silently

A bug in application code tends to announce itself. A page errors, a request fails, a log fills with stack traces. A bug in a data migration is quiet. It writes wrong values and exits cleanly, reporting success. The application keeps running, reports keep generating, and the wrong numbers sit there informing decisions until someone notices a total that cannot be right and traces it back to a script from months ago. You cannot rely on the system to tell you a migration went wrong. You have to build the telling yourself.

Dry run by default

Every data script I write does nothing on its first run. It reads, computes what it would change, prints a summary, and stops. Writing requires an explicit flag.

const DRY_RUN = !process.argv.includes('--apply');
for (const row of rows) {
  const change = computeChange(row);
  if (DRY_RUN) { console.log('would update', row.id, change); continue; }
  await applyChange(row.id, change);
}

The safe default is the whole point. When the destructive path needs an extra word typed, you cannot wipe a table by hitting up-arrow on the wrong line. And the dry-run output doubles as a review tool. Reading would update 4,000 rows when you expected forty is the cheapest bug you will ever catch.

Make it safe to run twice

Real migrations get interrupted. The connection drops at row 3,000 of 5,000, or you spot a flaw and need to rerun a corrected version. If the script is not safe to run more than once, every interruption becomes a careful manual reckoning. So I tag every row the script touches and skip rows that already carry the tag.

const MARKER = 'migrated-2026-05-pricing';
if (row.tags.includes(MARKER)) continue;   // already done
await applyChange(row.id, change);
await tagRow(row.id, MARKER);              // record that we did it

Now the script can be stopped and restarted freely. It resumes where it left off and never double-applies. The marker also leaves a permanent record of exactly which rows this migration touched, which is invaluable weeks later when you are reconstructing what happened.

Every import gets a verification script

This is the habit that matters most and the one almost nobody builds. For every script that writes data, I write a second script whose only job is to read the result back and compare it to the source. It changes nothing. It answers one question. Did the thing I intended actually happen?

I learned its value loading a structured plan from a spreadsheet into a database, around eighty rows across fifteen groupings, each carrying several fields. The import reported success and everything looked present. But only a verification pass, comparing the database back to the sheet field by field, would catch a row that arrived with a quietly missing value. The import saying it succeeded is not evidence it succeeded. The verification is the evidence.

let problems = 0;
for (const src of readSource()) {
  const row = await findInDb(src.id);
  if (!row) { console.error('MISSING', src.id); problems++; continue; }
  for (const field of fields)
    if (row[field] !== src[field]) { console.error('MISMATCH', src.id, field); problems++; }
}
console.log(problems === 0 ? 'verified clean' : problems + ' problems');

Run it right after the load and it confirms the import was clean. Run it a month later and it catches drift that crept in since, the kind no one would otherwise notice until it had done damage. GitLab’s backups looked fine too, right up until the day someone needed them. A migration is not finished when the script reports success. It is finished when a separate script has gone looking for the failure and found none.

Keep a migration log

One more habit ties these together. Every migration script writes a single line to a log when it applies changes: what ran, when, against how many rows, and which marker tag it used. It costs almost nothing, and it means that when a number looks wrong three months later, the first question, which migration touched this data, has an immediate answer instead of becoming an archaeology project. The verification script tells you whether a load was correct. The log tells you what happened and when. Together they turn data operations from a thing you do nervously and hope to forget into a thing you can actually audit.

None of this is exotic. Dry-run defaults, idempotent markers, paired verification, and a simple log are the kind of habits that feel like overhead on the day you build them and feel like the only sane way to work on the day they save you. Treat the script that mutates your source of truth with at least the care you give the code that merely reads it, because the script that writes is the one that can do real and lasting harm.

Featured image credit