Premium customers silently downgraded by a column rename — 2023

Cost: ~3% of premium customers misclassified as free-tier in lifecycle marketing for ~9 days; one regulator-reportable retention-offer mailing reissued · Time-to-detect: ~9 days (PR merge to first customer-success escalation) · Root cause class: T2 (schema-coherent / dangling reference)

What happened

A B2C subscription business ran a renaming sweep across its customer schema: customer_tier was renamed to tier in dim_customers, with the old name dropped in the same migration. The dbt PR changed every model the engineer could grep for. The PR's dbt build passed, the test suite passed, and the model deployed.

What the grep missed: a downstream model lifecycle_segment_export, owned by a different team and stored in a sibling repo, referenced dim_customers.customer_tier through a Jinja macro that constructed the column name dynamically — {{ customer_attr_col('tier') }}. The macro returned customer_tier as a string. Postgres did not error: the column had been dropped, but the macro's output passed through a coalesce(..., 'free') guard that an analyst had added years earlier as defensive coding. Every customer's tier silently became 'free'. For the ~3% of customers on the premium plan, lifecycle automation began sending free-trial-conversion emails — including a "come back, here's 20% off" offer to customers already paying full price.

A premium customer screenshotted the offer to support; CS escalated; the regression was traced in roughly an hour.

The pattern

A column rename in one repo broke a reference in another repo, but every layer between them — the SQL parser, dbt's compile step, the warehouse — accepted the break silently because each layer's local invariants still held. The macro produced a syntactically valid column name; the warehouse returned a typed result; the coalesce swallowed the NULL.

Any cross-repo or cross-team data dependency where a referenced symbol can be renamed without the renamer seeing every transitive consumer has this exposure: dbt models in separate repos, Looker views referencing warehouse columns, downstream Python ETL jobs, BI tools with hand-written SQL.

How veric would catch it

veric's T2 tier resolves every column reference in the workspace's model graph against the declared schema and flags any reference that does not resolve. Run as a PR check across the multi-repo workspace, the verifier would have produced: "model lifecycle_segment_export.premium_offer references column dim_customers.customer_tier; column not declared in current schema (declared columns: tier, customer_id, signup_at, ...) — T2 reference-sound VIOLATED."

The check catches the broader family: any workspace where a producer's schema and a consumer's references can drift apart without a hard, build-time failure.

Try it: open the example below and watch the verdict change as you toggle the offending pattern on and off.

Sources

Anonymised; pattern composited from public reports of the dbt + Fivetran column-rename failure mode (e.g. dbt Slack #i-made-a-graph 2022-2024 weekly recurrences) and the well-documented dynamic-Jinja-macro reference-loss class.