Fast Wrong Is Worse Than Slow Right

The fastest optimization I tested was also the most dangerous. It cut runtime dramatically, made the benchmark tables look easy, and would have silently corrupted production data. That project sharpened one lesson: fast wrong is worse than slow right.

Disclaimer: This case study is anonymized and generalized. It does not disclose client names, proprietary data, source code, credentials, internal architecture diagrams, or business-sensitive details. The focus is on transferable engineering patterns and lessons learned.

The Problem

A critical daily analytics pipeline had been running for years. Every morning, it recomputed several downstream tables from very large source data. The job was expensive enough that optimization mattered, but trusted enough that any redesign needed strong correctness guarantees.

The reason nobody had “just optimized it” earlier was simple: the pipeline was expensive, but it was trusted. Downstream teams used the output for operational work. A bad optimization would not just waste compute; it would break a system people already depended on.

At benchmark scale, the waste was obvious. Almost all runtime was spent on two full scans over the same huge source. Not the writes. Not the output tables. Not the window functions. The expensive part was structural.

How I Worked the Problem

I treated this as an architecture and validation problem, not a one-off timing exercise.

I built a benchmark harness that varied three things:

Axis	What I tested	Why it mattered
Data scale	Tens of millions through low single-digit billions of rows	Small data hides correctness bugs and scaling behavior.
Warehouse size	Small through large	Large warehouses can mask algorithmic waste by brute-force parallelism.
Correctness	Row-level EXCEPT validation against a known-good baseline	Aggregate counts can match while the actual rows are wrong.

I also used two coding agents in parallel, but with strict role separation. I used Codex CLI for planning, benchmark design, and architectural reasoning around query shape, invariants, and failure modes. I used Cortex Code for execution inside the Snowflake workflow: implementing the Snowpark changes, building the benchmark notebook, and iterating on the validation queries. The contract between all of us was the same: benchmark tables plus row-level validation, not vibes.

That operating pattern matters. On hard optimization work, agents are most useful when they compress the distance between hypothesis and evidence, not when you let them free-associate their way into production.

The Tempting Optimization

The obvious idea was an incremental design:

1. Seed a persistent activity table from historical data once.
2. On the daily run, scan only new rows since a watermark.
3. Append those new aggregates to the activity layer.
4. Recompute the smaller downstream outputs from the cached activity table instead of rescanning the full source.

This “Dual-Incremental” design looked fantastic in benchmark tables:

Dataset	Full Recompute	Dual-Incremental Daily	Speedup
Mid-scale	85 seconds	17 seconds	5x
Large-scale	360 seconds	30 seconds	12x

If I had done the usual optimization theater, I would have shipped it.

Then I ran the correctness validation.

The Bug That Scales With Data

The failure mode was subtle.

Some activities straddled the watermark: part of the history had already been processed in the initial seed, and new rows for the same activity arrived later. An append-only design cannot repair the old aggregate. It creates a second partial activity row instead of recomputing the one correct row from full history.

That produced phantom rows, missing rows on some filter paths, and wrong aggregates built from partial state.

On small data, everything looked clean. At larger scales, row-level validation started surfacing structural diffs: first a few, then a few dozen. That was enough. The architecture was wrong.

The important point is that the fastest design was not “needs a little tuning.” It was fundamentally invalid.

The Real Fix: Shared-Incremental with MERGE

The working design was incremental, but not append-based.

The final approach used one shared activity table and MERGE semantics:

• identify activity keys touched by new data
• re-read the full history for those keys
• recompute the aggregate from scratch
• MERGE the corrected result back into the persistent table

This moved the incremental boundary to the right level: not at arbitrary row fragments, but at the entity whose aggregate must remain correct.

A second pass was still needed for LAG-based logic, because updating one row can change the next row in the same device partition. Hard optimization work is usually hidden in these dependency edges, not in the headline idea.

What the Final Numbers Looked Like

Approach	Correctness	Outcome
Dual-Incremental	Failed row-level validation at large scale	Fastest, but unusable
Shared-Incremental + MERGE	Zero structural diffs in completed validation runs	50%+ compute savings; 5-10x faster than baseline
Full recompute	Correct	Baseline

In production terms, that meant:

• daily runtime dropped by 5-10x
• compute usage fell by more than 50%
• the one-time seed cost paid back quickly

The broken design was faster. The correct design was still transformative.

Why EXCEPT Validation, Not Checksums

This was the real lever: not a Snowpark trick, but benchmark discipline.

Small data hid the bug. Large warehouses hid the algorithmic cost. Aggregate checks would have let the broken design through. Row-level validation killed it before it got near production.

The validation queries were simple:

SELECT * FROM candidate
EXCEPT
SELECT * FROM ground_truth;

SELECT * FROM ground_truth
EXCEPT
SELECT * FROM candidate;

Simple is fine. The important part was running them systematically, across scales, across warehouse sizes, and against a known-good baseline.

What This Says About AI-Agent Work

The agents mattered, but not because they replaced judgment.

They let me explore architectures, implement benchmark harnesses, generate validation queries, and debug long-running Snowpark iterations much faster than I could have by hand. More importantly, they let me separate planning from execution in a way that matched the tools. I have been impressed by Snowflake’s Cortex Code ability to seamlessly orchestrate workflows on Snowflake, read logs, give updates, and even setup cron jobs for status checks for long running jobs. GPT-5.4 Pro (inside Codex CLI) was instrumental in crafting the shared incremental approach and debugging the dual incremental approach.

That is the mode I increasingly use on hard production problems:

• parallel agents with distinct responsibilities
• explicit validation gates
• benchmark tables as the source of truth
• human judgment on architectural calls and go/no-go decisions

The agents accelerated the work. The evidence standard stayed high.

Takeaway

The right question was not “how do I make this pipeline faster?” It was “what part of the current design is fundamentally wasting work, and how do I change that without breaking correctness?”

That is the optimization work that actually matters. Not shaving seconds off a bad architecture. Not shipping the prettiest benchmark. Changing query shape, proving correctness, and killing attractive wrong ideas before they escape.

Fast wrong is worse than slow right. The useful pattern was: redesign the work, benchmark across scales, validate row by row, and only then talk about speedup.