Lambda vs Kappa architecture for DE interviews

Why this comes up on the DE interview

When a hiring manager at Stripe, Uber, or DoorDash asks you to "design a real-time analytics pipeline" or "serve sub-second metrics with daily-correct rollups," they are probing your understanding of Lambda and Kappa. These are the canonical reference architectures for systems that need both fresh answers (now) and correct answers (eventually). Fumble the distinction and the interviewer pegs you as someone who's only ever shipped batch DAGs.

The good news: the conceptual surface area is small. Lambda runs two pipelines in parallel — a fast, approximate stream path and a slow, exact batch path — and the serving layer reconciles them. Kappa collapses the two into one stream and uses log replay to recompute history. Once you can sketch both on a whiteboard in under two minutes, you can move on to the more interesting question: which one would you build today, and why?

Lambda architecture

Lambda was proposed by Nathan Marz around 2011, when "Big Data" meant Hadoop MapReduce for the batch tier and Storm for the stream tier. The whole idea is that you can't have one engine that is simultaneously high-throughput, low-latency, and exactly-correct, so you build two specialized engines and combine their outputs.

Source events
   ├─→ Speed layer  (Kafka Streams / Flink / Spark Streaming)
   │       ↓
   │   Real-time view (approximate, last few minutes)
   │
   └─→ Batch layer  (Spark / nightly rebuild from raw)
           ↓
       Batch view  (exact, up to last batch boundary)

   Serving layer  →  merge(real-time view, batch view)

The speed layer consumes the source stream and produces an incremental near-real-time view. It's allowed to be wrong at the edges — duplicates and approximate aggregations are tolerated because the batch layer eventually overwrites affected windows.

The batch layer is the source of truth. It re-reads the entire immutable raw log on a schedule (often nightly) and produces a fully-correct snapshot. Any bug or backfill is fixed by re-running the job.

The serving layer answers queries by stitching both views together — "use the batch view for everything older than the last batch boundary, then patch with the speed view for anything newer." Druid, Pinot, and BigQuery-style serving systems all implement variations of this.

The original promise was elegant: bugs in stream code can never permanently corrupt your warehouse, because batch always rewrites the truth. In practice, the pain is in the duplication.

The biggest operational cost of Lambda is that the same business logic lives twice — once in stream, once in batch — and the two implementations drift. New metrics have to be added to both. Schema changes have to be deployed to both. Most teams that have run Lambda in anger will tell you the speed layer and the batch layer slowly diverge in subtle ways that surface as user-facing inconsistency.

Kappa architecture

Jay Kreps, then at LinkedIn, published the counter-proposal in 2014: Questioning the Lambda Architecture. His argument: if your stream framework is expressive enough and your log is long enough, you don't need a batch layer at all. Just keep one pipeline.

Source events  →  Durable log (Kafka, long retention)
                       ↓
                  Stream processor (Flink / Kafka Streams)
                       ↓
                  Serving store

To reprocess history — say you found a bug or added a metric — spin up a new consumer that reads the log from offset zero, writes to a parallel output table, then atomically swap consumers. The log is your replayable source of truth; the stream job is your only piece of business logic.

The preconditions are real and worth memorizing for the interview:

The win is operational: one codebase, one deploy, one on-call surface, no merge logic in the serving tier. The cost is that you have to build everything in stream semantics, which is harder than batch SQL, and your storage bill for the long-retention log is real.

Lambda vs Kappa side by side

The pattern most teams converge to in 2026 is "Kappa-by-default, Lambda only when forced." The forcing functions are usually regulatory (financial reporting wants a deterministic nightly close) or organizational (the batch team and the stream team are different orgs and you can't merge them).

When to pick which

Pick Lambda when: you already operate a mature batch warehouse (Spark on Snowflake, Databricks SQL, BigQuery) and want to bolt on fresh dashboards without rewriting your gold tables. You have strict financial accuracy requirements — payments, billing, regulatory reports — where the nightly batch result is the audited truth and the stream view is a UX nicety. Your team has more batch experience than stream experience, and the cost of training up on Flink state management is higher than the cost of running two pipelines.

Pick Kappa when: the project is greenfield, you control the source events, and you can mandate Kafka (or equivalent) with long retention from day one. Your dominant access pattern is fresh data — fraud scoring, ride-matching, feed ranking, real-time personalization — where the batch result is uninteresting because by the time it lands, the decision has already been made. Your team has Flink or Kafka Streams expertise and the platform team to support it.

Pick neither — go lakehouse when you can. See the next section.

The modern lakehouse alternative

In 2026, the conversation has moved past the Lambda/Kappa dichotomy. The combination of Iceberg or Delta Lake as the storage format, Spark Structured Streaming or Flink as the compute, and dbt as the transformation layer gives you something neither classical architecture offered: ACID-correct streaming appends and batch transforms over the same physical data.

Source events  →  Kafka  →  Stream job (append to Iceberg / Delta)
                                ↓
                       Bronze table  (raw, append-only, ACID)
                                ↓
                       dbt incremental models  (silver / gold)
                                ↓
                       Serving: Trino / DuckDB / BigQuery / warehouse

The table format handles the hard parts: snapshot isolation, time travel, schema evolution, and compaction. Stream writers append; batch transforms read consistent snapshots; both see the same data with no merge logic. This is essentially "Kappa with the log replaced by an open table format" — you get the single-source-of-truth property without paying for infinite Kafka retention, because the bronze table is the replayable log.

If you mention this in the interview, signal the tradeoffs: Iceberg time-travel retention costs storage, schema evolution still requires discipline, and stream-to-table compaction is a real ops job. But on a greenfield project at Notion, Linear, or a typical Series B SaaS, this is what senior DE interviewers expect to hear.

Common pitfalls

The first pitfall is claiming Lambda when you've never operated two pipelines. Interviewers can smell rehearsed paper-knowledge. If you've only built nightly Spark jobs, say so, and walk through what would break if you bolted on a stream layer — the duplication of business logic, the merge complexity in serving, the on-call doubling. Honest framing beats memorized architecture diagrams.

The second pitfall is proposing Kappa without addressing log retention. The architecture rests on being able to replay history, which requires the log still exists. If the interviewer asks "how long is your Kafka retention?" and you say "default seven days," you've just admitted Kappa doesn't work for you. The fix: name a number — "90 days hot, with tiered storage to S3 for the rest" — or pivot to a lakehouse where the bronze table is the durable log.

The third pitfall is conflating exactly-once with strongly-consistent. Flink can give you exactly-once processing semantics within a job, but if your downstream serving store is eventually consistent (most NoSQL stores, search indexes, denormalized caches), the end-to-end user-visible result is still eventually consistent. Senior interviewers will probe this — the answer is to be precise about which boundary you mean and not to overclaim.

The fourth pitfall is ignoring the cost of late events. In both Lambda and Kappa, late-arriving events are real and have to be handled in stream code: watermarks, allowed lateness, session windows. Candidates who treat "stream processing" as a black box that magically gives the same answer as batch get caught here. Be ready to name the watermark strategy you'd use for your domain — for example, "bounded out-of-orderness of 30 seconds for app analytics, side-output for anything later."

The fifth pitfall is picking Lambda for a greenfield project because it sounds safer. Running two pipelines is not safer; it is twice the ops surface and twice the deploys. On new projects in 2026, single-stream-into-lakehouse is the default — Lambda should require a written justification.

Related reading

• Spark Structured Streaming — DE interview
• Kafka Streams — DE interview
• Lakehouse: Iceberg vs Delta — DE interview
• ETL vs ELT — DE interview
• Apache Iceberg deep dive

If you want to drill DE architecture questions like this every day, NAILDD is launching with 1,500+ interview problems across exactly this pattern.

FAQ

Is Lambda architecture still used in production?

Yes, but mostly in legacy enterprises where a mature batch warehouse predates the stream layer by years. New systems built in 2024-2026 default to Kappa or lakehouse. If you're designing something new, Lambda should not be your first answer — and if you choose it, defend it with a specific reason (regulatory accuracy, organizational split, legacy batch investment).

What's the difference between Kappa and "just streaming"?

Kappa is a specific commitment: the log is the source of truth, and any historical recomputation happens by replaying the log. A team that says "we use streaming" but reaches for a batch backfill job whenever they need to reprocess is not running Kappa — they're running an ad-hoc streaming pipeline with manual batch escape hatches. The discipline of "everything goes through the log" is what makes Kappa Kappa.

Where does Spark Structured Streaming fit?

Structured Streaming is a stream-processing API that can play either role. Inside Lambda, it's a common choice for the speed layer because the same DataFrame code can be reused in the batch layer. Inside Kappa, it's the single processor. The framework itself is neutral — what matters is how you wire it into the rest of the architecture, and specifically whether you have a durable replayable source upstream.

Why do most modern designs collapse into lakehouse?

Because the table format (Iceberg, Delta, Hudi) gives you the one property both Lambda and Kappa were trying to achieve — a single, replayable, consistent record of truth — without the operational baggage of either. You don't need to maintain two pipelines (Lambda's pain) and you don't need to pay for years of Kafka retention (Kappa's pain). The bronze table is the log; everything downstream is just transforms over consistent snapshots.

How do I answer "design a real-time fraud system" using this?

Start by clarifying SLA: "decisions under 200ms" rules out batch as the primary path. Sketch a Kappa-style flow — events into Kafka, Flink applying the fraud model and writing decisions to a low-latency store, with a parallel append into an Iceberg bronze table for retraining and audit. Model training itself is a batch job over the bronze table, which is fine because training tolerates hours of latency. That answer shows you know which parts need stream and which don't.

Do interviewers expect me to name specific tools?

Yes, but in pairs and with reasoning. Pick the two or three you'd actually deploy and justify them — "Kafka for the log because of tiered storage; Flink for stateful processing because of event-time semantics; Iceberg for the table layer because of schema evolution." Listing ten technologies without picking is a junior signal.