Question library

Design a pipeline to ingest clickstream data at ~50k events/sec into a lakehouse. Walk me through the architecture end to end.

Hard

• Stream ingestion (Kafka/Kinesis) with partitioning strategy
• Exactly-once or idempotent writes
• Schema evolution and registry
• Columnar storage (Parquet/Iceberg) with compaction

A SQL query joining a 2B-row fact table to dimension tables is timing out. How do you diagnose and fix it?

Medium

• Reads the query plan first (EXPLAIN)
• Partition pruning and clustering
• Broadcast vs shuffle join choice
• Statistics freshness and skew

How would you model slowly changing dimensions for a customer analytics dataset?

Medium

• Names SCD types and picks Type 2 deliberately
• Effective/expiry dates with surrogate keys
• Trade-offs vs snapshots or Type 1
• Query ergonomics for analysts

Your Spark job fails with out-of-memory errors on a skewed join. What do you do?

Hard

• Identifies skew from the UI/metrics
• Salting or skew-hint strategies
• Broadcast the small side when possible
• Adaptive Query Execution and repartitioning

How do you guarantee data quality in a pipeline feeding executive dashboards?

Medium

• Data contracts at ingestion
• Automated tests (dbt tests / Great Expectations)
• Freshness SLAs and alerting
• Quarantine pattern for bad records

Kafka consumer lag grows every day during peak hours. Diagnose and resolve.

Medium

• Checks partition count vs consumer parallelism
• Batch size and poll tuning
• Downstream sink as the real bottleneck
• Autoscaling consumers on lag

Design a CDC pipeline from Postgres to the warehouse with minimal latency.

Hard

• Logical replication / Debezium
• Ordering and deduplication by LSN
• Initial snapshot then streaming catch-up
• Handling schema changes

When would you choose a lakehouse (Iceberg/Delta) over a warehouse like Snowflake or BigQuery — and when not?

Easy

• Cost model and workload shape
• Governance and ACID needs
• Ecosystem maturity trade-offs
• Avoids dogma — argues from requirements

A bank's ops team wants 'AI that just handles exceptions.' You have a two-week on-site. How do you scope something shippable without boiling the ocean?

Hard

• Narrows to one painful, measurable workflow
• Defines success with a number the customer already tracks
• Time-boxes discovery vs build vs harden
• Names what is explicitly out of scope and why

You built a working prototype on a customer's data. Their security team now blocks SaaS APIs and requires all inference on-prem. What do you do in the next 72 hours?

Hard

• Maps the constraint (network, data residency, model weights)
• Offers a realistic on-prem or VPC architecture
• Protects the demo path with a temporary exception or air-gapped replay
• Does not promise a cloud redesign they cannot accept

Your champion at the customer just left. The new VP thinks the project is a science fair. How do you keep it alive — or kill it cleanly?

Medium

• Re-sells in the new stakeholder's language (risk, cost, time)
• Shows a live artifact, not a slide
• Finds a second owner in the org
• Is willing to recommend a pause with a written handoff

You are tempted to hard-code a customer-specific parser because it would close the deal this week. When is that the right call, and how do you keep it from becoming a forever fork?

Medium

• Separates reversible hacks from platform commitments
• Puts the one-off behind an interface / config
• Sets an expiry or migration ticket before shipping
• Explains the product vs services trade-off to the customer

Walk me through how you would debug a production failure on-site when you have logs, a skeptical customer looking over your shoulder, and no access to their identity provider.

Medium

• Reproduces with the smallest failing input
• Splits 'our code' vs 'their environment' with evidence
• Communicates status in plain language while debugging
• Leaves a durable fix and a runbook, not a hero patch

How do you decide whether a customer win should become a product feature versus staying a professional-services artifact?

Easy

• Looks for repeat demand across accounts
• Estimates support burden and configuration surface
• Checks whether the abstraction survives a second customer
• Pushes back when the feature is really a one-company workflow

A team wants to 'fine-tune GPT' on their 40k support tickets. When do you choose RAG, fine-tuning, long context, or a tool-using agent — and how do you decide?

Hard

• Separates knowledge freshness from style/format learning
• Calls out evals before any training spend
• Names failure modes of each option (stale RAG, overfitting, cost)
• Picks a hybrid when the job is mixed (retrieve + constrained generation)

Design an evaluation harness for a domain Q&A assistant that must cite sources. What do you measure offline vs online?

Hard

• Gold set with faithfulness / citation accuracy, not just BLEU
• Slice metrics (rare docs, adversarial questions, multi-hop)
• Online: thumbs, escalation rate, time-to-resolution
• Human review sampling and disagreement handling

Your RAG app is leaking prompt-injection from retrieved HTML ('ignore previous instructions and…'). How do you harden it without killing useful retrieval?

Hard

• Treats retrieved text as untrusted data, not instructions
• Instruction hierarchy / system prompt isolation
• Output constraints (schema, allowlisted tools, citation required)
• Monitoring for tool-call and policy-bypass attempts

You are at 10M queries/month. Sketch routing, caching, and model choice so quality holds while unit cost drops.

Medium

• Cheap classifier / small model for easy queries
• Semantic cache with a known staleness policy
• Escalation to a larger model on low confidence
• Token budgets, truncation, and batch where latency allows

The product needs 'memory' across a 45-minute session plus facts from last quarter's tickets. How do you manage context?

Medium

• Distinguishes working memory vs long-term store
• Summarization with loss awareness
• Retrieval keyed by user/org/thread, not a giant prompt
• Forgets / redacts on policy and user request

When is a multi-agent graph the wrong architecture compared to a single orchestrator with tools?

Easy

• Agents add latency, cost, and failure surfaces
• Prefers a single loop when the task is linear
• Uses multiple roles only when specialization or isolation is real
• Insists on traces and a single source of truth for state

You have a notebook that 'works on the test set.' Walk me through taking it to a production endpoint serving 200 QPS with a 150ms p95 budget.

Hard

• Profiles where time actually goes (model vs I/O vs features)
• Batching, quantization, or a smaller distilled model
• Async / degrade path when the budget is blown
• Rollout: shadow, canary, kill switch

Training-serving skew is suspected: offline AUC is 0.91, production looks random. How do you diagnose?

Hard

• Compares feature distributions and code paths, not just labels
• Checks clocks, IDs, leakage, and default values
• Logs a feature snapshot per request for replay
• Fixes the pipeline rather than retraining blindly

Design the human-in-the-loop and fallback for a support copilot that must not invent refund policy.

Medium

• Grounds answers in retrieved policy with citations
• Refuses / escalates when confidence or retrieval is weak
• Clear UX for 'draft vs send'
• Audit log of what the model said and which doc it used

What online and offline metrics would you use to decide whether to ship a new ranking or LLM prompt change?

Medium

• Offline eval set that matches production slices
• Online: primary business metric plus guardrails (latency, toxicity, cost)
• Sample size and novelty effects
• Holdback / interleaving rather than a one-day vibe check

How would you build a data flywheel so production traffic makes the system better without poisoning it?

Medium

• Labels from implicit signals plus sampled human review
• Filters bots, leaks, and out-of-policy content
• Retrain / index refresh cadence with regression tests
• Privacy and consent constraints on using user data

RAG vs fine-tune for an internal support copilot on a 2,000-page wiki that changes weekly. Pick one and defend it.

Easy

• Wiki churn favors retrieval over weights
• Fine-tune only for style/format if retrieval quality is already good
• Names how they would eval the choice
• Does not pretend fine-tuning updates facts daily

Design a gen-AI platform that five product teams can ship on without each one wiring a hosted model API, logging, and PII handling from scratch.

Hard

• Shared gateway: auth, routing, retries, timeouts
• Central tracing of prompts, tools, and cost per tenant
• Policy layer (PII, retention, allowed models) as config
• Self-serve path that does not become a ticket bottleneck

A product team wants to send full customer CRM records to a third-party model. How do you architect for this legally and technically?

Hard

• Data classification and minimization before the prompt
• Tenant isolation and encryption / VPC / zero-retention options
• Contractual + technical controls (DPAs, region, logging redaction)
• A documented 'no' path and an approved alternative (self-host, retrieval-only)

Your primary model provider has a 40-minute outage. What does failover look like so products keep working with known quality loss?

Medium

• Abstracted provider interface, not scattered SDKs
• Capability mapping (context length, tools, JSON mode)
• Degraded prompts / smaller models with user-visible honesty
• Load tests and a runbook, not a hope-based switch

How do you attribute cost and enforce quotas so one team cannot burn the company budget on a recursive agent loop?

Medium

• Per-tenant and per-app budgets with hard stops
• Token and tool-call accounting in the gateway
• Loop / max-step limits on agents
• Showback dashboards teams actually look at

What does a production-grade guardrail stack look like — input, tool use, and output — without turning every answer into a refusal?

Medium

• Layered: classifier + rules + constrained decoding / schema
• Different policy by surface (internal vs customer-facing)
• Allowlisted tools with argument validation
• Measures false-positive rate, not just blocks

When should a company not use generative AI for a workflow, even if a demo looks magical?

Easy

• Irreversible actions, thin eval, or required determinism
• Cost/latency that cannot meet the job
• Data you cannot legally or ethically send to a model
• Prefers classical software when the rules are already explicit