Scenarios
Browse, search, and filter benchmark evaluation scenarios.
Discover evaluation scenarios by domain, competency, feature, tier, and starting-state model. Each scenario maps to a realistic data engineering workflow.
Fix an analytics HTTP API that fails or returns inconsistent results under concurrent requests. The API uses a single shared ClickHouse connection that is not thread-safe.
Tier 2 · Starts broken · 1 task · clickhouse ·
Diagnose and fix a misconfigured Postgres setup: wrong connection string, partial init script, and hardcoded credentials in a Python script.
Tier 1 · Starts broken · 1 task · postgres ·
Build a change-data-capture pipeline from Postgres to Redpanda: capture table changes and stream them to a Kafka topic for downstream consumers.
Tier 2 · Starts clean · 1 task · postgres, redpanda ·
Fix a broken historical backfill pipeline. The target table has incorrect partition or order keys that cause slow inserts and failed backfills for historical date ranges.
Tier 2 · Starts broken · 1 task · clickhouse ·
Add a new column to a live ClickHouse table with existing data. The migration must not block reads/writes and must backfill the new column for existing rows.
Tier 2 · Starts broken · 1 task · clickhouse ·
Fix a broken ClickHouse materialized view that fails to refresh. The MV definition has incorrect dependencies and does not match the source table schema.
Tier 2 · Starts broken · 1 task · clickhouse ·
A ClickHouse table has a suboptimal ORDER BY that causes full scans for common query patterns. Redesign the ORDER BY key to match filter columns (region, event_ts) and improve query latency.
Tier 1 · Starts broken · 1 task · clickhouse ·
Design a ClickHouse table with TTL so that raw events older than 90 days are automatically dropped. Create the table and verify the TTL expression is correct.
Tier 1 · Starts clean · 1 task · clickhouse ·
Fix inconsistent metric definitions across ClickHouse and Postgres. The same metric (e.g. daily_active_users) is computed differently in each store, causing reporting mismatches.
Tier 2 · Starts broken · 1 task · postgres, clickhouse ·
Diagnose and fix a Kafka consumer that cannot keep up with producer throughput. Consumer lag is growing; optimize batch size, parallelism, or fetch config so the sink table catches up.
Tier 1 · Starts broken · 1 task · redpanda, clickhouse ·
Fix broken reconciliation between Postgres, Redpanda, and ClickHouse. Build a reconciliation job that detects and reports drift (missing rows, count mismatches).
Tier 3 · Starts broken · 1 task · postgres, redpanda, clickhouse ·
Load five messy CSV files into clean ClickHouse tables. Handle inconsistent dates, mixed null representations, duplicate headers, and trailing delimiters.
Tier 1 · Starts clean · 1 task · clickhouse ·
Create a semantic layer with derived and composite metrics. Build materialized views or a metrics table that computes conversion_rate, avg_order_value, and daily_active_users from raw events.
Tier 1 · Starts clean · 1 task · clickhouse ·
Configure a dead-letter queue (DLQ) for a Kafka consumer that fails on malformed messages. The consumer currently drops failed messages; add a DLQ topic and wire the consumer to route failures there.
Tier 1 · Starts broken · 1 task · redpanda, clickhouse ·
Fix a broken event-sourcing setup: event log in Redpanda is out of sync with materialized view in Postgres. Replay events to restore consistency.
Tier 3 · Starts broken · 1 task · postgres, redpanda ·
Diagnose and fix a broken end-to-end pipeline: Postgres CDC to Redpanda to ClickHouse. Multiple failure points including misconfigured connector, wrong topic, and missing ClickHouse table.
Tier 3 · Starts broken · 1 task · postgres, redpanda, clickhouse ·
Build a pipeline from Postgres to ClickHouse that produces identical results whether run once or three times. Handle updates to existing rows by primary key.
Tier 1 · Starts clean · 1 task · postgres, clickhouse ·
Build a complete pipeline: ingest raw product-interaction events from Postgres, transform and load into ClickHouse, and expose a lightweight HTTP API that serves pre-aggregated insights (top products, conversion funnel, hourly trends).
Tier 2 · Starts clean · 3 tasks · postgres, clickhouse ·
Fix a failed backfill of a new metric definition. The backfill job partially ran, left orphaned data, and the metric table has gaps or duplicates. Recover and complete the backfill idempotently.
Tier 2 · Starts broken · 1 task · postgres, clickhouse ·
Build a query builder API that supports filtering and grouping by multiple dimensions (date, region, product). Choose appropriate storage (ClickHouse vs Postgres) per data type and expose a unified query interface.
Tier 2 · Starts clean · 1 task · postgres, clickhouse ·
Fix a broken migration from Postgres (OLTP) to ClickHouse (OLAP). The sync job fails, schema is mismatched, and incremental load is not working. Restore the migration pipeline.
Tier 3 · Starts broken · 1 task · postgres, clickhouse ·
Build a paginated analytics API that serves large result sets from ClickHouse with cursor-based or offset pagination. Support page size and consistent ordering.
Tier 2 · Starts clean · 1 task · clickhouse ·
Add indexes to a Postgres table with 500k rows. Queries filtering by user_id and created_at are slow; create appropriate indexes to meet latency targets.
Tier 1 · Starts broken · 1 task · postgres ·
Fix a startup race: the app connects to Postgres immediately and fails when the database is not yet ready. Add retry/backoff logic so the app waits for Postgres to be ready.
Tier 1 · Starts broken · 1 task · postgres ·
Fix a large unpartitioned Postgres table that is slow for date-range queries. Partition by date and migrate existing data.
Tier 2 · Starts broken · 1 task · postgres ·
Add data quality checks to a working pipeline. A chaos script injects nulls, duplicates, schema drift, and stale timestamps. Agent must detect all four problems without false positives.
Tier 1 · Starts broken · 1 task · postgres, clickhouse ·
Fix a broken realtime metrics pipeline: Redpanda streams metrics but ClickHouse materialized view is stale or failing. Restore sub-second aggregation from stream to OLAP.
Tier 3 · Starts broken · 1 task · redpanda, clickhouse ·
Add a column to a synthetic dimension table, create the corresponding ClickHouse target, build a cross-database view, and verify old queries still work.
Tier 1 · Starts clean · 1 task · postgres, clickhouse ·
An HTTP API serves analytical queries from ClickHouse but responses take 2+ seconds. Optimize the queries, add materialized views, or pre-aggregate so /api/metrics and /api/breakdown return in under 200ms.
Tier 1 · Starts broken · 1 task · clickhouse ·
Rewrite five slow analytical queries against a 10M-row ClickHouse table to run under latency thresholds without changing result sets.
Tier 1 · Starts broken · 1 task · clickhouse ·
Fix a broken Redpanda topic schema: add a new optional field to the event schema while maintaining backward compatibility for existing consumers.
Tier 2 · Starts broken · 1 task · redpanda ·
Consume events from a Redpanda topic and load them into ClickHouse for analytical queries. Build a streaming-to-OLAP pipeline with appropriate table design.
Tier 2 · Starts clean · 1 task · redpanda, clickhouse ·
Redesign a naive ClickHouse table with 5M rows to serve three representative query patterns. Set partition keys, order keys, and compression for target latencies.
Tier 1 · Starts clean · 1 task · clickhouse ·
Fix broken time-grain rollup tables (hourly, daily, monthly) that have incorrect aggregations or missing grains. The rollups should support dashboard queries at different time resolutions.
Tier 2 · Starts broken · 1 task · clickhouse ·
Fix a misconfigured Kafka topic that causes consumer rebalance storms. The topic has a single partition with high throughput; consumers cannot parallelize and lag grows. Correct the partition count and replication factor.
Tier 1 · Starts broken · 1 task · redpanda, clickhouse ·
Build a three-layer transformation from raw JSON events in Postgres to a sessionized, aggregated daily mart in ClickHouse.
Tier 1 · Starts clean · 1 task · postgres, clickhouse ·
Contribute New Scenarios
Scaffold with dec-bench create, register with dec-bench registry add, then open a PR with dec-bench registry publish.