Data Engineering System
Pipelight: Interactive Event-Driven Analytics Platform
Events flow from a click to modeled metrics within ~30 seconds, with 54 dbt tests on every cycle, malformed data quarantined instead of dropped, and five on-demand failure scenarios that show how the pipeline behaves under stress.
- Storefront
- FastAPI
- Redpanda
- Consumer
- dbt
- Metrics API
Problem
Most pipeline demos hide the hard parts. Bad schemas, duplicate events, late arrivals, and traffic spikes are what break real event pipelines, and it is rarely visible how data moves from a user action to a trusted business metric.
Solution
Built an end-to-end streaming analytics platform where a React storefront emits ecommerce events into a FastAPI ingestion service, Redpanda streams them to an async validating consumer, dbt models them into marts, and live dashboards show the result.
Results
Events flow from a click to modeled metrics within ~30 seconds, with 54 dbt tests on every cycle, malformed data quarantined instead of dropped, and five on-demand failure scenarios that show how the pipeline behaves under stress.
Solution
Every storefront action becomes a typed event envelope published through an Event API into Redpanda. An async consumer validates each event against a shared schema, writing valid records to raw.raw_events and routing malformed ones to raw.invalid_events with an error reason. A dbt runner transforms raw data into staging models and marts (fct_orders, agg_events_daily, agg_revenue_daily) on a continuous loop, with an Airflow DAG orchestrating scheduled runs. A dedicated Metrics API serves KPIs, freshness, and dbt job status to a pipeline monitoring UI and Grafana dashboards.
Architecture
React + Vite storefront, FastAPI Event API, Redpanda (Kafka-compatible) broker, async Python consumer with schema validation and quarantine, PostgreSQL raw/staging/mart schemas, dbt-core transformations and tests, Airflow orchestration, FastAPI Metrics API, and Grafana dashboards, all run with Docker Compose.
Scalability Considerations
Scale by partitioning Redpanda topics by event type or user, running multiple consumers in a consumer group, switching large marts to incremental dbt models, and moving raw storage to a columnar warehouse as volume grows.
Engineering Decisions
- Used Redpanda as a Kafka-compatible broker to decouple ingestion from processing without the operational weight of a full Kafka cluster.
- Quarantined invalid events in a dedicated table with error messages instead of dropping them, so bad data stays observable and replayable.
- Shared one event envelope model between the API and consumer to keep the producer and validator contract in sync.
- Served metrics through a separate read API so dashboards never query raw tables or compete with ingestion writes.
- Built scenario generators (traffic spike, bad schema, duplicates, late arrivals, payment failure surge) to exercise failure modes on demand.
Tradeoffs Considered
- A 30-second dbt loop keeps metrics fresh, but it is micro-batch rather than true streaming and reruns models even when little has changed.
- Running both a continuous dbt runner and Airflow adds overlap, but it lets the platform degrade gracefully to the lightweight runner on constrained hosts.
- Deduplicating in dbt rather than at ingestion keeps the consumer simple, but raw tables temporarily hold duplicates until models run.
Results
- Delivered a working, observable pipeline from user action to business metric with live freshness and health status.
- Ran 54 automated dbt tests on every transformation cycle, surfaced in the UI via the Metrics API.
- Made failure modes demonstrable: bad schemas, duplicates, late data, and spikes can each be triggered and observed end to end.
What I Would Do In Production
Add a schema registry with versioned contracts, dead-letter replay tooling, consumer lag and freshness alerting, incremental dbt models, CI for dbt and service tests, secrets management, and authentication on the ingestion and metrics APIs.