Tech Stack

Kubernetes

Envoy Gateway

All services run on a Haven-compliant Kubernetes cluster, deployed with Helm and helmfile. Envoy Gateway and cert-manager provide HTTPS and URL routing; secrets come from a vault via External Secrets.

Apache Airflow

Orchestrates the full pipeline: upload processing, LLM extraction, geocoding, ClickHouse loading, dbt builds, and GIS sync. Includes scheduled polling every 5 minutes.

FastAPI Upload

Pixtral LLM

Nominatim

Multi-channel data collection: CSV ingestion, drag-and-drop letter upload with AI-powered address extraction (Pixtral vision LLM), and self-hosted geocoding via Nominatim.

ClickHouse Bronze

RustFS

Raw data lands in the ClickHouse bronze layer. Uploaded letter files are stored in RustFS, a self-hosted S3-compatible object store.

ClickHouse Silver / Gold

Silver layer provides cleaned and combined staging tables. Gold layer delivers analytics-ready dimensions, facts, and marts with 13+ models for coverage analysis, proximity scoring, and more.

dbt

27+ SQL models transforming raw bronze data through silver staging to gold analytics. Includes 50+ automated data quality tests, incremental models, and custom generic tests.

dbt Docs

Auto-generated data catalog with full lineage graph from bronze to gold. Shows table schemas, column descriptions, test coverage, and model dependencies.

Grafana

ClickHouse Play

CH-UI

Six pre-built Grafana dashboards for API traffic, upload tracking, LLM extraction stats, geocoding health, GIS monitoring, and charging telemetry analytics. ClickHouse Play and CH-UI for ad-hoc SQL queries.

Pixtral 12B

Scaleway AI

Mistral's Pixtral 12B vision model hosted on Scaleway extracts street addresses from scanned handwritten letters as structured JSON with confidence scoring.

Tyk Gateway

FastAPI

Redis

REST API with Swagger docs, authenticated via Tyk gateway with API key management and rate limiting (100 req/60s). Serves analytics, GIS endpoints, and letter uploads.

Prometheus

Grafana

Prometheus scrapes FastAPI metrics every 15 seconds. Grafana provides the visualization layer with six pre-built dashboards monitoring the entire platform.

GeoServer

GeoNode

PostGIS

Nominatim

GeoServer publishes WMS/WFS layers for QGIS and ArcGIS. GeoNode provides a data catalog portal. PostGIS stores spatial data synced from ClickHouse. Nominatim handles self-hosted geocoding.

Upload Portal

CH-UI

Drag-and-drop letter upload portal, full-featured ClickHouse web interface (CH-UI).