Python · Parquet · DuckDB · dbt · Airflow · data engineering
nordlake
A small lakehouse on Norwegian open data. Python pulls sources into an append-only bronze layer of Parquet files, dbt turns them into silver and gold tables on DuckDB, and Airflow runs the lot nightly. Everything runs on a laptop and in CI, with no services. It is young and says so.
View on GitHub ↗
From feed to table
Every source goes down the same lines. Bronze never changes shape, so a new source is an adapter and a staging model, and nothing else.
- Sourceevery job ad on arbeidsplassen.nav.no, from Nav's public feedand Claude Code usage, metadata only
- Bronzeappend-only Parquet, one file per batch, partitioned by ingest datethe same five columns for every source
- Cursorone JSON cursor per source, stored after the batch is on diska crash re-reads, it never skips
- Silverdbt on DuckDB extracts the columns and keeps the latest row per keydedupe lives here, not in bronze
- Goldads per day and sector, skills asked for per week, cost per projecttested in dbt
- NightlyAirflow runs both ingests in parallel, then dbt buildthe DAG calls the same CLI you do
The problem
Norwegian public-sector data jobs ask for the same handful of tools: dbt, Airflow, Iceberg, Trino and Spark. I wanted the lakehouse pattern they describe working end to end on real Norwegian data, small enough to run on a laptop and in CI, and honest about which parts are still a plan.
Its first question answers itself: which technologies do job ads in Norway ask for, week by week.
What I built
- An ingest framework: a source is a class with a name and a
fetch(cursor) that yields batches of records with a stable key, a timestamp and the raw payload.
- A Nav adapter that reads the public vacancy feed and only fetches what changed after the first run.
- A Claude Code adapter that reads token usage from local session logs, and never stores prompts, output or tool results.
- dbt models for silver and gold, with model prices and a skills list as seeds, and a
make demo that runs end to end on recorded fixtures.
Decisions worth explaining
- DuckDB first, Trino later
- Bronze is plain Parquet and dbt reads it straight from the files, so the demo runs in CI in under a minute with no services. The layout is chosen so that moving to Iceberg on object storage is a storage swap, not a redesign.
- Append-only bronze
- Raw records are never rewritten. Deduplication and typing happen in silver, so a bad model can be fixed and rebuilt from the same files.
- Metadata only
- The transcript source keeps ids, timestamps, models and token counts. Prompt text never reaches the lake, and project names can be hashed too.
- Respect the source
- Finn.no's terms prohibit crawling, so its ads are out of scope. Named contact persons in Nav ads are dropped before anything is written.
- One code path
- The Airflow DAG only calls the CLI that runs by hand, so there is one thing to debug when a night fails.
Stack
What is next
The company registry and SSB as sources, Iceberg tables on MinIO with Trino as the shared query engine, a data catalogue generated from dbt's docs, and dashboards on top. Job data comes from Nav's public vacancy feed, used under its terms.