Skip to content
← All projects
Data Engineering2026

spotify-elt-pipeline

An end-to-end ELT pipeline that turns my own Spotify listening history into an analytics-ready warehouse. A Python job pulls play data from the Spotify Web API and lands it raw in PostgreSQL; dbt then models it into a star schema across staging, intermediate, and mart layers; and Metabase sits on top for exploration. The whole stack runs on Docker Compose and is covered by 39 dbt tests plus a pytest suite, so every run is reproducible and fully replayable.

  • Python
  • PostgreSQL
  • dbt
  • Docker Compose
  • Metabase
  • pytest

Architecture

  1. Spotify Web API
    Source
  2. Raw JSON
    Extract · Python
  3. PostgreSQL — raw
    Load
  4. PostgreSQL — analytics
    Transform · dbt
  5. Metabase
    Serve · BI

In the dashboard

Nested Metabase donut breaking down 27 plays by artist (outer ring) grouped by genre (inner ring): Creedence Clearwater Revival 22%, The Doors 15%, Pink Floyd 11%, then Foo Fighters, Nirvana, Dire Straits, Ja Rule, Jimi Hendrix, Larry June, and Pearl Jam at ~7% each.
Plays by artist, grouped by genre — served from the dbt marts in Metabase
Metabase SQL editor running the genre-by-plays query — SELECT genre_name, count(*) joining analytics.fact_plays to dim_genre through the artist–genre bridge — with the 144-play result rendered as a donut: rock 28%, classic rock 17%, alternative rock 10%, hip hop 10%, rap 9%.
The genre query in Metabase, running against the star schema

Highlights

  • Extracts personal Spotify listening data through the Spotify Web API in Python, lands it raw in PostgreSQL, and transforms it in-warehouse with dbt.
  • Models an analytics-ready star schema (fact_plays with dim_track, dim_artist, dim_genre, dim_date, and an artist–genre bridge) using layered dbt models across staging, intermediate, and mart layers.
  • Enforces data quality with 39 dbt tests and a pytest suite covering API client units and idempotent loader integration, keeping runs fully replayable.