Meltano Speedrun: from 0 to ELT in 90 seconds

Open source data integration has never been easier, or faster: with Meltano, extracting data from GitLab and loading it into PostgreSQL (or Snowflake, BigQuery, Redshift…) takes just 90 seconds from initializing a new Meltano project to viewing the loaded data:

Resulting Meltano project repository: https://gitlab.com/meltano/speedrun

Tools

More sources: https://meltano.com/plugins/extractors/

More destinations: https://meltano.com/plugins/loaders/

Prerequisites

  1. Install Meltano: https://meltano.com/docs/getting-started.html#install-meltano
  2. Set up PostgreSQL locally and create a database named `speedrun`

Commands

meltano init speedrun
cd speedrun

meltano add extractor tap-gitlab
meltano config tap-gitlab set projects meltano/meltano
meltano config tap-gitlab set start_date 2021-04-01T00:00:00Z
meltano select tap-gitlab tags

meltano add loader target-postgres
meltano config target-postgres set postgres_username [username]
meltano config target-postgres set postgres_database speedrun

meltano elt tap-gitlab target-postgres

psql -d speedrun
SELECT * FROM tap_gitlab.tags ORDER BY _sdc_received_at;