Open source data integration has never been easier, or faster: with Meltano, extracting data from GitLab and loading it into PostgreSQL (or Snowflake, BigQuery, Redshift…) takes just 90 seconds from initializing a new Meltano project to viewing the loaded data:
Resulting Meltano project repository: https://gitlab.com/meltano/speedrun
Tools
- Meltano: ELT for the DataOps era – https://meltano.com/
tap-gitlab
: GitLab extractor – https://meltano.com/plugins/extractors/gitlab.htmltarget-postgres
: PostgreSQL loader – https://meltano.com/plugins/loaders/postgres.html
More sources: https://meltano.com/plugins/extractors/
More destinations: https://meltano.com/plugins/loaders/
Prerequisites
- Install Meltano: https://meltano.com/docs/getting-started.html#install-meltano
- Set up PostgreSQL locally and create a database named `speedrun`
Commands
meltano init speedrun
cd speedrun
meltano add extractor tap-gitlab
meltano config tap-gitlab set projects meltano/meltano
meltano config tap-gitlab set start_date 2021-04-01T00:00:00Z
meltano select tap-gitlab tags
meltano add loader target-postgres
meltano config target-postgres set postgres_username [username]
meltano config target-postgres set postgres_database speedrun
meltano elt tap-gitlab target-postgres
psql -d speedrun
SELECT * FROM tap_gitlab.tags ORDER BY _sdc_received_at;