# Plugins

# Installation

# Locally with the CLI

The general pattern for installing a plugin on a Meltano instance is to use the meltano add command. Examples can be found in each plugin.

# On a CI Pipeline

A docker image should be build containing all the latest curated version of the taps/targets, each isolated into its own virtualenv.

This way we do not run into docker-in-docker problems (buffering, permissions, security).

Meltano should provide a wrapper script to manage the execution of the selected components:

meltano extract tap-zendesk --to target-postgres

# Extractors

Meltano Extractors are commonly prefixed with tap.

# Carbon Intensity

tap-carbon-intensity pulls data from the Official Carbon Intensity API for Great Britain, which was developed by the National Grid. For more information, check out http://carbonintensity.org.uk/.

# Info

• Data Source: Carbon Intensity API
• Repository: https://gitlab.com/meltano/tap-carbon-intensity

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-carbon-intensity

If you are successful, you should see Added and installed extractors 'tap-carbon-intensity' on your window. It

# CSV

tap-csv is a CSV reader that is optimized for tasks where the file structure is highly predictable.

# Info

• Data Source: Traditionally-delimited CSV files (commas separated columns, newlines indicate new rows, double quoted values) as defined by the defaults of the python csv library.
• Repository: https://gitlab.com/meltano/tap-csv

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-csv

If you are successful, you should see Added and installed extractors 'tap-csv' on your window. It

# Configuration

.env

export TAP_CSV_FILES_DEFINITION="files_def.json"

Where files_def.json is a json file with all the CSV files to be loaded:

files_def.json

[
  { "entity" : "leads",
    "file" : "/path/to/leads.csv",
    "keys" : ["Id"]
  },
  { "entity" : "opportunities",
    "file" : "/path/to/opportunities.csv",
    "keys" : ["Id"]
  }
]

Description of available options:

• entity: The entity name to be passed to singer (i.e. the table name).
• file: Local path to the file to be ingested. Note that this may be a directory, in which case all files in that directory and any of its subdirectories will be recursively processed.
• keys: The names of the columns that constitute the unique keys for that entity.

# Fastly

tap-fastly pulls raw data from Fastly and produces JSON-formatted data per the Singer spec.

# Info

• Data Source: Fastly
• Repository: https://gitlab.com/meltano/tap-fastly

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-fastly

If you are successful, you should see Added and installed extractors 'tap-fastly' on your window. It

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export FASTLY_API_TOKEN="yourFastlyApiToken"
# The date uses ISO-8601 and supports time if desired
export FASTLY_START_DATE="YYYY-MM-DD"

# GitLab

tap-gitlab pulls raw data from GitLab's REST API and extracts the following resources from GitLab. It then outputs the schema for each resource and incrementally pulls data based on the input state

# Info

• Data Source: GitLab's REST API
• Repository: https://gitlab.com/meltano/tap-gitlab

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-gitlab

If you are successful, you should see Added and installed extractors 'tap-gitlab' on your window.

3. Get your GitLab access token

   • Login to your GitLab account
   • Navigate to your profile page
   • Create an access token

4. Create a JSON file called config.json containing:

   • Access token you just created
   • API URL for your GitLab account. If you are using the public gitlab.com this will be https://gitlab.com/api/v4
   • Groups to track (space separated)
   • Projects to track (space separated)

   Notes:

   • either groups or projects need to be provided
   • filling in 'groups' but leaving 'projects' empty will sync all group projects.
   • filling in 'projects' but leaving 'groups' empty will sync selected projects.
   • filling in 'groups' and 'projects' will sync selected projects of those groups.

   {
     "api_url": "https://gitlab.com/api/v4",
     "private_token": "your-access-token",
     "groups": "myorg mygroup", 
     "projects": "myorg/repo-a myorg/repo-b",
     "start_date": "2018-01-01T00:00:00Z"
   }
   

5. [Optional] Create the initial state file

   You can provide JSON file that contains a date for the API endpoints to force the application to only fetch data newer than those dates. If you omit the file it will fetch all GitLab data

   {
     "branches": "2017-01-17T00:00:00Z",
     "commits": "2017-01-17T00:00:00Z",
     "issues": "2017-01-17T00:00:00Z",
     "projects": "2017-01-17T00:00:00Z",
     "project_milestones": "2017-01-17T00:00:00Z", 
     "users": "2017-01-17T00:00:00Z",
     "group_milestones": "2017-01-17T00:00:00Z"
   }
   

   Note:

   • currently, groups don't have a date field which can be tracked

# Configuration

# Marketo

tap-marketo pulls raw data from Marketo's REST API and extracts activity types, activites, and leads from Marketo.

# Info

• Data Source: Marketo's REST API
• Repository: https://gitlab.com/meltano/tap-marketo

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-marketo

If you are successful, you should see Added and installed extractors 'tap-marketo' on your window. It

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export MARKETO_CLIENT_ID="yourClientId"
export MARKETO_CLIENT_SECRET="yourClientSecret"
export MARKETO_ENDPOINT="yourEndpointUrl"
export MARKETO_IDENTITY="yourIdentity"
export MARKETO_START_TIME="yourStartTime"

# MongoDB

tap-mongodb pulls raw data from a MongoDB source.

# Info

• Data Source: MongoDB source
• Repository: https://github.com/singer-io/tap-mongodb

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-mongodb

If you are successful, you should see Added and installed extractors 'tap-mongodb' on your window.

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

# MongoDB databse host URI
export MONGODB_HOST=""
# MongoDB database port
export MONGODB_PORT=""
# MongoDB database username
export MONGODB_USERNAME=""
# MongoDB database password
export MONGODB_PASSWORD=""
# MongoDB database name
export MONGODB_DBNAME=""

# Salesforce

tap-salesforce is an extractor that pulls data from a Salesforce database and produced JSON-formatted data following the Singer spec.

# Info

• Data Source: Salesforce
• Repository: https://gitlab.com/meltano/tap-salesforce

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-salesforce

If you are successful, you should see Added and installed extractors 'tap-salesforce' on your window. It

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export SFDC_CLIENT_ID="yourSalesforceClientId"
export SFDC_PASSWORD="yourSalesforcePassword"
export SFDC_SECURITY_TOKEN="yourSalesforceSecurityToken"
export SFDC_START_DATE="yourSalesforceStartDate"
export SFDC_USERNAME="yourSalesforceUsername"

# Stripe

tap-stripe is an extractor that pulls data from Stripe's API and produces JSON-formatted data following the Singer spec.

# Info

• Data Source: Stripe's API
• Repository: https://github.com/meltano/tap-stripe

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-carbon-intensity

If you are successful, you should see Added and installed extractors 'tap-carbon-intensity' on your window. It

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export STRIPE_API_KEY="yourStripeApiKey"
# The date uses ISO-8601 and supports time if desired
export STRIPE_START_DATE="YYYY-MM-DD"

# Zendesk

tap-zendesk is an extractor that pulls data from a Zendesk REST API and produces JSON-formatted data following the Singer spec.

# Info

• Data Source: Zendesk REST API
• Repository: https://github.com/meltano/tap-zendesk

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-zendesk

If you are successful, you should see Added and installed extractors 'tap-zendesk' on your window. It

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export ZENDESK_EMAIL="yourZendeskEmail"
export ZENDESK_API_TOKEN="yourZendeskApiToken"
export ZENDESK_SUBDOMAIN="yourZendeskSubdomain"
# The date uses ISO-8601 and supports time if desired
export ZENDESK_START_DATE="yourZendeskStartDate"

# Zuora

tap-zuora is an extractor that pulls data from a Zuora REST API and produces JSON-formatted data following the Singer spec.

# Info

• Data Source: Zuora REST API
• Repository: https://github.com/singer-io/tap-zuora

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-zendesk

If you are successful, you should see Added and installed extractors 'tap-zendesk' on your window. It

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export ZUORA_USERNAME=""
export ZUORA_PASSWORD=""
export ZUORA_API_TOKEN=""   # preferred to ZUORA_PASSWORD
export ZUORA_API_TYPE=""    # specifically 'REST' or 'AQuA'
export ZUORA_PARTNER_ID=""  # optional, only for the 'AQuA` API type
export ZUORA_START_DATE=""
export ZUORA_SANDBOX=""     # specifically 'true' or 'false'

# Loaders

A loader is a component for the bulk import of data. Currently, Meltano supports Singer.io targets as loaders.

# CSV

target-csv is a loader that works with other extractors in order to move data into CSV-formatted files.

# Info

• Data Warehouse: CSV Files
• Repository: https://gitlab.com/meltano/target-csv

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add loader target-csv

If you are successful, you should see Added and installed loaders 'target-csv' on your window.

# Configuration

If you want to customize your delimited or quote character, open meltano.yml for your desired project and update the configuration there.

  - config:
      delimiter": "\t"
      quotechar": ''''
    name: target-csv
    pip_url: git+https://gitlab.com/meltano/target-csv.git

# Snowflake

target-snowflake is a loader that works with other extractors in order to move data into a Snowflake database.

# Info

• Data Warehouse: Snowflake
• Repository: https://gitlab.com/meltano/target-snowflake

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export SF_ACCOUNT=""
export SF_USER=""
export SF_PASSWORD=""
export SF_ROLE=""       # in UPPERCASE
export SF_DATABASE=""   # in UPPERCASE
export SF_WAREHOUSE=""  # in UPPERCASE

# Postgres

target-postgres is a loader that works with other extractors in order to move data into a Postgres database.

# Info

• Data Warehouse: Postgres
• Repository: https://github.com/meltano/target-postgres

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export PG_ADDRESS=""
export PG_USERNAME=""
export PG_PORT=""
export PG_PASSWORD=""
export PG_DATABASE=""
export PG_SCHEMA=""

# Sqlite

target-sqlite is a loader that works with other extractors in order to move data into a SQLite database.

# Info

• Data Warehouse: SQLite
• Repository: https://gitlab.com/meltano/target-sqlite

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export SQLITE_DATABASE=""