Deployment

Amazon Web Services (AWS)

Prerequisites

This guide assumes that you have a functioning Docker image where your Meltano project is already bundled with the Meltano installation. To track this issue, follow meltano#624.

In this section, we will be going over how you can deploy a Meltano Docker image to AWS.

Setting Up Elastic Container Service (ECS)

  1. Login to AWS Console
  2. Search for ECS and click on the link

Define container and task

  1. We will create a new "Container definition" by clicking on the Configure button in the "custom" card
  2. Fill out the form with the following data:
  • Container name: Meltano
  • Image: YOUR_DOCKER_IMAGE_URL
    • Examples:
      • docker.io/namespace/image-name:tag
      • registry.gitlab.com/namespace/project/image-name:tag
  • Memory Limits (MiB): Soft limit 1024
  • Port mappings: 5000 tcp
  1. Click Update button to finish setting up your container defintion
  2. Click Edit next to the "Task defintion" heading
  3. Update the form with the following:
  • Task definition name: meltano-run
  • Network mode: awsvpc
  • Task execution role: ecsTaskExecutionRole
  • Compatibilities: FARGATE
  • Task memory: 1GB (1024)
  • Task CPU: 0.25 vCPU (256)
  1. Click Next to move to the next step

Review service properties

  1. Verify that the properties are as follows:
  • Service name: meltano-service
  • Number of desired tasks: 1
  • Security group: Automatically create new
  • Load balancer type: None
  1. Click Next to move on to the next step

Configure your cluster

The main configuration here is the Cluster name. We provide a suggestion below, but feel free to name it as you wish.

  • Cluster name: meltano-cluster
  • VPC ID: Automatically create new
  • Subnets: Automatically create new

Review cluster configuration

After you click Next, you will have the opportunity to review all of the properties that you set. Once you confirm that the settings are correct, click Create to setup your ECS.

You should now see a page where Amazon prepares the services we configured. There will be spinning icons on the right of each service that will live update as it finished. Once you see everything has setup properly, you're cluster has been successfully deployed!

Final steps

  1. Open the page with your cluster
  2. Click on the "Tasks" tab
  3. You should see a task that has a status of RUNNING for "Last Status"
  4. Click on the Task ID link (e.g., 0b35dea-3ca..)
  5. Under "Network", you can find the URL for your instance under "Public IP" (e.g., 18.18.134.18)
  6. Open a new tab in your browser and visit this new URL with port 5000 (e.g., 18.18.134.18:5000)

TIP

The IP address can be mapped to a domain using Route53. We will be writing up a guide on how to do this. You can follow along at meltano#625.

Configure network access

TIP

This section is only necessary if you do not have a Security Group that allows for port 5000 inbound.

Once you complete the cluster setup, you should be brought to the detail page for the service. You should be default on a tab called "Details" with a "Network Access" section.

  1. Locate the "Details" tab
  2. Under "Network Access", click on the link next to "Security Groups" (e.g., sg-f0dj093dkjf10)
  3. This should open a new tab with your security group
  4. Click on the "Inbound Rules" tab on the bottom of the page
  5. Click "Edit Rules"
  6. Delete any existing rules
  7. Click "Add Rule" with the following properties:
  • Type: Custom TCP Rule
  • Protocol: TCP
  • Port Range: 5000
  • Source: Custom 0.0.0.0/0
  1. Click Save rules

Coming Next

  • Creating a data warehouse for Meltano
  • Running ELT in the Cloud
  • Growing and Scaling Meltano
  • Running Meltano with AWS EC2
  • Using AWS Lambda with Meltano
Last Updated: 4/30/2019, 9:27:40 PM