Amazon Web Services (AWS)
This guide assumes that you have a functioning Docker image where your Meltano project is already bundled with the Meltano installation. To track this issue, follow meltano#624.
In this section, we will be going over how you can deploy a Meltano Docker image to AWS.
Setting Up Elastic Container Service (ECS)
- Login to AWS Console
- Search for ECS and click on the link
Define container and task
- We will create a new "Container definition" by clicking on the
Configure button in the "custom" card
- Fill out the form with the following data:
- Container name: Meltano
- Image: YOUR_DOCKER_IMAGE_URL
- Memory Limits (MiB): Soft limit 1024
- Port mappings: 5000 tcp
Update button to finish setting up your container defintion
Edit next to the "Task defintion" heading
- Update the form with the following:
- Task definition name: meltano-run
- Network mode: awsvpc
- Task execution role: ecsTaskExecutionRole
- Compatibilities: FARGATE
- Task memory: 1GB (1024)
- Task CPU: 0.25 vCPU (256)
Next to move to the next step
Review service properties
- Verify that the properties are as follows:
- Service name: meltano-service
- Number of desired tasks: 1
- Security group: Automatically create new
- Load balancer type: None
Next to move on to the next step
Configure your cluster
The main configuration here is the
Cluster name. We provide a suggestion below, but feel free to name it as you wish.
- Cluster name: meltano-cluster
- VPC ID: Automatically create new
- Subnets: Automatically create new
Review cluster configuration
After you click
Next, you will have the opportunity to review all of the properties that you set. Once you confirm that the settings are correct, click
Create to setup your ECS.
You should now see a page where Amazon prepares the services we configured. There will be spinning icons on the right of each service that will live update as it finished. Once you see everything has setup properly, you're cluster has been successfully deployed!
- Open the page with your cluster
- Click on the "Tasks" tab
- You should see a task that has a status of
RUNNING for "Last Status"
- Click on the Task ID link (e.g., 0b35dea-3ca..)
- Under "Network", you can find the URL for your instance under "Public IP" (e.g., 22.214.171.124)
- Open a new tab in your browser and visit this new URL with port 5000 (e.g., 126.96.36.199:5000)
The IP address can be mapped to a domain using Route53. We will be writing up a guide on how to do this. You can follow along at meltano#625.
Configure network access
This section is only necessary if you do not have a Security Group that allows for port 5000 inbound.
Once you complete the cluster setup, you should be brought to the detail page for the service. You should be default on a tab called "Details" with a "Network Access" section.
- Locate the "Details" tab
- Under "Network Access", click on the link next to "Security Groups" (e.g., sg-f0dj093dkjf10)
- This should open a new tab with your security group
- Click on the "Inbound Rules" tab on the bottom of the page
- Click "Edit Rules"
- Delete any existing rules
- Click "Add Rule" with the following properties:
- Type: Custom TCP Rule
- Protocol: TCP
- Port Range: 5000
- Source: Custom 0.0.0.0/0
- Creating a data warehouse for Meltano
- Running ELT in the Cloud
- Growing and Scaling Meltano
- Running Meltano with AWS EC2
- Using AWS Lambda with Meltano
Last Updated: 4/30/2019, 9:27:40 PM