Deploy Long-Running ETL Pipelines to ECS with Fargate | by Ida Silfverskiöld | Mar, 2024

ETL stands for Extract, Remodel, and Load. An ETL pipeline is actually only a information transformation course of — extracting information from one place, doing one thing with it, after which loading it again to the identical or a distinct place.

If you’re working with pure language processing by way of APIs, which I’m guessing most will begin doing, you’ll be able to simply hit the timeout threshold of AWS Lambda when processing your information, particularly if no less than one perform exceeds quarter-hour. So, whereas Lambda is nice as a result of it’s fast and actually low cost, the timeout generally is a trouble.

The selection right here is to deploy your code as a container that has the choice of operating so long as it must and run it on a schedule. So, as an alternative of spinning up a perform as you do with Lambda, we will spin up a container to run in an ECS cluster utilizing Fargate.

For clarification, Lambda, ECS and EventBridge are all AWS Companies.

Simply as with Lambda, the price of operating a container for an hour or two is minimal. Nonetheless, it’s a bit extra difficult than operating a serverless perform. However in the event you’re studying this, you then’ve in all probability run into the identical points and are questioning what the simplest technique to transition is.

I’ve created a quite simple ETL template that makes use of Google BigQuery to extract and cargo information. This template will get you up and operating inside a couple of minutes in the event you comply with alongside.

Utilizing BigQuery is completely non-compulsory however I normally retailer my long run information there.

As an alternative of constructing one thing advanced right here, I’ll present you the right way to construct one thing minimal and maintain it actually lean.

If you happen to don’t must course of information in parallel, you shouldn’t want to incorporate one thing like Airflow. I’ve seen a couple of articles on the market that unnecessarily arrange advanced workflows, which aren’t strictly obligatory for simple information transformation.

Moreover, in the event you really feel such as you need to add on to this later, that possibility is yours.

Workflow

We’ll construct our script in Python as we’re doing information transformation, then bundle it up with Docker and push it to an ECR repository.

From right here, we will create a process definition utilizing AWS Fargate and run it on a schedule in an ECS cluster.

Don’t fear if this feels overseas; you’ll perceive all these providers and what they do as we go alongside.

Know-how

If you’re new to working with containers, then consider ECS (Elastic Container Service) as one thing that helps us arrange an surroundings the place we will run a number of containers concurrently.

Fargate, alternatively, helps us simplify the administration and setup of the containers themselves utilizing Docker photographs — that are known as duties in AWS.

There’s the choice of utilizing EC2 to arrange your containers, however you would need to do much more handbook work. Fargate manages the underlying cases for us, whereas with EC2, you’re required to handle and deploy your personal compute cases. Therefore, Fargate is also known as the ‘serverless’ possibility.

I discovered a thread on Reddit discussing this, in the event you’re eager to learn a bit about how customers discover utilizing EC2 versus Fargate. It can provide you an thought of how individuals evaluate EC2 and Fargate.

Not that I’m saying Reddit is the supply of fact, but it surely’s helpful for getting a way of person views.

Prices

The first concern I normally have is to maintain the code operating effectively whereas additionally managing the full value.

As we’re solely operating the container when we have to, we solely pay for the quantity of assets we use. The value we pay is decided by a number of elements, such because the variety of duties operating, the execution period of every process, the variety of digital CPUs (vCPUs) used for the duty, and reminiscence utilization.

However to present you a tough thought, on a excessive degree, the full value for operating one process is round $0.01384 per hour for the EU area, relying on the assets you’ve provisioned.

If we had been to check this value with AWS Glue we will get a little bit of perspective whether it is good or not.

If an ETL job requires 4 DPUs (the default quantity for an AWS Glue job) and runs for an hour, it could value 4 DPUs * $0.44 = $1.76. This value is for just one hour and is considerably greater than operating a easy container.

That is, after all, a simplified calculation, and the precise variety of DPUs can fluctuate relying on the job. You’ll be able to try AWS Glue pricing in additional element on their pricing web page.

To run long-running scripts, establishing your personal container and deploying it on ECS with Fargate is sensible, each when it comes to effectivity and value.

To comply with this text, I’ve created a easy ETL template that can assist you stand up and operating rapidly.

This template makes use of BigQuery to extract and cargo information. It can extract a couple of rows, do one thing easy after which load it again to BigQuery.

After I run my pipelines I’ve different issues that remodel information — I take advantage of APIs for pure language processing that runs for a couple of hours within the morning — however that’s as much as you so as to add on later. That is simply to present you a template that might be simple to work with.

To comply with alongside this tutorial, the principle steps might be as follows:

Organising your native code.Organising an IAM person & the AWS CLI.Construct & push Docker picture to AWS.Create an ECS process definition.Create an ECS cluster.Schedule to your duties.

In complete it shouldn’t take you longer than 20 minutes to get by means of this, utilizing the code I’ll offer you. This assumes you could have an AWS account prepared, and if not, add on 5 to 10 minutes.

The Code

First create a brand new folder domestically and find into it.

mkdir etl-pipelinescd etl-pipelines

Be sure to have python put in.

python –version

If not, set up it domestically.

When you’re prepared, you’ll be able to go forward and clone the template I’ve already arrange.

git clone https://github.com/ilsilfverskiold/etl-pipeline-fargate.git

When it has completed fetching the code, open it up in your code editor.

First verify the principle.py file to look how I’ve structured the code to grasp what it does.

Primarily, it should fetch all names with “Doe” in it from a desk in BigQuery that you simply specify, remodel these names after which insert them again into the identical information desk as new rows.

You’ll be able to go into every helper perform to see how we arrange the SQL Question job, remodel the info after which insert it again to the BigQuery desk.

The thought is after all that you simply arrange one thing extra advanced however it is a easy check run to make it simple to tweak the code.

Setting Up BigQuery

If you wish to proceed with the code I’ve ready you will have to arrange a couple of issues in BigQuery. In any other case you’ll be able to skip this half.

Listed below are the issues you will have:

A BigQuery desk with a area of ‘identify’ as a string.A number of rows within the information desk with the identify “Doe” in it.A service account that may have entry to this dataset.

To get a service account you will have to navigate to IAM within the Google Cloud Console after which to Service Accounts.

As soon as there, create a brand new service account.

As soon as it has been created, you will have to present your service account BigQuery Person entry globally by way of IAM.

Additionally, you will have to present this service account entry to the dataset itself which you do in BigQuery straight by way of the dataset’s Share button after which by urgent Add Principal.

After you’ve given the person the suitable permissions, be sure to return to the Service Accounts after which obtain a key. This provides you with a json file that you have to put in your root folder.

Now, a very powerful half is ensuring the code has entry to the google credentials and is utilizing the proper information desk.

You’ll need the json file you’ve downloaded with the Google credentials in your root folder as google_credentials.json and you then need to specify the proper desk ID.

Now you would possibly argue that you do not need to retailer your credentials domestically which is just proper.

You’ll be able to add within the possibility of storing your json file in AWS Secrets and techniques Supervisor later. Nonetheless, to start out, this might be simpler.

Run ETL Pipeline Domestically

We’ll run this code domestically first, simply so we will see that it really works.

So, arrange a Python digital surroundings and activate it.

python -m venv etl-envsource etl-env/bin/activate # On Home windows use `venvScriptsactivate`

Then set up dependencies. We solely have google-cloud-bigquery in there however ideally you should have extra dependencies.

pip set up -r necessities.txt

Run the principle script.

python fundamental.py

This could log ‘New rows have been added’ in your terminal. This can then affirm that the code works as we’ve supposed.

The Docker Picture

Now to push this code to ECS we must bundle it up right into a Docker picture which suggests that you’ll want Docker put in domestically.

If you happen to do not need Docker put in, you’ll be able to obtain it right here.

Docker helps us package deal an software and its dependencies into a picture, which could be simply acknowledged and run on any system. Utilizing ECS, it’s required of us to bundle our code into Docker photographs, that are then referenced by a process definition to run as containers.

I’ve already arrange a Dockerfile in your folder. You must have the ability to look into it there.

FROM –platform=linux/amd64 python:3.11-slim

WORKDIR /app

COPY . /app

RUN pip set up –no-cache-dir -r necessities.txt

CMD [“python”, “main.py”]

As you see, I’ve stored this actually lean as we’re not connecting internet site visitors to any ports right here.

We’re specifying AMD64 which you’ll not want in case you are not on a Mac with an M1 chip but it surely shouldn’t damage. This can specify to AWS the structure of the docker picture so we don’t run into compatibility points.

Create an IAM Person

When working with AWS, entry will should be specified. A lot of the points you’ll run into are permission points. We’ll be working with the CLI domestically, and for this to work we’ll must create an IAM person that may want fairly broad permissions.

Go to the AWS console after which navigate to IAM. Create a brand new person, add permissions after which create a brand new coverage to connect to it.

I’ve specified the permissions wanted in your code within the aws_iam_user.json file. You’ll see a brief snippet beneath of what this json file seems like.

{“Model”: “2012-10-17″,”Assertion”: [{“Sid”: “VisualEditor0″,”Effect”: “Allow”,”Action”: [“logs:CreateLogGroup”,”iam:CreateRole”,”iam:AttachRolePolicy”,”iam:PutRolePolicy”,”ecs:DescribeTaskDefinition”,…more],”Useful resource”: “*”}]}

You’ll want to enter this file to get all of the permissions you will have to set, that is only a brief snippet. I’ve set it to fairly a couple of, which you’ll need to tweak to your personal preferences later.

When you’ve created the IAM person and also you’ve added the proper permissions to it, you will have to generate an entry key. Select ‘Command Line Interface (CLI)’ when requested about your use case.

Obtain the credentials. We’ll use these to authenticate in a bit.

Arrange the AWS CLI

Subsequent, we’ll join our terminal to our AWS account.

If you happen to don’t have the CLI arrange but you’ll be able to comply with the directions right here. It’s very easy to set this up.

When you’ve put in the AWS CLI you’ll must authenticate with the IAM person we simply created.

aws configure

Use the credentials we downloaded from the IAM person within the earlier step.

Create an ECR Repository

Now, we will get began with the DevOps of all of it.

We’ll first must create a repository in Elastic Container Registry. ECR is the place we will retailer and handle our docker photographs. We’ll have the ability to reference these photographs from ECR after we arrange our process definitions.

To create a brand new ECR repository run this command in your terminal. This can create a repository referred to as bigquery-etl-pipeline.

aws ecr create-repository — repository-name bigquery-etl-pipeline

Be aware the repository URI you get again.

From right here we have now to construct the docker picture after which push this picture to this repository.

To do that you’ll be able to technically go into the AWS console and discover the ECR repository we simply created. Right here AWS will allow us to see your complete push instructions we have to run to authenticate, construct and push our docker picture to this ECR repository.

Nonetheless, in case you are on a Mac I’d recommendation you to specify the structure when constructing the docker picture or you could run into points.

If you’re following together with me, then begin with authenticating your docker consumer like so.

aws ecr get-login-password –region YOUR_REGION | docker login –username AWS –password-stdin YOUR_ACCOUNT_ID.dkr.ecr.YOUR_REGION.amazonaws.com

Make sure you change the values, area and account ID the place relevant.

Construct the docker picture.

docker buildx construct –platform=linux/amd64 -t bigquery-etl-pipeline .

That is the place I’ve tweaked the command to specify the linux/amd64 structure.

Tag the docker picture.

docker tag bigquery-etl-pipeline:newest YOUR_ACCOUNT_ID.dkr.ecr.YOUR_REGION.amazonaws.com/bigquery-etl-pipeline:newest

Push the docker picture.

docker push YOUR_ACCOUNT_ID.dkr.ecr.YOUR_REGION.amazonaws.com/bigquery-etl-pipeline:newest

If every thing labored as deliberate you’ll see one thing like this in your terminal.

9f691c4f0216: Pushed ca0189907a60: Pushed 687f796c98d5: Pushed 6beef49679a3: Pushed b0dce122021b: Pushed 4de04bd13c4a: Pushed cf9b23ff5651: Pushed 644fed2a3898: Pushed

Now that we have now pushed the docker picture to an ECR repository, we will use it to arrange our process definition utilizing Fargate.

If you happen to run into EOF points right here it’s most certainly associated to IAM permissions. Make sure you give it every thing it wants, on this case full entry to ECR to tag and push the picture.

Roles & Log Teams

Keep in mind what I advised you earlier than, the most important points you’ll run into in AWS pertains to roles between totally different providers.

For this to stream neatly we’ll have to ensure we arrange a couple of issues earlier than we begin establishing a process definition and an ECS cluster.

To do that, we first must create a process position — this position is the position that may want entry to providers within the AWS ecosystem from our container — after which the execution position — so the container will have the ability to pull the docker picture from ECR.

aws iam create-role –role-name etl-pipeline-task-role –assume-role-policy-document file://ecs-tasks-trust-policy.jsonaws iam create-role – role-name etl-pipeline-execution-role – assume-role-policy-document file://ecs-tasks-trust-policy.json

I’ve specified a json file referred to as ecs-tasks-trust-policy.json in your folder domestically that it’s going to use to create these roles.

For the script that we’re pushing, it gained’t must have permission to entry different AWS providers so for now there isn’t any want to connect insurance policies to the duty position. Nonetheless, you could need to do that later.

Nonetheless, for the execution position although we might want to give it ECR entry to tug the docker picture.

To connect the coverage AmazonECSTaskExecutionRolePolicy to the execution position run this command.

aws iam attach-role-policy –role-name etl-pipeline-execution-role –policy-arn arn:aws:iam::aws:coverage/service-role/AmazonECSTaskExecutionRolePolicy

We additionally create one final position whereas we’re at it, a service position.

aws iam create-service-linked-role – aws-service-name ecs.amazonaws.com

If you happen to don’t create the service position you could find yourself with an errors corresponding to ‘Unable to imagine the service linked position. Please confirm that the ECS service linked position exists’ once you attempt to run a process.

The very last thing we create a log group. Making a log group is important for capturing and accessing logs generated by your container.

To create a log group you’ll be able to run this command.

aws logs create-log-group – log-group-name /ecs/etl-pipeline-logs

When you’ve created the execution position, the duty position, the service position after which the log group we will proceed to arrange the ECS process definition.

Create an ECS Process Definition

A process definition is a blueprint to your duties, specifying what container picture to make use of, how a lot CPU and reminiscence is required, and different configurations. We use this blueprint to run duties in our ECS cluster.

I’ve already arrange the duty definition in your code at task-definition.json. Nonetheless, you have to set your account id in addition to area in there to ensure it runs because it ought to.

{“household”: “my-etl-task”,”taskRoleArn”: “arn:aws:iam::ACCOUNT_ID:position/etl-pipeline-task-role”,”executionRoleArn”: “arn:aws:iam::ACCOUNT_ID:position/etl-pipeline-execution-role”,”networkMode”: “awsvpc”,”containerDefinitions”: [{“name”: “my-etl-container”,”image”: “ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/bigquery-etl-pipeline:latest”, “cpu”: 256,”memory”: 512,”essential”: true,”logConfiguration”: {“logDriver”: “awslogs”,”options”: {“awslogs-group”: “/ecs/etl-pipeline-logs”,”awslogs-region”: “REGION”,”awslogs-stream-prefix”: “ecs”}}}],”requiresCompatibilities”: [“FARGATE”],”cpu”: “256”,”reminiscence”: “512”}

Keep in mind the URI we acquired again after we created the ECR repository? That is the place we’ll use it. Keep in mind the execution position, the duty position and the log group? We’ll use it there as properly.

If you happen to’ve named the ECR repository together with the roles and log group precisely what I named mine then you’ll be able to merely change the account ID and Area on this json in any other case be sure the URI is the proper one.

You may also set CPU and reminiscence right here for what you’ll must run your process — i.e. your code. I’ve set .25 vCPU and 512 mb as reminiscence.

When you’re glad you’ll be able to register the duty definition in your terminal.

aws ecs register-task-definition –cli-input-json file://task-definition.json

Now you need to have the ability to go into Amazon Elastic Container Service after which discover the duty we’ve created underneath Process Definitions.

This process — i.e. blueprint — gained’t run on it’s personal, we have to invoke it later.

Create an ECS Cluster

An ECS Cluster serves as a logical grouping of duties or providers. You specify this cluster when operating duties or creating providers.

To create a cluster by way of the CLI run this command.

aws ecs create-cluster –cluster-name etl-pipeline-cluster

When you run this command, you’ll have the ability to see this cluster in ECS in your AWS console in the event you look there.

We’ll connect the Process Definition we simply created to this cluster after we run it for the subsequent half.

Run Process

Earlier than we will run the duty we have to get ahold of the subnets which are obtainable to us together with a safety group id.

We are able to do that straight within the terminal by way of the CLI.

Run this command within the terminal to get the obtainable subnets.

aws ec2 describe-subnets

You’ll get again an array of objects right here, and also you’re on the lookout for the SubnetId for every object.

If you happen to run into points right here, be sure your IAM has the suitable permissions. See the aws_iam_user.json file in your root folder for the permissions the IAM person linked to the CLI will want. I’ll stress this, as a result of it’s the principle points that I all the time run into.

To get the safety group ID you’ll be able to run this command.

aws ec2 describe-security-groups

You’re on the lookout for GroupId right here within the terminal.

If you happen to acquired no less than one SubnetId after which a GroupId for a safety group, we’re able to run the duty to check that the blueprint — i.e. process definition — works.

aws ecs run-task –cluster etl-pipeline-cluster –launch-type FARGATE –task-definition my-etl-task –count 1 –network-configuration “awsvpcConfiguration={subnets=[SUBNET_ID],securityGroups=[SECURITY_GROUP_ID],assignPublicIp=ENABLED}”

Do bear in mind to alter the names in the event you’ve named your cluster and process definition in another way. Keep in mind to additionally set your subnet ID and safety group ID.

Now you’ll be able to navigate to the AWS console to see the duty operating.

If you’re having points you’ll be able to look into the logs.

If profitable, you need to see a couple of remodeled rows added to BigQuery.

EventBridge Schedule

Now, we’ve managed to arrange the duty to run in an ECS cluster. However what we’re taken with is to make it run on a schedule. That is the place EventBridge is available in.

EventBridge will arrange our scheduled occasions, and we will set this up utilizing the CLI as properly. Nonetheless, earlier than we arrange the schedule we first must create a brand new position.

That is life when working with AWS, every thing must have permission to work together with one another.

On this case, EventBridge will want permission to name the ECS cluster on our behalf.

Within the repository you could have a file referred to as trust-policy-for-eventbridge.json that I’ve already put there, we’ll use this file to create this EventBridge position.

Paste this into the terminal and run it.

aws iam create-role –role-name ecsEventsRole –assume-role-policy-document file://trust-policy-for-eventbridge.json

We then have to connect a coverage to this position.

aws iam attach-role-policy –role-name ecsEventsRole –policy-arn arn:aws:iam::aws:coverage/AmazonECS_FullAccess

We’d like it to no less than have the ability to have ecs:RunTask however we’ve given it full entry. If you happen to want to restrict the permissions, you’ll be able to create a customized coverage with simply the mandatory permissions as an alternative.

Now let’s arrange the rule to schedule the duty to run with the duty definition day-after-day at 5 am UTC. That is normally the time I’d like for it to course of information for me so if it fails I can look into it after breakfast.

aws occasions put-rule –name “ETLPipelineDailyRun” –schedule-expression “cron(0 5 * * ? *)” –state ENABLED

You must obtain again an object with a area referred to as RuleArn right here. That is simply to verify that it labored.

Subsequent step is now to affiliate the rule with the ECS process definition.

aws occasions put-targets –rule “ETLPipelineDailyRun” –targets “[{“Id”:”1″,”Arn”:”arn:aws:ecs:REGION:ACCOUNT_NUMBER:cluster/etl-pipeline-cluster”,”RoleArn”:”arn:aws:iam::ACCOUNT_NUMBER:role/ecsEventsRole”,”EcsParameters”:{“TaskDefinitionArn”:”arn:aws:ecs:REGION:ACCOUNT_NUMBER:task-definition/my-etl-task”,”TaskCount”:1,”LaunchType”:”FARGATE”,”NetworkConfiguration”:{“awsvpcConfiguration”:{“Subnets”:[“SUBNET_ID”],”SecurityGroups”:[“SECURITY_GROUP_ID”],”AssignPublicIp”:”ENABLED”}}}}]”

Keep in mind to set your personal values right here for area, account quantity, subnet and safety group.

Use the subnets and safety group that we acquired earlier. You’ll be able to set a number of subnets.

When you’ve run the command the duty is scheduled for five am day-after-day and also you’ll discover it underneath Scheduled Duties within the AWS Console.

AWS Secrets and techniques Supervisor (Optionally available)

So protecting your Google credentials within the root folder isn’t ultimate, even in the event you’ve restricted entry to your datasets for the Google service account.

Right here we will add on the choice of transferring these credentials to a different AWS service after which accessing it from our container.

For this to work you’ll have to maneuver the credentials file to Secrets and techniques Supervisor, tweak the code so it may fetch it to authenticate and make it possible for the duty position has permissions to entry AWS Secrets and techniques Supervisor in your behalf.

While you’re performed you’ll be able to merely push the up to date docker picture to your ECR repo you arrange earlier than.

The Finish Outcome

Now you’ve acquired a quite simple ETL pipeline operating in a container on AWS on a schedule. The thought is that you simply add to it to do your personal information transformations.

Hopefully this was a helpful piece for anybody that’s transitioning to establishing their long-running information transformation scripts on ECS in a easy, value efficient and simple approach.

Let me know in the event you run into any points in case there’s something I missed to incorporate.

❤