scribd
/
scribd.github.io
mirror of https://github.com/scribd/scribd.github.io
1
0
Fork 0
Code Issues Releases Wiki Activity

Fargate deploy post updates

This commit is contained in:
Nakul Pathak 2021-03-22 13:57:03 -04:00 committed by R. Tyler Croy
parent e6ef06c8e3
commit b107b9252a
1 changed files with 12 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
_posts/2021-03-18-faster-fargate-deploys.md
View File

@ -9,7 +9,7 @@ tags:
team: Internal Tools
---
Scribd moved its monolith to AWS in April 2020 and as part of the migration, we had to design and implement new deployment pipelines in our *new and shiny* [ECS Fargate](https://aws.amazon.com/fargate/) infrastructure. In this post, we'll share how we improved our deployment speeds from ~40 minutes to less than 20 minutes.
Scribd moved its monolith to AWS in April 2020 and as part of the migration, we had to design and implement a deployment pipeline for our new (and *shiny*) [ECS Fargate](https://aws.amazon.com/fargate/) infrastructure. In this post, we'll share how we improved our deployment speeds from ~40 minutes to less than 20 minutes.
### Original Implementation
@ -21,38 +21,40 @@ Our starting implementation involved a few steps:
### Improvements
#### Fargate Service Updates
By far, the slowest part of our deployment was waiting for ECS services to finish updating. We use the default rolling deployment which stops and starts tasks to force re-pulling of the freshly-uploaded [ECR](https://aws.amazon.com/ecr/) image. We were able to reduce this time to 16-18 minutes with the following -
By far, the slowest part of our deployment was waiting for ECS services to finish updating. We use the default rolling deployment which stops and starts tasks to trigger a re-pulling of the freshly-uploaded [ECR](https://aws.amazon.com/ecr/) image. Here are some changes we implemented -
* **Docker Image Size Reduction** - The first thing everyone thinks of when considering ECS Fargate speedups is how to reduce the image pull time since Fargate (unlike EC2) [has no image caching](https://github.com/aws/containers-roadmap/issues/696). However, unless you can drastically reduce your image size (think 1Gb to 100Mb), this will not lead to significant time reductions. We reduced our compressed image size from ~900Mb to ~700Mb and it led to **little to no improvement**. It did lead to a cleaner image but that wasn't our initial goal.
* [**Deregistration Delay**](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#deregistration-delay) - This is a property on your load balancer's target group that dictates how long a task stays in *Draining* state after it stops receiving requests. We looked in Datadog APM for the p99 latencies of our longest-running requests and changed the value from the **default 300 seconds** to 17s. This reduced service refreshes to ~22 minutes.
* [**Deregistration Delay**](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#deregistration-delay) - This is a property on a load balancer's target group that dictates how long a task stays in *Draining* state after it stops receiving requests. We looked at Datadog APM for the p99 of our longest running request and set the delay to 17s from the **default of 300s**. This reduced service refreshes to ~22 minutes.
* **ECS Throttling** - During deployments, we investigated the "Events" tab of our main web ECS service. There were events with the following messages -
- *"service production-web operations are being throttled on elb. Will try again later."*
- *"service production-web operations are being throttled. Will try again later."*
Due to Scribd's high Fargate task volume, the number of requests we were making during rolling deploys to start and stop tasks was too high for AWS' default limits. We opened support tickets with the ELB and Fargate teams and were able to get those limits improved/removed. This further reduced service deploy time to 16-18 minutes.
* **Network Load Balancer Health Checks** - From testing in staging, we noticed that reducing our network load balancer's [health-check intervals and thresholds](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html) helped reduce staging deploy time from ~9 to ~6 minutes. However, it only translated to 1-2 minutes saved in production with much higher number of ECS tasks. You do want to be careful with the value to avoid false-positive health checks and keep in mind that updating these values requires re-creation of the ECS service it points to.
Due to Scribd's high Fargate task volume, the number of start and stop requests we were making was too high for AWS' default limits. We opened support tickets with the ELB and Fargate teams to get those limits increased. This further reduced service deploy time to 16-18 minutes.
* **Network Load Balancer Health Checks** - From testing in staging, we noticed that reducing our network load balancer's [health-check intervals and thresholds](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html) helped reduce staging deploy time from ~9 to ~6 minutes. However, it only translated to 1-2 minutes saved in production with much higher number of ECS tasks. You do want to be careful with the value to avoid false-positive health checks and keep in mind that updating these values requires re-creation of the ECS service the load balancer points to.
#### Asset Deployment Improvements
Our asset deployments were run using Capistrano. The job `ssh`-ed onto our asset servers, ran a series of tasks to download, unzip, and correctly place assets. There were some issues with this approach -
Our asset deployments were run using Capistrano. The job `ssh`-ed onto our asset servers, ran a series of [Rake tasks]((https://guides.rubyonrails.org/v4.2/command_line.html#rake)) to download, unzip, and correctly place assets. There were some issues with this approach -
* Dependency on Capistrano gem forced us to use the monolith Docker image as the job's base image
* Our ECS service refresh job runs `docker push/pull` tasks to upload the latest image to ECR. Since we wanted to avoid Docker-in-Docker due to further bloating of the monolith image for this one case, we had separate jobs for asset and container deployment. This forced us to waste valuable Gitlab job startup and shutdown time.
* Running Rake tasks required loading the application which adds time to the job
* Our ECS service refresh job runs `docker push/pull` tasks to upload the latest image to ECR. This forced us to have separate jobs for asset and service deployments to avoid adding a Docker dependency to the monolith image.
To resolve these issues, we decided to remove Capistrano as a dependency and wrote Ruby and Bash code that performed the exact same tasks. This was added to our service deployment job and brought asset deploy time from 2.5 minutes to 30s.
To resolve these issues, we decided to remove Capistrano & Rake as dependencies and wrote pure Ruby and Bash code to perform the tasks. This unified the two jobs and brought asset deploy time from 2.5 minutes to 30s.
#### Database Migration
In our case, running a database migration task in Fargate involved starting a new task instance of our `database_migration` task family. Due to Fargate startup slowness, this task would take 3 minutes to run a simple `bundle exec rails db:migrate`.
To resolve this, we used `git` and [Gitlab environments](https://docs.gitlab.com/ee/api/environments.html#get-a-specific-environment) to look for modified files in the `db/migrate` directory. If none were found, we would skip running the migration task. Since majority of our deployments don't run database migration tasks, this shaved off 3 minutes from most jobs.
```
env_json=$(curl --silent --header "<SECRET_ENV_STUFF>" "<gitlab-repository-path>/environments/:id")
env_json=$(curl --silent --header "PRIVATE-TOKEN: <your_access_token>" "<gitlab-repository-path>/environments/<id>")
last_deployment_sha=$(echo $env_json | jq -r '.last_deployment.sha')
git diff --name-only $CI_COMMIT_SHA $last_deployment_sha | grep db/migrate
```
#### Other things to look for
If you run sidecar containers like Datadog, make sure that you're providing enough memory and CPU to those sidecars to avoid waiting on the sidecar to come up while your main container is ready.
If you run sidecar containers like Datadog, make sure that you're providing enough memory and CPU to those containers to avoid waiting on them to be ready while your main container has already started.
We hope this helps you speed up your deployments and gain greater efficiency!