Add a blog post to highlight @HamiltonHord's presentation

This commit is contained in:
R Tyler Croy 2021-07-08 08:47:42 -07:00
parent 1f0d161c56
commit 9cf69cfde1
1 changed files with 51 additions and 0 deletions

View File

@ -0,0 +1,51 @@
---
layout: post
title: "Automating Databricks with Terraform"
team: Core Platform
author: rtyler
tags:
- databricks
- terraform
- featured
---
The long term success of our data platform relies on putting tools into the
hands of developers and data scientists to “choose their own adventure”. A big
part of that story has been [Databricks](https://databricks.com) which we
recently integrated with [Terraform](https://terraform.io) to make it easy to
scale a top-notch developer experience. At the 2021 Data and AI Summit, Core
Platform infrastructure engineer [Hamilton
Hord](https://github.com/HamiltonHord) and Databricks engineer [Serge
Smertin](https://github.com/nfx) presented on the Databricks terraform provider
and how it's been used by Scribd.
In the session embedded below, they share the details on the [Databricks (Labs)
Terraform
integration](https://github.com/databrickslabs/terraform-provider-databricks)
and how it can automate literally every aspect required for a production-grade
platform: data security, permissions, continuous deployment and so on. They
also discuss the ways in which our Core Platform team enables internal
customers without acting as gatekeepers for data platform changes. Just about
anything they might need in Databricks is a pull request away!
<center>
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/h8LrVmb4W2Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</center>
In hindsight, it's mind-boggling how much manual configuration we had to
previously maintain. With the Terraform provider for Databricks we can very
easily test, reproduce, and audit hundreds of different business critical
Databricks resources. Coupling Terraform with the recent "multi-workspace"
support that Databricks unveiled in 2020 means we can also now provision an
entirely new environment in a few hours!
Investing in data platform tools and automation is a key part of the vision for
Platform Engineering which encompasses Data Engineering, Data Operations, and
Core Platform. We have a [number of open positions](/careers/#open-positions)
at the moment, but I wanted to call special attention to the [Data Engineering
Manager](https://jobs.lever.co/scribd/e2187c1c-a1d6-4b77-bde6-acc997f68156)
role for which we're currently hiring. The leader of the Data Engineering team
will help deliver data tools and solutions for internal customers building on
top of Delta Lake, Databricks, Airflow, and Kafka. Suffice it to say, there's a
lot of really interesting work to be done!