Introduction
Description
As an engineer you very often have to quickly set up a simple ETL script that just does the job. In this project I show you how you can easily do this on AWS. You'll use the live data from a weather API and write it into the timeseries database TDengine.
What will I learn?
Timeseries Databases
Learn the basics of timeseries databases, how they work and when you would use them.
Use an online weather API
Setup and explore a public weather API. Create a Python script that reads data from the API.
Create a Dockerized ETL script on AWS
Package your script into a Docker container and use Amazon Elastic Container Registry, Lambda and EventBridge to deploy it as a serverless ETL script.
Setup Timeseries Database TDengine
Learn about TDengine, setup an instance through TDengine cloud and configure your database.
Visualize data with Grafana
Visualize the data you extracted from the API and stored in TDengine with Grafana. You'll learn how to setup and configure Grafana with TDengine.
Course Curriculum
- Quick note from Andreas before you begin (0:43)
- Introduction (1:26)
- Setup Of The Project (2:52)
- Time Series Data Basics (2:20)
- Big Pros Of Timeseries Databases (2:06)
- About TDengine (1:22)
- Setup Weather API (1:04)
- Code query API (2:41)
- TDengine Setup (3:04)
- Connect Python To TDengine (1:50)
- Lambda Docker Container & Push To ECR (1:55)
- AWS Setup (1:36)
- Create Lambda Function Using Docker image (1:04)
- Schedule Function With EventBridge (1:25)
- Cloud Watch Lambda Events (0:27)
- Grafana Setup (3:01)
Pricing
Dockerized ETL is included in our Data Engineering Academy