Build Streaming Data Pipelines in Azure

Description

Microsoft Azure is a cloud platform in which you can store and manage data and deploy virtual machines. It consists of more than 200 individual products and cloud services and enables companies to develop, run and manage applications in the cloud. Azure also provides many frameworks and tools to run applications in multiple clouds, locally or at the edge.

What you will learn

In this training, Kristijan Bakarić teaches you how to build a Streaming Data Pipeline in Azure by working on a hands-on project. This project introduces a set of Azure services to ingest (APIM), store (Blob Storage), process (Azure Functions), serve (Cosmos DB) and visualize (Power BI) flows of twitter data coming in as JSON messages.

After the project introduction and a high level overview of the solution architecture, you will build its components and connect them into a data pipeline. Language of choice for local data preprocessing and developing Azure functions is Python.

During this project, you create a JSON file that contains messages and write a Python program that will send objects in JSON as messages via HTTP requests to Azure API Management. You also learn how to develop and deploy Azure functions by using Python and Visual Studio Code and create an Azure function project with one default dummy function.

Furthermore, you are going to create and combine Event Hubs, Azure Functions and Cosmos DB. In this context, you will also write tweets to Cosmos DB from Event Hub and connect a Power BI Desktop to your Cosmos DB.

Requirements

Azure Account
Software Development Basics
Basic Python Skills
Fundamentals of Data Stores (See 2.3 Choosing Data Sources Training in Academy)
Fundamentals of APIs (See 3.1 API Design & Development with FastAPI)
Fundamentals of Message Queues (See 3.3 Apache Kafka Training in Academy)

About the Author

Kristijan Bakarić

Kristijan is a hybrid of a scientist and an engineer working for an international energy company since 2013.

He worked in the business of oil and gas exploration as an explorationist for half of his career and the second half with a variety of subsurface data (analytics, science and engineering) projects, and most recently in digitization and digitalization of operations.

He holds a wide, generalist knowledge in a variety of domains, tools and technologies with a current focus in Data Engineering and Azure.

He is passionate about data, technology and learning, and especially sharing his knowledge and experience with others.

Connect with Kristijan on LinkedIn or Twitter