Description

Microsoft Azure is a cloud platform in which you can store and manage data and deploy virtual machines. It consists of more than 200 individual products and cloud services and enables companies to develop, run and manage applications in the cloud. Azure also provides many frameworks and tools to run applications in multiple clouds, locally or at the edge.



What you will learn

In this training, Kristijan Bakarić teaches you how to build a Streaming Data Pipeline in Azure by working on a hands-on project. This project introduces a set of Azure services to ingest (APIM), store (Blob Storage), process (Azure Functions), serve (Cosmos DB) and visualize (Power BI) flows of twitter data coming in as JSON messages. 

After the project introduction and a high level overview of the solution architecture, you will build its components and connect them into a data pipeline. Language of choice for local data preprocessing and developing Azure functions is Python.

During this project, you create a JSON file that contains messages and write a Python program that will send objects in JSON as messages via HTTP requests to Azure API Management. You also learn how to develop and deploy Azure functions by using Python and Visual Studio Code and create an Azure function project with one default dummy function.

Furthermore, you are going to create and combine Event Hubs, Azure Functions and Cosmos DB. In this context, you will also write tweets to Cosmos DB from Event Hub and connect a Power BI Desktop to your Cosmos DB.



Requirements

  • Azure Account
  • Software Development Basics
  • Basic Python Skills
  • Fundamentals of Data Stores (See 2.3 Choosing Data Sources Training in Academy)
  • Fundamentals of APIs (See 3.1 API Design & Development with FastAPI)
  • Fundamentals of Message Queues (See 3.3 Apache Kafka Training in Academy)



About the Author

Kristijan Bakarić


Kristijan is a hybrid of a scientist and an engineer working for an international energy company since 2013.

He worked in the business of oil and gas exploration as an explorationist for half of his career and the second half with a variety of subsurface data (analytics, science and engineering) projects, and most recently in digitization and digitalization of operations.

He holds a wide, generalist knowledge in a variety of domains, tools and technologies with a current focus in Data Engineering and Azure.

He is passionate about data, technology and learning, and especially sharing his knowledge and experience with others.

Connect with Kristijan on LinkedIn or Twitter

Project Curriculum



  01 Project Introduction
Available in days
days after you enroll
  02 Datasets and Local Preprocessing
Available in days
days after you enroll
  03 Azure Functions and Blob Storage
Available in days
days after you enroll
  04 Add Azure Function to Azure API management APIM
Available in days
days after you enroll
  05 Create and Combine Event Hubs, Azure Function and Cosmos DB
Available in days
days after you enroll
  06 Write Tweets to Cosmos DB (Core SQL) from Event Hub
Available in days
days after you enroll
  07 Connect Power BI Desktop to your Cosmos DB
Available in days
days after you enroll