Masterclass Python for Data Engineers in Fabric (2 days)

Introduction

The world of data engineering is changing. Being fluent in T-SQL is not enough anymore. Whether you want to implement a Delta Lake using Data Bricks, use Jupyter Notebooks to analyse your data or built a ETL pipeline in Microsoft Fabric, Python is the new kid on the block.

Topics

Module 0: Overview

Use cases Python
Modules overview
Installation

Module 1: Syntax + Expressions

Syntax
String Expressions
Number Expressions
DateTime Expressions
Logical Expressions
List Expressions
Loops
Dictionary Expressions

Module 2: Modular programming

Functions
Classes
Pip package manager
Modules, Packages
Error handling
Jupyter Notebooks + Markdown

Module 3: Data Wranging with Pandas

Pandas Data Frames / NumPy Arrays / Vectorized Operations
Import files: CSV, Parquet + Excel (SharePoint)
Transforming Data: Filter, Group By, Select, Join, Unpivot
Read / write to Azure Data Lake Storage Gen 2 (ADLS)
Import from SQL Server with SQL Alchemy

Module 4: Lakehouse / Apache Spark / Big Data

Big Data Architecture: Data Warehouse/ Data Lake Lakehouse / Fabric
Spark overview: architecture , RDD, Dataframe
Import files: CSV, Parquet files
Transforming Data: Filter, Group By, Select
Import from SQL Server
Save to Lakehouse: Files + Tables
Delta Lake
CRUD operations

Module 5: Calling REST API's

REST API's + HTTP
Request / Response
ODATA

Target audience

Data engineers seeking knowledge in data manipulation and integration with Python using Pandas and Spark. Suitable for those interested in modular programming, Lakehouse architectures, and REST APIs.

Prerequisites

Prerequisites: Familiarity with basic programming concepts, an understanding of database structures, and experience with data formats such as CSV and Excel. Prior knowledge of SQL and data storage principles is beneficial.

Information

Interested in this training? Click here for more information.