Build scalable data engineering pipelines and manage end-to-end workflows using Databricks

This course teaches how to use Databricks for data engineering, including working with Apache Spark, Delta Lake, data pipelines, job orchestration and performance optimization for batch and streaming workloads.

  • Why get trained: Learn how to build and manage data pipelines using Apache Spark, Delta Lake, Databricks Workflows and structured streaming for scalable data engineering.
  • Why it matters: Data engineering capabilities enable organizations to process, transform and deliver reliable data pipelines that power analytics, AI and business intelligence.
  • Who should attend: Data engineers, data professionals and developers responsible for building and managing scalable data pipelines using Databricks and Apache Spark.

Build the capability to design and implement scalable data engineering workflows on Databricks with Trainocate. HRD Corp Claimable.

Overview

This is an introductory course that serves as an appropriate entry point to learn Data Engineering with Databricks.

Below, we describe each of the four, four-hour modules included in this course.

1. Data Ingestion with Lakeflow Connect

This course provides a comprehensive introduction to Lakeflow Connect as a scalable and simplified solution for ingesting data into Databricks from a variety of data sources. You will begin by exploring the different types of connectors within Lakeflow Connect (Standard and Managed), learn about various ingestion techniques, including batch, incremental batch, and streaming, and then review the key benefits of Delta tables and the Medallion architecture.

From there, you will gain practical skills to efficiently ingest data from cloud object storage using Lakeflow Connect Standard Connectors with methods such as CREATE TABLE AS (CTAS), COPY INTO, and Auto Loader, along with the benefits and considerations of each approach. You will then learn how to append metadata columns to your bronze level tables during ingestion into the Databricks data intelligence platform. This is followed by working with the rescued data column, which handles records that don’t match the schema of your bronze table, including strategies for managing this rescued data.

The course also introduces techniques for ingesting and flattening semi-structured JSON data, as well as enterprise-grade data ingestion using Lakeflow Connect Managed Connectors.

Finally, learners will explore alternative ingestion strategies, including MERGE INTO operations and leveraging the Databricks Marketplace, equipping you with foundational knowledge to support modern data engineering ingestion.

2. Deploy Workloads with Lakeflow Jobs

Deploy Workloads with Lakeflow Jobs course teaches how to orchestrate and automate data, analytics, and AI workflows using Lakeflow Jobs. You will learn to make robust, production-ready pipelines with flexible scheduling, advanced orchestration, and best practices for reliability and efficiency-all natively integrated within the Databricks Data intelligence Platform. Prior experience with Databricks, Python and SQL is recommended.

3. Build Data Pipelines with Lakeflow Spark Declarative Pipelines 

This course introduces users to the essential concepts and skills needed to build data pipelines using Lakeflow Spark Declarative Pipelines (SDP) in Databricks for incremental batch or streaming ingestion and processing through multiple streaming tables and materialized views. Designed for data engineers new to Spark Declarative Pipelines, the course provides a comprehensive overview of core components such as incremental data processing, streaming tables, materialized views, and temporary views, highlighting their specific purposes and differences.

Topics covered include:

– Developing and debugging ETL pipelines with the multi-file editor in Spark Declarative Pipelines using SQL (with Python code examples provided)

– How Spark Declarative Pipelines track data dependencies in a pipeline through the pipeline graph

– Configuring pipeline compute resources, data assets, trigger modes, and other advanced options

Next, the course introduces data quality expectations in Spark Declarative Pipelines, guiding users through the process of integrating expectations into pipelines to validate and enforce data integrity. Learners will then explore how to put a pipeline into production, including scheduling options, and enabling pipeline event logging to monitor pipeline performance and health.

Finally, the course covers how to implement Change Data Capture (CDC) using the AUTO CDC INTO syntax within Spark Declarative Pipelines to manage slowly changing dimensions (SCD Type 1 and Type 2), preparing users to integrate CDC into their own pipelines.

4. Data Management and Governance with Unity Catalog

In this course, you’ll learn about data management and governance using Databricks Unity Catalog. It covers foundational concepts of data governance, complexities in managing data lakes, Unity Catalog’s architecture, security, administration, and advanced topics like fine-grained access control, data segregation, and privilege management.

* This course seeks to prepare students to complete the Associate Data Engineering certification exam, and provides the requisite knowledge to take the course Advanced Data Engineering with Databricks.

Skills Covered

  • Data Ingestion with Lakeflow Connect
  • Deploy Workloads with Lakeflow Jobs
  • Build Data Pipelines with Lakeflow Spark Declarative Pipelines
  • Data Management and Governance with Unity Catalog

Prerequisites

1. Data Ingestion with Lakeflow Connect

  • Basic understanding of the Databricks Data Intelligence platform, including Databricks Workspaces, Apache Spark, Delta Lake, the Medallion Architecture and Unity Catalog.
  • Experience working with various file formats (e.g., Parquet, CSV, JSON, TXT).
  • Proficiency in SQL and Python.
  • Familiarity with running code in Databricks notebooks.

2. Deploy Workloads with Lakeflow Jobs

  • Beginner familiarity with basic cloud concepts (virtual machines, object storage, identity management)
  • Ability to perform basic code development tasks (create compute, run code in notebooks, use basic notebook operations, import repos from git, etc.)
  • Intermediate familiarity with basic SQL concepts (CREATE, SELECT, INSERT, UPDATE, DELETE, WHILE, GROUP BY, JOIN, etc.)

3. Build Data Pipelines with Lakeflow Spark Declarative Pipelines

  • Basic understanding of the Databricks Data Intelligence platform, including Databricks Workspaces, Apache Spark, Delta Lake, the Medallion Architecture and Unity Catalog.
  • Experience ingesting raw data into Delta tables, including using the read_files SQL function to load formats like CSV, JSON, TXT, and Parquet.
  • Proficiency in transforming data using SQL, including writing intermediate-level queries and a basic understanding of SQL joins.

4. Data Management and Governance with Unity Catalog

  • Beginner familiarity with cloud computing concepts (virtual machines, object storage, etc.)
  • Intermediate experience with basic SQL concepts such as SQL commands, aggregate functions, filters and sorting, indexes, tables, and views.
  • Basic knowledge of Python programming, jupyter notebook interface, and PySpark fundamentals.

Target Audience

  • Everyone who is interested

Course Curriculum

Module 1: Data Ingestion with Lakeflow Connect

  • Introduction to Data Engineering in Databricks
  • Cloud Storage Ingestion with LakeFlow Connect Standard Connector
  • Enterprise Data Ingestion with LakeFlow Connect Managed Connectors
  • Ingestion Alternatives

Module 2: Deploy Workloads with Lakeflow Jobs

  • Introduction to Data Engineering in Databricks
  • Lakeflow Jobs Core Concepts
  • Creating and Scheduling Jobs
  • Advance Lakeflow Jobs Features

Module 3: Build Data Pipelines with Lakeflow Spark Declarative Pipelines

  • Introduction to Data Engineering in Databricks
  • Lakeflow Spark Declarative Pipeline Fundamentals
  • Building Lakeflow Spark Declarative Pipelines

Module 4: Data Management and Governance with Unity Catalog

  • Data Governance Overview
  • Demo: Populating the Metastore
  • Lab: Navigating the Metastore
  • Organization and Access Patterns
  • Demo: Upgrading Tables to Unity Catalog
  • Security and Administration in Unity Catalog
  • Databricks Marketplace Overview
  • Privileges in Unity Catalog
  • Demo: Controlling Access to Data
  • Fine-Grained Access Control
  • Lab: Migrating and Managing Data in Unity Catalog

Dates & Locations

Let’s make it work for you

Can’t find a date that fits? Need to train your whole team? Looking for a discount?
Speak to one of our learning experts today.

x Clear all selections

June 22, 2026 - June 23, 2026

Location: Online
Modal: VILT
Availability: TBC
Exam:
RM 900
RM 6750

July 28, 2026 - July 29, 2026

Location: Online
Modal: VILT
Availability: TBC
Exam:
RM 900
RM 6750
Trainocate exam and cert

Exam & Certification

Databricks Certified Data Engineer Associate.

The Databricks Certified Data Engineer Associate certification exam assesses an individual’s ability to use the Databricks Data Intelligence Platform to complete introductory data engineering tasks. This includes an understanding of the Data Intelligence Platform and its workspace, its architecture, and its capabilities.

It also assesses the ability to perform ETL tasks using Apache Spark SQL or PySpark, covering extraction, complex data handling and User defined functions. Finally, the exam assesses the tester’s ability to  deploy and orchestrate workloads with Databricks workflows configuring and scheduling jobs effectively.
Individuals who pass this certification exam can be expected to complete basic data engineering tasks using Databricks and its associated tools.

The exam covers:

  1. Databricks Intelligence Platform – 10%

  2. Development and Ingestion – 30%

  3. Data Processing & Transformations – 31%

  4. Productionizing Data Pipelines – 18%

  5. Data Governance & Quality – 11%

Training & Certification Guide

Frequently Asked Questions

Data Engineering with Databricks course provides guidance to build scalable, production-ready data engineering pipelines using Databricks and Delta Lake.

The course focuses on data ingestion, transformation, orchestration, governance, streaming pipelines, and Lakehouse architecture using the Databricks Data Intelligence Platform.

Key learning areas:

  • ETL and ELT pipelines
  • Delta Lake and Lakehouse architecture
  • Structured Streaming and Auto Loader
  • Workflow orchestration
  • Unity Catalog governance
  • Data quality and reliability engineering

The course aligns closely with enterprise-scale data engineering practices used across analytics and AI environments.

Pro Tip: Focus on understanding end-to-end pipeline design and operational reliability, not just notebook development

Data Engineering with Databricks is designed for professionals responsible for building and maintaining modern data platforms and analytics pipelines.

The course is ideal for learners transitioning into cloud-native data engineering and Lakehouse-based architectures.

Best suited for:

  • Data Engineers
  • Analytics Engineers
  • Cloud Data Engineers
  • Big Data Developers
  • AI Infrastructure Engineers

Recommended prerequisites:

  • SQL fundamentals
  • Basic Python knowledge
  • Understanding of data processing concepts

Pro Tip: Strong SQL and cloud fundamentals significantly improve your ability to operationalize scalable data engineering workflows.

You will learn how to build, govern, optimize, and operationalize enterprise-grade data engineering pipelines using Databricks.

The course emphasizes practical skills required for modern analytics and AI infrastructure environments.

Skills gained:

  • Building scalable ETL pipelines
  • Managing Delta Lake tables
  • Using Auto Loader and Structured Streaming
  • Implementing Unity Catalog governance
  • Orchestrating production workflows
  • Monitoring and troubleshooting pipelines

Modern data engineering increasingly combines analytics, governance, streaming, and AI infrastructure operations.

Pro Tip: Data governance and operational monitoring skills are becoming just as important as pipeline development itself.

The Databricks Lakehouse Platform combines data lakes and data warehouses into a unified architecture for analytics and AI.

The platform enables organizations to manage data engineering, analytics, machine learning, and AI workloads within a single scalable environment.

Key capabilities include:

  • Data engineering pipelines
  • SQL analytics and BI
  • Machine learning workflows
  • AI and generative AI operations
  • Governance with Unity Catalog

Databricks positions the Lakehouse architecture as a modern alternative to traditional fragmented analytics stacks.

Pro Tip: Lakehouse architectures are becoming foundational for enterprise AI and analytics modernization strategies.

Yes, this course prepares learners for the Databricks Certified Data Engineer Associate certification.

The certification validates practical data engineering skills including pipeline orchestration, governance, streaming, deployment, and Lakehouse operations.

Exam areas include:

  • Lakehouse architecture fundamentals
  • ETL and declarative pipelines
  • Structured Streaming and Auto Loader
  • Unity Catalog governance
  • Workflow orchestration
  • Spark SQL and data processing

Pro Tip: Hands-on implementation practice is essential because Databricks certifications focus heavily on practical operational workflows.

Data Engineering with Databricks supports modern data engineering, AI infrastructure, and cloud analytics-related roles.

Organizations increasingly require professionals who can operationalize scalable data platforms supporting analytics, AI, and machine learning workloads.

Relevant roles:

  • Data Engineer
  • Cloud Data Engineer
  • Analytics Engineer
  • AI Data Infrastructure Engineer
  • Big Data Engineer
  • Lakehouse Platform Engineer

Databricks and Lakehouse expertise continue growing in demand as organizations modernize enterprise analytics and AI infrastructure.

Pro Tip: Combining Databricks expertise with cloud platforms, AI engineering, or governance knowledge can significantly strengthen your long-term career opportunities.

Speak to a Training Consultant

All courses are HRD Claimable.
Get in touch with our team via the form or WhatsApp us on +6011-5119 6631

Preferred mode of training
Checkboxes