DP-3027: Implement a data engineering solution with Azure Databricks

At the end of this module, you’re able to:

• Understand Spark structured streaming.
• Some techniques to optimize structured streaming.
• How to handle late arriving or out of order events.
• How to set up real-time-sources for incremental processing.

At the end of this module, you’re able to:

• Use Event driven architectures with Delta Live Tables
• Ingest streaming data
• Achieve Data consistency and reliability
• Scale streaming workloads with Delta Live Tables

In this module, you learn how to:

• Use serverless compute and parallelism with Delta live tables
• Perform cost based optimization and query performance
• Use Change Data Capture (CDC)
• Apply enhanced autoscaling capabilities
• Implement Observability and enhance data quality metric

In this module, you learn how to:

• Implement version control and Git integration.
• Perform unit testing and integration testing.
• Maintain environment and configuration management.
• Implement rollback and roll-forward strategies.

In this module, you learn how to:

• Implement job scheduling and automation.
• Optimize workflows with parameters.
• Handle dependency management.
• Implement error handling and retry mechanisms.
• Explore best practices and guidelines

At the end of this module, you’re able to:

• Implement data encryption techniques
• Manage access controls
• Implement data masking and anonymization
• Use compliance frameworks and secure data sharing
• Use data lineage and metadata management
• Roll out governance automation

In this module, you’ll learn how to:

• Create and configure SQL Warehouses in Azure Databricks.
• Create databases and tables.
• Create queries and dashboards.

In this module, you’ll learn how to:

• Describe how Azure Databricks notebooks can be run in a pipeline.
• Create an Azure Data Factory linked service for Azure Databricks.
• Use a Notebook activity in a pipeline.
• Pass parameters to a notebook.