Whether it is an enterprise data warehouse migration, a big data re-platforming, or a greenfield implementation; each project traditionally starts with a proof of concept. 

This Proof of Concept playbook provides a high-level methodology for planning, preparing, and running an effective proof of concept project. An effective proof of concept validates the fact that certain concepts have the potential for a real-world production application. The overall objective of a proof of concept is to validate potential solutions to technical problems, such as how systems can be integrated or how results can be achieved through a specific configuration. 

This blog post aims at providing you with a summary of the Azure Synapse Analytics Proof of Concept Playbook. Let’s look at who this e-book is targeted at first:

• Technical experts planning their own in-house Azure Synapse proof of concept project 
• Business owners who will be part of the execution or evaluation of an Azure Synapse proof of concept project 
• Anyone looking to learn more about data warehousing proof of concept projects  

Some of the topics that this e-book will cover include: 

• Guidance on what makes an effective proof of concept
• Guidance on how to make valid comparisons between systems
• Guidance on the technical aspects of running an Azure Synapse proof of concept
• A road map to relevant technical content from Azure Synapse
• Guidance on how to evaluate proof of concept results to back business decisions
• Guidance on how to find additional help

Data Warehousing with Dedicated SQL Pool 

Preparing for your proof of concept 

Before going through the process of creating Goals for your Azure Synapse Analytics POC it is worth taking some time to understand the service’s capabilities and how they might apply to your POC. To make the most of your POC execution time, read about it in the Service Overview. Additionally, it is worth reading through the Azure Synapse SQL Pools Architectural Overview to familiarize yourself with how SQL pools separate compute and storage to provide industry-leading performance. 

Identify sponsors and potential blockers 

Now that you are familiar with Azure Synapse. It is time to make sure that your proof of concept has the necessary backing and will not hit any roadblocks.  

Now is the time to: 

• Identify any restrictions or guidelines that your organization has about moving data to the cloud.
• Identify executive and business sponsorship for a cloud-based data warehouse project.
• Verify that your workload is appropriate for Azure Synapse.

Setting your timeline 

A proof of concept is a scoped, time-bounded exercise with specific, measurable goals and metrics of success. Ideally, it should have some basis in business reality so that the results are meaningful. In our experience, proof of concepts has the best outcome when they are timeboxed to two weeks. This provides enough time for work to be completed without the burden of too many use cases and complex test matrices. 

Data Lake Exploration with Serverless SQL Pool 

Preparing for your proof of concept 

A proof of concept (PoC) project can help you make an informed business decision about implementing a big data and advanced analytics environment on a cloud-based platform, leveraging the serverless SQL pool functionality in Azure Synapse. 

If you need to explore data in the data lake, gain insights from it or optimize your existing data transformation pipeline, you can benefit from using the serverless SQL pool resource.  

It is suitable for the following scenarios: 

• Basic discovery and exploration – Quickly reason about the data in various formats (Parquet, CSV, JSON) in your data lake, so you can plan how to extract insights from it.
• Logical data warehouse – Provide a relational abstraction on top of raw or disparate data without relocating and transforming data, allowing always up-to-date view of your data.
• Data transformation – Simple, scalable, and performant way to transform data in the lake using T-SQL, so it can be fed to BI and other tools, or loaded into a relational data store (dedicated SQL pools in Azure Synapse, Azure SQL Database, etc.).

Big Data Analytics with Apache Spark Pool 

Preparing for your proof of concept 

A proof of concept (PoC) project can help you make an informed business decision about migrating your on-premises big data and advanced analytics platform to a cloud-based big data and advanced analytics service, leveraging Azure Synapse Analytics for Apache Spark workloads. 

A Spark POC project will identify your key goals and business drivers that cloud-based big data and advanced analytics platforms must support and will test key metrics and prove key behaviours that are critical to the success of your data engineering, machine learning model building and training etc. needs. A proof of concept is a quickly executed project that focuses on key questions and is not designed to be deployed to a production environment but is designed to execute quick tests and then be discarded. 

Develop an understanding of these key concepts:  

• Apache Spark and its distributed architecture
• Concepts of RDD and partitions (in-memory and physical) in Spark
• Azure Synapse workspace, different compute engines, pipeline, and monitoring
• Separation of computing and storage in Spark pool
• Authentication and Authorization in Azure Synapse
• Native connectors to integrate with dedicated SQL pool in Azure Synapse, Azure Cosmos DB etc.

Conclusion 

An effective data proof of concept project starts with a well-designed plan and concludes with measurable test results that can be used to make data-backed business decisions. 

Azure Synapse is a limitless cloud-based analytics service with unmatched time to an insight that accelerates the delivery of BI, AI, and intelligent applications for the enterprise. You will gain many benefits from Azure Synapse, including performance, speed, improved security and compliance, elasticity, managed infrastructure, scalability, and cost savings. 

This guide has provided a high-level methodology to prepare for, and execute a proof of concept to help you use Azure Synapse as a data warehouse with a dedicated SQL pool, a data lake with  serverless SQL pool, and/or for big data analytics with Apache Spark pool. 

Invent with purpose, realize cost savings, and make your organization more efficient with Microsoft Azure’s open and flexible cloud computing platform.