A. Introduction

Creating a data pipeline is one thing; bringing it into production is another. This is especially true for a modern data pipeline in which multiple services are used for advanced analytics. Examples are transforming unstructured data to structured data, training of ML models and embedding OCR. Integration of multiple services can be complicated and deployment to production has to be controlled. In this blog, an example project is provided as follows:

• 1. Setup an Azure DevOps project for contineous deployment
• 2. Deploy Azure resources of data pipeline using infrastructure as code
• 3. Run and monitor data pipeline

The code from…

Introduction

The Microsoft identity platform is key to secure access to your web app. Users can authenticate to your app using their Azure AD identity or social accounts. The authorization model can be used to grant permissions to your backend app or standard APIs like Microsoft Graph. In this blog, a web application is discussed that does the following:

• 1. Azure AD login with user role: basic or user role: premium
• 2. Access to MS Graph using delegated permissions of signed-in user
• 3. Access to backend using application permissions and app role

The code of the project can be found here…

1. Introduction

Python Flask is a popular tool to create web applications. Using Azure AD, users can authenticate to the REST APIs and retrieve data from Azure SQL. In this blog, a sample Python web application is created as follows:

• 1a: User logs in to web app and acquires a token
• 1b: User calls a REST API to request a dataset
• 2: Web app uses claims in token to verify user access to dataset
• 3: Web app retrieves data from Azure SQL. Web app can be configured such that either the a) managed identity of the app or b) signed-in user identity…

0. Introduction

Azure Cosmos DB is a fully managed multi-database service. It enables you to build highly responsive applications worldwide. As part of Cosmos DB, Gremlin is supported for graph databases. Since Cosmos DB is optimized for fast processing (OLTP), traversal limits may apply for heavy analytic workloads (OLAP). In that case, Azure Databricks and GraphFrames can be used as an alternative to do advanced analytics, see also architecture below.

In the remaining of blog, the following is done:

• OLTP: write graph data to Cosmos DB using the Gremlin API and Python
• OLAP: read data from Cosmos DB, analyze data in Azure…

A. Introduction

Selenium is the standard tool for automated web browser testing. On top of that, Selenium is a popular tool for web scraping. When creating a web scraper in Azure, Azure Functions is a logical candidate to run your code in. However, the default Azure Functions image does not contain the dependencies that Selenium requires. In this blog, a web scraper in Azure Functions is created as follows:

• Create and deploy docker image as Azure Function with Selenium
• Scrape websites periodically and store results

The architecture of web scraper is depicted below.

In the remaining the steps are discussed to deploy…

1. Azure Storage backup - Introduction

Azure Storage always stores multiple copies of your data. When Geo-redundant Storage (GRS) is used, it is also replicated to the paired region. This way, GRS prevents that data is lost in case of disaster. However, GRS cannot prevent data loss when application errors corrupt data. Corrupted data is then just replicated to other zones/regions. In that case, a backup is needed to restore your data. Two backup strategies are as follows:

• Snapshot creation: In case a blob is added or modified, a snapshot is created from the current situation. Because of the nature of blobs, this is an efficient…

Secure Azure Functions with Azure AD, Key Vault and VNETs. Then connect to Azure SQL using firewall rules and Managed Identity of Function.

A. Azure Functions Security - Introduction

Azure Functions is a popular tool to create small snippets of code that can execute simple tasks. Azure Functions can be triggered using queue triggers, HTTP triggers or time triggers. A typical pattern of an Azure Function is as follows:

• Init: Retrieve state from Storage Account
• Request: Endpoint is called by another application/user
• Processing: Data is processed using other Azure resources
• Response: Result is replied to caller

Pattern is depicted below, in which data is retrieved from…

1. Introduction

Azure Data Factory (ADFv2) is a popular tool to orchestrate data ingestion from on-premises to cloud. In every ADFv2 pipeline, security is an important topic. Common security aspects are the following:

• Azure Active Directory (AAD) access control to data and endpoints
• Managed Identity (MI) to prevent key management processes
• Virtual Network (VNET) isolation of data and endpoints

In the remainder of this blog, it is discussed how an ADFv2 pipeline can be secured using AAD, MI, VNETs and firewall rules. For more details on security of Azure Functions, see my other blog. …