Leverage snapshots, soft deletes and Azure Functions to build a point-in-time recovery solution on Azure Data Lake Storage. — TLTR: Clone project and run scripts in https://github.com/rebremer/azure-functions-datalake-recovery-pitr/blob/main/script.ps1 to deploy solution 1. Introduction A common fear in enterprise data lakes is that data gets corrupted because of application bugs or human errors. In that case, it may be required to do point-in-time recovery (PiTR) and to restore a snapshot. PiTR is not…

Using Azure Synapse, ADLSgen2, Python and Web Apps — 1. Introduction Synapse serverless SQL pools is a service to query data in data lakes. Key is that data can be accessed without the need to copy data into SQL tables. Typically, serverless pools are not used to serve external APIs. This is because external APIs require predictable and low latency response…

Learn how to build an MLOps pipeline with Azure API management to deploy models as a secure endpoint. — 1. Introduction Machine Learning Operations (MLOps) aims to deploy and maintain machine learning models in production. A common artifact of an MLOps pipeline is a realtime scoring endpoint that can be consumed by end user applications. Key is to secure this endpoint using network isolation and authentication. …

Learn to connect multiple Databricks workspaces to 1 central ADLSgen2 account — 1. Introduction TLTR: Clone this git project, set params and run 0_script.sh to deploy 1 ALDSgen2 hub and N Databricks spokes A data lake is a centralized repository of data that allows enterprises to create business value from data. Azure Databricks is a popular tool to analyze data and build data pipelines…

Leverage Azure DevOps and pytest for Data Factory unit testing — 0. Introduction TLTR: Create Azure DevOps git project using azure-pipelines.yml, create build artifact, deploy ADFv2 and SQLDB bacpac, trigger pytest to do unit tests Unit testing is a software engineering practice that focuses on testing individual parts of code. In unit testing, the following best practices are applicable: Fast. Unit tests should…

Learn how to plan for disaster recovery for data services in Azure — 0. Introduction Disaster recovery aims for the continuation of IT systems after disruptive events. Data services are a vital part of every IT system and shall be protected against these events. Most Azure PaaS data services have service tiers that support zone redundancy. This implies that when disaster is limited to a…

Using Azure DevOps, Databricks Spark, Cosmos DB Gremlin API and Azure Data Factory — A. Introduction Creating a data pipeline is one thing; bringing it into production is another. This is especially true for a modern data pipeline in which multiple services are used for advanced analytics. Examples are transforming unstructured data to structured data, training of ML models and embedding OCR. Integration of multiple services…

Manage access to your app with identities, roles and permissions — Introduction The Microsoft identity platform is key to secure access to your web app. Users can authenticate to your app using their Azure AD identity or social accounts. The authorization model can be used to grant permissions to your backend app or standard APIs like Microsoft Graph. In this blog, a…

Learn to use identities and tokens in web apps and Azure SQL — 1. Introduction Python Flask is a popular tool to create web applications. Using Azure AD, users can authenticate to the REST APIs and retrieve data from Azure SQL. In this blog, a sample Python web application is created as follows: 1a: User logs in to web app and acquires a token 1b…