Setup secret-less access from Azure Data Factory to AWS S3 — 1. Introduction A common challenge for developers is the management of credentials to secure communication between services. Azure Managed Identities eliminate the need for developers to manage these credentials. Applications can use managed identities to obtain Azure AD tokens to access resources in Azure. …

Learn to use Synapse exfiltration protection, private endpoints and Azure AD authentication to create a secure connection — 1. Introduction Azure Functions is a popular tool to create REST APIs. Teams can use APIs to expose their applications, which can then be consumed by other teams. A common pattern is to connect Synapse pipelines to Azure Functions, for instance, to run small computations provided by other teams, create metadata or…

Analysis on ingestion/consumption patterns including delta lake PoC — 1. Introduction A lot of companies consider setting up an Enterprise Data Lake. The idea is to store data in a centralized repository. The main stakeholders of the data lake are the following two entities: Data producer: Entity that ingests data to the data lake. This is typically the entity that does…

Using Azure DevOps and Azure Data Factory CI/CD utilities — 1. Introduction In software projects, DEV/TST/UAT environments are used to develop and test code. After that, code is released to the PRD environment. Azure Data Factory (ADF) projects follow the same phased approach. However, the following ADF challenges need to be taken care of: ADF project can only be deployed as a…

Using Databricks audit logs, Log Analytics and Azure Monitor — 1. Introduction Azure Databricks is a popular tool for enterprises to do data science. However, it shall be prevented that Databricks is used to exfiltrate data. For that purpose, a security baseline for Azure Databricks shall be implemented. …

Leverage snapshots, soft deletes and Azure Functions to build a point-in-time recovery solution on Azure Data Lake Storage. — TLTR: Clone project and run scripts in https://github.com/rebremer/azure-functions-datalake-recovery-pitr/blob/main/script.ps1 to deploy solution 1. Introduction A common fear in enterprise data lakes is that data gets corrupted because of application bugs or human errors. In that case, it may be required to do point-in-time recovery (PiTR) and to restore a snapshot. PiTR is not…

Using Azure Synapse, ADLSgen2, Python and Web Apps — 1. Introduction Synapse serverless SQL pools is a service to query data in data lakes. Key is that data can be accessed without the need to copy data into SQL tables. Typically, serverless pools are not used to serve external APIs. This is because external APIs require predictable and low latency response…

Learn how to build an MLOps pipeline with Azure API management to deploy models as a secure endpoint. — 1. Introduction Machine Learning Operations (MLOps) aims to deploy and maintain machine learning models in production. A common artifact of an MLOps pipeline is a realtime scoring endpoint that can be consumed by end user applications. Key is to secure this endpoint using network isolation and authentication. …

Learn to connect multiple Databricks workspaces to 1 central ADLSgen2 account — 1. Introduction TLTR: Clone this git project, set params and run 0_script.sh to deploy 1 ALDSgen2 hub and N Databricks spokes A data lake is a centralized repository of data that allows enterprises to create business value from data. Azure Databricks is a popular tool to analyze data and build data pipelines…