Learn to connect multiple Databricks workspaces to 1 central ADLSgen2 account

1. Introduction

  • Defense in depth: ADLSgen2 contains sensitive data and shall be secured using private endpoints and Azure AD (disabling access keys). …


Leverage Azure DevOps and pytest for Data Factory unit testing

0. Introduction

  • Fast. Unit tests should take very little time to run. Milliseconds.
  • Isolated. Unit tests are standalone, can be run in isolation
  • Repeatable. Running a unit test shall always return the same results
  • Self-Checking. Automatically detect whether a test passed or failed
  • Timely. No disproportionately long time to write code for unit test


Server in data center, image by Kvistholt Photography on Unsplash

Learn how to plan for disaster recovery for data services in Azure

0. Introduction

  • Regional disaster: Outage in multiple data centers in a region caused by a natural disaster, e.g…


Using Azure DevOps, Databricks Spark, Cosmos DB Gremlin API and Azure Data Factory

A. Introduction

  • 1. Setup an Azure DevOps project for contineous deployment
  • 2. Deploy Azure resources of data pipeline using infrastructure as code
  • 3. Run and monitor data pipeline


Manage access to your app with identities, roles and permissions

Introduction

  • 1. Azure AD login with user role: basic or user role: premium
  • 2. Access to MS Graph using delegated permissions of signed-in user
  • 3. Access to backend using application permissions and app role


Learn to use identities and tokens in web apps and Azure SQL

1. Introduction

  • 1a: User logs in to web app and acquires a token
  • 1b: User calls a REST API to request a dataset
  • 2: Web app uses claims in token to verify user access to dataset
  • 3: Web app retrieves data from Azure SQL. Web app can be configured such that either the a) managed identity of the app or b) signed-in user identity…


Learn to write graph data in Cosmos DB using Gremlin and then to read/analyze data in Azure Databricks with GraphFrames.

0. Introduction

0. Architecture (image by Author)
  • OLTP: write graph data to Cosmos DB using the Gremlin API and Python
  • OLAP: read data from Cosmos DB, analyze data in Azure…


Learn how to create an Azure Function using a custom docker image and it aSelenium web scraper in Python

A. Introduction

  • Create and deploy docker image as Azure Function with Selenium
  • Scrape websites periodically and store results


Learn how to automatically backup your data lake using blob snapshots and Data Factory incremental backups

1. Azure Storage backup - Introduction

  • Snapshot creation: In case a blob is added or modified, a snapshot is created from the current situation. Because of the nature of blobs, this is an efficient…


A. Azure Functions Security - Introduction

  • Init: Retrieve state from Storage Account
  • Request: Endpoint is called by another application/user
  • Processing: Data is processed using other Azure resources
  • Response: Result is replied to caller

René Bremer

Data Solution Architect @ Microsoft, working with Azure services as ADFv2, ADLSgen2, Azure DevOps, Databricks, Function Apps and SQL. Opinions here are mine.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store