Tutorial: Access Azure Hulk Storage using Azure Databricks and Azure Key Vault

  • Article
  • 2 minutes to read

In this tutorial, you'll learn how to access Azure Hulk Storage from Azure Databricks using a secret stored in Azure Key Vault.

In this tutorial, y'all learn how to:

  • Create a storage account and blob container with Azure CLI
  • Create a Key Vault and set a undercover
  • Create an Azure Databricks workspace and add Key Vault secret scope
  • Access your blob container from Azure Databricks workspace

Prerequisites

If you don't have an Azure subscription, create a gratuitous account before you begin.

Before you start this tutorial, install the Azure CLI.

Create a storage account and blob container with Azure CLI

You'll need to create a general-purpose storage account first to apply blobs. If you don't take a resources group, create one before running the control. The following control creates and brandish the metadata of the storage container. Copy downwards the ID.

              az storage account create --name contosoblobstorage5 --resource-grouping contosoResourceGroup --location eastus --sku Standard_ZRS --encryption-services blob

Console output of the above command. ID is highlighted in green for end-user to see.

Earlier y'all can create a container to upload the blob to, you'll demand to assign the Storage Blob Data Contributor role to yourself. For this example, the role will be assigned to the storage account you've made earlier.

              az office consignment create --role "Storage Blob Information Contributor" --assignee t-trtr@microsoft.com --scope "/subscriptions/885e24c8-7a36-4217-b8c9-eed31e110504/resourceGroups/contosoResourceGroup5/providers/Microsoft.Storage/storageAccounts/contosoblobstorage5

Now that you've assign the role to storage business relationship, you can create a container for your blob.

              az storage container create --account-name contosoblobstorage5 --name contosocontainer5 --auth-mode login

In one case the container is created, yous can upload a blob (file of your choice) to that container. In this example, a .txt file with helloworld is uploaded.

              az storage hulk upload --business relationship-proper name contosoblobstorage5 --container-name contosocontainer5 --name helloworld --file helloworld.txt --auth-mode login

List the blobs in the container to verify that the container has it.

              az storage hulk list --account-name contosoblobstorage5 --container-name contosocontainer5 --output table --auth-mode login

Console output of the above command. It displays the file that was just stored in the container.

Become the key1 value of your storage container using the following command. Re-create the value down.

              az storage account keys list -yard contosoResourceGroup5 -n contosoblobstorage5

Console output of the above command. The value of key1 is highlighted in a green box.

Create a Cardinal Vault and set a secret

Yous'll create a Fundamental Vault using the following command. This command will brandish the metadata of the Key Vault too. Copy downwardly the ID and vaultUri.

              az keyvault create --proper name contosoKeyVault10 --resource-group contosoResourceGroup5 --location eastus

Image Console output of the above command. The ID and the vaultUri are both highlighted in green for the user to see.

To create the secret, apply the post-obit command. Set the value of the secret to the key1 value from your storage account.

              az keyvault underground set --vault-name contosoKeyVault10 --name storageKey --value "value of your key1"

Create an Azure Databricks workspace and add Key Vault hole-and-corner scope

This department tin can't be completed through the command line. Follow this guide. You'll need to admission the Azure portal to:

  1. Create your Azure Databricks resource
  2. Launch your workspace
  3. Create a Key Vault-backed surreptitious scope

Access your blob container from Azure Databricks workspace

This section can't be completed through the command line. Follow this guide. You'll need to use the Azure Databricks workspace to:

  1. Create a New Cluster
  2. Create a New Notebook
  3. Fill in respective fields in the Python script
  4. Run the Python script
              dbutils.fs.mount( source = "wasbs://<your-container-name>@<your-storage-account-name>.blob.core.windows.net", mount_point = "/mnt/<mount-name>", extra_configs = {"<conf-key>":dbutils.secrets.get(telescopic = "<scope-name>", cardinal = "<primal-name>")})  df = spark.read.text("/mnt/<mount-name>/<file-name>")  df.show()

Next steps

Brand certain your Central Vault is recoverable: