How to Create Databricks Connection

Aim of This Article

Explains how to create a Databricks connection on a partition. The connection lets an Automated Data Upload read tables from a Databricks workspace. In PlatformManager, a Data Download can also use the connection to write to a Databricks table. For this export, the connection needs a Staging volume.

Related sections: Partition Connections (Reference Manual), Databricks Integration (overview), How to Create Automated Data Upload from Databricks, How to Create Data Download to Databricks.

Prerequisites

  • The use-databricks-source feature flag is enabled for the account.

  • You have the Partition Data Management – edit permission.

  • In Databricks, you have:

    • A service principal with an OAuth secret. A personal access token does not work.

    • A SQL warehouse on which the service principal has CAN USE.

    • USE CATALOG, USE SCHEMA, and SELECT on the tables you want to load.

    • For a Data Download to Databricks, a Unity Catalog volume for staging, and these privileges for the service principal:

      • USE CATALOG and USE SCHEMA on the catalog and schema of the target table, and SELECT and MODIFY on the table.

      • USE CATALOG and USE SCHEMA on the catalog and schema of the staging volume, and READ VOLUME and WRITE VOLUME on the volume.

Grant read privileges only on the tables that Pricefx should read. Grant write privileges only on the target tables and the staging volume. The upload wizard lists only the catalogs, schemas, and tables that the service principal can see. PlatformManager reports an export privilege granted only through a nested group as missing.

Steps

  1. Go to Account > Partitions > {PartitionName} > Data Management > Connections tab.

  2. Click Create Connection.

  3. Enter a Name for the connection.

  4. In Connection Type, select Databricks.

  5. Fill in the Databricks fields:

    • Workspace host: the host name of your workspace URL without https://, port, path, or trailing slash. Examples:

      • AWS: dbc-xxxxxxxx-xxxx.cloud.databricks.com.

      • Microsoft Azure: adb-1234567890123456.7.azuredatabricks.net.

      • Google Cloud: 1234567890123456.7.gcp.databricks.com.

    • Warehouse ID: in Databricks, go to SQL Warehouses > {warehouse} > Connection details. Letters and digits only.

    • OAuth client ID: the Application ID of the service principal, under Settings > Identity and access > Service principals.

    • OAuth client secret: an OAuth secret generated for that service principal.

    • Default catalog and Default schema (optional): used only by Test connection.

    • Staging volume (optional): the Unity Catalog volume where Pricefx stages exported files, as catalog.schema.volume, for example main.sales.stage. Leave it empty for a connection that only uploads. Each part starts with a letter or underscore and contains only letters, numbers, and underscores. Otherwise the form shows "Use the form catalog.schema.volume with letters, numbers and underscores".

  6. Click Test connection and wait for the result.

    Test connection does not check the Staging volume or the export privileges. PlatformManager checks them when you save a Data Download to Databricks. A privilege error names the service principal and the missing privileges.

  7. Click Save.

The connection appears on the Connections tab with type Databricks.

Test Connection Results

Test connection gets a token, checks the warehouse, and runs SELECT 1. It can take up to about 30 seconds.

Message

What to do

"Connected. Warehouse {name} is running."

Nothing. The connection works.

"The OAuth client ID or secret was rejected."

Check the client ID and secret of the service principal.

"Could not reach the workspace's token endpoint."

Try again. This is usually temporary.

"That doesn't look like a valid workspace hostname..."

Remove https://, a port, or a path from Workspace host. IP addresses and internal host names are not supported.

"Could not reach the workspace host."

Check the host name.

"The service principal cannot run queries on this warehouse."

Grant the service principal CAN USE on the warehouse.

"The SQL warehouse is not running."

Start the warehouse and test again.

"No warehouse matches that warehouse ID."

Copy the ID from the warehouse Connection details tab.

"The test query failed."

Check the default catalog and schema.

Editing and Deleting a Connection

  • To edit, use Edit in the row menu. The connection type cannot change. Leave OAuth client secret blank to keep the current secret.

  • To add a staging volume to an existing Databricks connection, use Edit in the row menu, enter the Staging volume, and click Save. A Data Download can export to Databricks only through a connection with a staging volume. For the export errors and how to fix them, see How to Create Data Download to Databricks.

  • To delete, use Delete in the row menu. You cannot delete a connection that a Data Upload or Data Download still uses.

See Also