Aim of This Article
Explains how to create a Databricks connection on a partition. The connection lets an Automated Data Upload read tables from a Databricks workspace. In PlatformManager, a Data Download can also use the connection to write to a Databricks table. For this export, the connection needs a Staging volume.
Related sections: Partition Connections (Reference Manual), Databricks Integration (overview), How to Create Automated Data Upload from Databricks, How to Create Data Download to Databricks.
Prerequisites
-
The
use-databricks-sourcefeature flag is enabled for the account. -
You have the
Partition Data Management – editpermission. -
In Databricks, you have:
-
A service principal with an OAuth secret. A personal access token does not work.
-
A SQL warehouse on which the service principal has
CAN USE. -
USE CATALOG,USE SCHEMA, andSELECTon the tables you want to load. -
For a Data Download to Databricks, a Unity Catalog volume for staging, and these privileges for the service principal:
-
USE CATALOGandUSE SCHEMAon the catalog and schema of the target table, andSELECTandMODIFYon the table. -
USE CATALOGandUSE SCHEMAon the catalog and schema of the staging volume, andREAD VOLUMEandWRITE VOLUMEon the volume.
-
-
Grant read privileges only on the tables that Pricefx should read. Grant write privileges only on the target tables and the staging volume. The upload wizard lists only the catalogs, schemas, and tables that the service principal can see. PlatformManager reports an export privilege granted only through a nested group as missing.
Steps
-
Go to Account > Partitions > {PartitionName} > Data Management > Connections tab.
-
Click Create Connection.
-
Enter a Name for the connection.
-
In Connection Type, select Databricks.
-
Fill in the Databricks fields:
-
Workspace host: the host name of your workspace URL without
https://, port, path, or trailing slash. Examples:-
AWS:
dbc-xxxxxxxx-xxxx.cloud.databricks.com. -
Microsoft Azure:
adb-1234567890123456.7.azuredatabricks.net. -
Google Cloud:
1234567890123456.7.gcp.databricks.com.
-
-
Warehouse ID: in Databricks, go to SQL Warehouses > {warehouse} > Connection details. Letters and digits only.
-
OAuth client ID: the Application ID of the service principal, under Settings > Identity and access > Service principals.
-
OAuth client secret: an OAuth secret generated for that service principal.
-
Default catalog and Default schema (optional): used only by Test connection.
-
Staging volume (optional): the Unity Catalog volume where Pricefx stages exported files, as catalog.schema.volume, for example main.sales.stage. Leave it empty for a connection that only uploads. Each part starts with a letter or underscore and contains only letters, numbers, and underscores. Otherwise the form shows "Use the form catalog.schema.volume with letters, numbers and underscores".
-
-
Click Test connection and wait for the result.
Test connection does not check the Staging volume or the export privileges. PlatformManager checks them when you save a Data Download to Databricks. A privilege error names the service principal and the missing privileges.
-
Click Save.
The connection appears on the Connections tab with type Databricks.
Test Connection Results
Test connection gets a token, checks the warehouse, and runs SELECT 1. It can take up to about 30 seconds.
|
Message |
What to do |
|---|---|
|
"Connected. Warehouse {name} is running." |
Nothing. The connection works. |
|
"The OAuth client ID or secret was rejected." |
Check the client ID and secret of the service principal. |
|
"Could not reach the workspace's token endpoint." |
Try again. This is usually temporary. |
|
"That doesn't look like a valid workspace hostname..." |
Remove |
|
"Could not reach the workspace host." |
Check the host name. |
|
"The service principal cannot run queries on this warehouse." |
Grant the service principal |
|
"The SQL warehouse is not running." |
Start the warehouse and test again. |
|
"No warehouse matches that warehouse ID." |
Copy the ID from the warehouse Connection details tab. |
|
"The test query failed." |
Check the default catalog and schema. |
Editing and Deleting a Connection
-
To edit, use Edit in the row menu. The connection type cannot change. Leave OAuth client secret blank to keep the current secret.
-
To add a staging volume to an existing Databricks connection, use Edit in the row menu, enter the Staging volume, and click Save. A Data Download can export to Databricks only through a connection with a staging volume. For the export errors and how to fix them, see How to Create Data Download to Databricks.
-
To delete, use Delete in the row menu. You cannot delete a connection that a Data Upload or Data Download still uses.