Breadcrumbs

How to Create Databricks Connection

Aim of This Article

Explains how to create a Databricks connection on a partition. The connection lets an Automated Data Upload read tables from a Databricks workspace.

Related sections: Partition Connections (Reference Manual), How to Create Automated Data Upload from Databricks.

Prerequisites

  • The use-databricks-source feature flag is enabled for the account.

  • You have the Partition Data Management – edit permission.

  • In Databricks, you have:

    • A service principal with an OAuth secret. A personal access token does not work.

    • A SQL warehouse on which the service principal has CAN USE.

    • USE CATALOG, USE SCHEMA, and SELECT on the tables you want to load.

Steps

  1. Go to Account > Partitions > {PartitionName} > Data Management > Connections tab.

  2. Click Create Connection.

  3. Enter a Name for the connection.

  4. In Connection Type, select Databricks.

  5. Fill in the Databricks fields:

    • Workspace host: the host name of your workspace URL without https://, port, path, or trailing slash (for example, dbc-xxxxxxxx-xxxx.cloud.databricks.com).

    • Warehouse ID: in Databricks, go to SQL Warehouses > {warehouse} > Connection details. Letters and digits only.

    • OAuth client ID: the Application ID of the service principal, under Settings > Identity and access > Service principals.

    • OAuth client secret: an OAuth secret generated for that service principal.

    • Default catalog and Default schema (optional): used only by Test connection.

  6. Click Test connection and wait for the result.

  7. Click Save.

The connection appears on the Connections tab with type Databricks.

Test Connection Results

Test connection gets a token, checks the warehouse, and runs SELECT 1. It can take up to about 30 seconds.

Message

What to do

"Connected. Warehouse {name} is running."

Nothing. The connection works.

"The OAuth client ID or secret was rejected."

Check the client ID and secret of the service principal.

"Could not reach the workspace's token endpoint."

Try again. This is usually temporary.

"That doesn't look like a valid workspace hostname..."

Remove https://, a port, or a path from Workspace host. IP addresses and internal host names are not supported.

"Could not reach the workspace host."

Check the host name.

"The service principal cannot run queries on this warehouse."

Grant the service principal CAN USE on the warehouse.

"The SQL warehouse is not running."

Start the warehouse and test again.

"No warehouse matches that warehouse ID."

Copy the ID from the warehouse Connection details tab.

"The test query failed."

Check the default catalog and schema.

Editing and Deleting a Connection

  • To edit, use Edit in the row menu. The connection type cannot change. Leave OAuth client secret blank to keep the current secret.

  • To delete, use Delete in the row menu. You cannot delete a connection that a Data Upload or Data Download still uses.

Notes

  • PlatformManager stores the client secret encrypted and never shows it again.

  • A Databricks connection can only be a data source. It is not offered as a Data Download destination.

See Also