Aim of This Article
Explains how to create a Data Download in PlatformManager that writes into an existing Databricks (Unity Catalog) table instead of files. A Data Download to Databricks uses the same entity, filter, schedule and notifications as a file download. The download maps Pricefx fields to the columns of the table, and file options are not available.
Navigation: Account > Partitions > {PartitionName} > Data Management > Data Downloads
Prerequisites
A Data Download to Databricks needs the following:
-
The account has the use-databricks-source feature flag. Databricks uploads use the same flag.
-
The account's IaaS configuration uses IntegrationManager (IM) 8.3.0 or newer.
-
You have the Partition Data Management – edit permission.
-
A Databricks connection exists on the partition, and the connection has a Staging volume. See How to Create Databricks Connection.
-
The service principal of the Databricks connection has the privileges in the table below. A privilege granted only through a group nested in another group counts as missing.
-
The target table exists in Databricks. Pricefx does not create tables.
|
Databricks object |
Privileges for a Data Download to Databricks |
|---|---|
|
Catalog and schema of the table |
|
|
Table |
|
|
Catalog and schema of the staging volume |
|
|
Staging volume |
|
Choose a Write Mode for a Data Download to Databricks
The Write mode of a Data Download to Databricks sets how each run writes into the table. You select it in Step 2. You can change it after you save the download.
|
Write mode |
Effect and requirements |
|---|---|
|
Overwrite |
The default. Each run replaces all rows of the table with the exported rows. A run that exports no rows leaves the table unchanged. Incremental Export is not available. |
|
Append |
Each run inserts the exported rows into the table. Existing rows stay. Incremental Export is available. Append is not idempotent: when Pricefx repeats a write, the same rows can land in the table twice. Where duplicates matter, use Merge. |
|
Merge |
Pricefx updates a row with the same Merge keys values and inserts other rows. The table must be a Delta table. Select at least one merge key. Incremental Export is available. |
Create Data Download to Databricks
A Data Download to Databricks can be a Manual Download or an Automated Download. Two wizard steps have different names in the two types:
|
Step |
Manual Download |
Automated Download |
|---|---|---|
|
3 |
Fields |
Data Mapping |
|
5 |
Advanced Options |
Additional Options |
Step 1: General Settings of Data Download to Databricks
To start a Data Download to Databricks:
-
Go to Account > Partitions > {PartitionName} > Data Management > Data Downloads.
-
Click New Data Download, then select Manual Download or Automated Download.
-
Enter a Name.
-
Select the Entity Type: the Pricefx entity to export.
-
If Entity Name appears, select the Pricefx table to export.
-
Click Continue.
Step 2: Destination Type of Data Download to Databricks
-
In Export to, select Databricks Table.
-
In the Destination table section, select the Connection.
If the connection has no staging volume, the form shows "To export to Databricks, the connection {name} needs a staging volume. Pricefx stages the exported data there before it loads it into the table."
-
If the staging volume message appears, click Edit connection {name}.
The connection opens in a new browser tab. The Data Download form keeps your input.
-
In the connection tab, add the Staging volume and save the connection.
-
In the Data Download form, click Reload connections.
The staging volume message disappears.
-
Select the Catalog, the Schema, and the Table. If a table is missing from the list, click Refresh tables.
The Table list contains only managed and external tables. The list does not show views or foreign tables.
Overwrite, the default write mode, replaces all rows of the table on each run that exports data. To keep the existing rows, select Append or Merge.
-
Select the Write mode: Overwrite, Append, or Merge.
-
Click Continue.
Step 3: Map Fields of Data Download to Databricks
Step 3 of a Data Download to Databricks maps Pricefx Fields to Table Columns. The list of table columns comes from the selected table.
-
Check the rows that the form adds. The form adds a row for each table column whose name matches the name or label of a Pricefx field, ignoring case.
-
To fill another column, click Add Field. Then select the Pricefx field and the table column. The column type shows next to the column.
-
For the Merge write mode, select at least one column in Merge keys.
Select mapped columns whose combined values identify each exported row uniquely. Every key column must have a value in each exported row.
-
Click Continue.
The form checks the mapping against the table. An error blocks saving. Examples are a column mapped twice, or an unmapped required column without a default. A type conversion that can lose data shows a warning only, for example text into a number column.
Step 4: Filter Rows of Data Download to Databricks
The Filter step of a Data Download to Databricks is optional and works as for a file download. To export all rows, click Continue.
Step 5: Options of Data Download to Databricks
For a Manual Download to Databricks, the Advanced Options step shows only Maximum Data Batch Size and Incremental Export. For an Automated Download, the Additional Options step shows Download Schedule, then a section with the same two options and the notifications. File-format and file-output options are not available for Databricks destinations.
-
For an Automated Download, set the Download Schedule and the optional notifications as described in How to Create Automated Data Download.
-
In Maximum Data Batch Size, keep the default or enter a batch size.
-
Optional: For an incremental export with Append or Merge, select Download only data modified after a certain date. For a full export, leave the option clear and go to step 6.
With the Overwrite write mode, the form disables the option. Its tooltip reads "Not available with the Overwrite write mode. Choose Append or Merge to export only changed data." When you select Overwrite in Step 2, the form clears the option.
-
If you selected Download only data modified after a certain date, select a Data To Be Downloaded value. The values are Data modified since last download and Data modified after date in custom timestamp column. For how the two values differ, see the Incremental Export section of How to Create Automated Data Download.
The download overwrites the selected Pricefx attribute, including on records that the filter excludes. Select an attribute reserved for export timestamps, without business data.
-
If you selected Data modified after date in custom timestamp column, select the Custom Timestamp Column Name.
-
Click Run export for a Manual Download, or Save for an Automated Download.
Expected Result
The Data Download to Databricks appears in the Data Downloads list. To show only these downloads, filter the Destination column to Databricks Table.
Check the Result of a Data Download to Databricks
To start a saved Data Download to Databricks immediately, open its row menu in the Data Downloads list and select Run Download. This works for Manual and Automated Downloads.
To check the runs, open the row menu of the download and select History Of Runs:
-
Processed in the Status column marks a successful run.
-
The File Name column shows the full name of the destination table: the catalog, schema and table names, separated by dots.
-
Downloaded Records shows the number of rows that the run wrote into the table.
-
Error in the Status column marks a failed run. To read the failure details, hover over the status badge.
Before each run, Pricefx reads the connection and the current table definition and checks the saved mapping against the table. An error stops the run before Pricefx sends any data. Table errors start with "The Databricks destination is no longer valid:". Other errors name the connection or the staging volume. The run checks only that the connection has a staging volume. The run does not check that the volume exists or that the service principal has its privileges, so the write in Databricks can still fail.
Fix a Blocked Save or Run of a Data Download to Databricks
A blocking Databricks error on the form stops Save and Run export. When you click either button, the form shows "You cannot save or export this download. Fix the Databricks errors shown on the form first." The errors stay on the form. Fix them and click again.
|
Symptom |
Cause |
Resolution |
|---|---|---|
|
The form says that the connection needs a staging volume. |
The connection has no staging volume. |
Click Edit connection {name}, add the Staging volume, and save the connection. Then click Reload connections. |
|
A run fails with "To export to Databricks, add a staging volume (catalog.schema.volume) to the connection in Connections." |
Someone removed the staging volume from the connection after you saved the download. |
Add the Staging volume to the connection and run the download again. |
|
The form reports missing privileges for the principal of the connection. |
The service principal lacks a privilege on the table or the staging volume. |
In Databricks, grant the privileges from the Prerequisites to the service principal, or to a group that the service principal belongs to directly. |
|
The form shows "Write privileges could not be verified" or "The Databricks destination could not be checked. The check runs again when you save." |
PlatformManager could not read the grants or reach Databricks. |
You can still save. |
|
The form reports that the staging volume does not exist. |
The staging volume of the connection is not in Databricks. |
Create the volume in Databricks, or correct the Staging volume on the connection. |
|
The form reports that Merge needs a Delta table. |
The table is not a Delta table. |
Select Append or Overwrite, or select a Delta table. |
|
In History Of Runs, a Merge run shows Error, and the failure details start with |
Two exported rows have the same merge key values, or a key column is empty. |
Correct the data, the filter, or the Merge keys, and run the download again. |
|
The form shows "Exporting to Databricks requires IM 8.3.0 or newer; the account's IaaS configuration uses {version}". |
The IaaS configuration uses IntegrationManager older than 8.3.0. |
Upgrade the IaaS configuration to IntegrationManager 8.3.0 or newer. |
|
Save fails with "Exporting to Databricks is not enabled for this account". |
The account does not have the use-databricks-source feature flag. |
Request the feature flag for the account. |
|
A run fails with "Databricks download destinations are not enabled". |
Someone switched the feature flag off after you saved the download. |
Request the feature flag for the account. Pricefx keeps the download. |
|
A run fails with "The Databricks destination is no longer valid:" and a list of errors. |
The table changed after you saved the download. |
Fix the listed errors in Databricks or in the download. |
Edit Data Download to Databricks
You can change the Write mode, the mapping and the Merge keys of a Data Download to Databricks.
-
You cannot change the connection or the table after you save the download.
-
A download cannot switch between a file and a Databricks table. To change the destination, create a new download.
-
Data Downloads have no copy function, so you cannot copy a Data Download to Databricks.
-
A Data Download to Databricks has no available files to download and no Secure Shell (SSH) File Transfer Protocol (SFTP) users.
The edit form of a Data Download to Databricks does not offer columns added to the Databricks table after the download was saved. A save checks the table again and stores its columns when you change the entity, mapping, write mode or merge keys. A change of the Maximum Data Batch Size or the incremental export settings has the same effect. A save that changes only the name, filter, schedule or notifications keeps the stored columns. A new required column can block saving when you cannot select it. Then create a new download for the table, or undo the table change in Databricks.
To edit a Data Download to Databricks:
-
In the Data Downloads list, open the row menu of the download and select Edit.
-
To change the write mode, on the Destination Type tab, select Overwrite, Append, or Merge. Selecting Overwrite clears Download only data modified after a certain date.
-
To change the mapping, open the Fields tab (Data Mapping for an Automated Download) and update the Pricefx Fields to Table Columns mappings.
-
For Merge, select at least one mapped column in Merge keys.
-
Click Save.
Expected Result
The Data Downloads list shows the Data Download to Databricks, and the next run uses the new settings.
See Also
Related pages for a Data Download to Databricks: