Datamart Operations Reference | Pricefx Knowledge Base

Summary: Reference for all datamart/data source operations -- load, flush, truncate, calculate, refresh, and data mirroring from Products/Customers.

Overview

Datamart operations allow you to manage Pricefx Data Marts (DM), Data Sources (DMDS), and Data Feeds (DMF). These elements are defined in the pfx namespace (http://www.pricefx.eu/schema/pfx) and are used within integration route configurations.

dmLoad

Loads data into a datamart data source. This is the primary element for pushing external data (e.g., from CSV or database) into a Pricefx Data Source.

Attributes

Attribute	Type	Required	Description
`id`	`xsd:ID`	Yes	Unique bean identifier for this element.
`dsUniqueName`	`xsd:string`	Yes	The unique name of the target Data Source (DMDS) in Pricefx.
`converterStrategyType`	`xsd:string`	No	Strategy for type conversion during load (e.g., handling numbers, dates).
`mapper`	`xsd:string`	Yes	Reference to a `mapper` or `loadMapper` bean that defines field mappings.
`businessKeys`	`xsd:string`	No	Comma-separated list of business key field names for upsert behavior.
`partitionPfxApi`	`xsd:string`	No	Target partition override. Used when loading data into a partition different from the default connection.

Example: Load CSV into a Data Source

XML

<pfx:mapper id="salesDataMapper" type="loaddata">
    <pfx:body in="ProductID"    out="attribute1"/>
    <pfx:body in="Region"       out="attribute2"/>
    <pfx:body in="Revenue"      out="attribute3" converter="StringToNumber"/>
    <pfx:body in="TransDate"    out="attribute4" converter="StringToDate" format="yyyy-MM-dd"/>
</pfx:mapper>

<pfx:dmLoad id="loadSalesData"
            dsUniqueName="SalesData_DS"
            mapper="salesDataMapper"/>

Example: Load with Business Keys (upsert)

XML

<pfx:dmLoad id="loadSalesDataUpsert"
            dsUniqueName="SalesData_DS"
            mapper="salesDataMapper"
            businessKeys="attribute1,attribute2"/>

Example: Load into a Specific Partition

XML

<pfx:dmLoad id="loadSalesDataPartition"
            dsUniqueName="SalesData_DS"
            mapper="salesDataMapper"
            partitionPfxApi="otherPartitionConnection"/>

dmFlush

Flushes (submits) data from a Data Source into a Data Feed. After loading data into a DMDS with dmLoad, you must flush it so the data becomes available in the associated DMF.

Attributes

Attribute	Type	Required	Description
`id`	`xsd:ID`	Yes	Unique bean identifier.
`dataSourceName`	`xsd:string`	Yes	Name of the Data Source (DMDS) to flush.
`dataFeedName`	`xsd:string`	Yes	Name of the target Data Feed (DMF) to flush into.
`incrementalDate`	`xsd:string`	No	Date expression for incremental flush. Only data newer than this date will be flushed. Supports Camel Simple expressions.

Example: Flush a Data Source to a Data Feed

XML

<pfx:dmFlush id="flushSalesData"
             dataSourceName="SalesData_DS"
             dataFeedName="SalesData_DMF"/>

Example: Incremental Flush

XML

<pfx:dmFlush id="flushSalesDataIncremental"
             dataSourceName="SalesData_DS"
             dataFeedName="SalesData_DMF"
             incrementalDate="${header.lastRunDate}"/>

dmTruncate

Truncates (deletes) data from a Data Source or Data Feed. Use this to clear existing data before a full reload.

Attributes

Attribute	Type	Required	Description
`id`	`xsd:ID`	Yes	Unique bean identifier.
`targetName`	`xsd:string`	Yes	Name of the DMDS or DMF to truncate.
`typedId`	`xsd:string`	No	Typed identifier of the target (e.g., `DMDS.12345`). Alternative to name-based targeting.
`incrementalDate`	`xsd:string`	No	Date expression for incremental truncation. Only data older than this date will be removed.
`dtoFilter`	`xsd:string`	No	Reference to a filter bean for conditional truncation. Only matching records are removed.

Example: Full Truncate of a Data Feed

XML

<pfx:dmTruncate id="truncateSalesDMF"
                targetName="SalesData_DMF"/>

Example: Truncate with Filter

XML

<pfx:filter id="oldDataFilter">
    <pfx:and>
        <pfx:criterion fieldName="attribute4" operator="lessThan" value="2025-01-01"/>
    </pfx:and>
</pfx:filter>

<pfx:dmTruncate id="truncateOldSales"
                targetName="SalesData_DMF"
                dtoFilter="oldDataFilter"/>

Example: Incremental Truncate

XML

<pfx:dmTruncate id="truncateIncremental"
                targetName="SalesData_DS"
                incrementalDate="${header.cutoffDate}"/>

dmCalculate

Triggers a datamart calculation. This runs the logic defined on a Datamart (DM) to process its Data Feeds and produce calculated results.

Attributes

Attribute	Type	Required	Description
`id`	`xsd:ID`	Yes	Unique bean identifier.
`typedId`	`xsd:string`	Yes	Typed identifier of the Datamart (e.g., `DM.12345`).
`targetName`	`xsd:string`	Yes	Name of the Datamart to calculate.
`incrementalDate`	`xsd:string`	No	Date expression for incremental calculation.

Example: Full Datamart Calculation

XML

<pfx:dmCalculate id="calculateSalesDM"
                 typedId="DM.12345"
                 targetName="SalesDatamart"/>

Example: Incremental Calculation

XML

<pfx:dmCalculate id="calculateSalesDMIncremental"
                 typedId="DM.12345"
                 targetName="SalesDatamart"
                 incrementalDate="${header.lastCalcDate}"/>

dmRefresh

Refreshes a datamart, triggering a full recalculation of all its feeds and logic. Typically used after data loads and flushes are complete.

Attributes

Attribute	Type	Required	Description
`id`	`xsd:ID`	Yes	Unique bean identifier.
`dataMartName`	`xsd:string`	No	Name of the Datamart to refresh. If omitted, refreshes the default datamart.
`incrementalDate`	`xsd:string`	No	Date expression for incremental refresh.

Example: Full Refresh

XML

<pfx:dmRefresh id="refreshSalesDM"
               dataMartName="SalesDatamart"/>

Example: Incremental Refresh

XML

<pfx:dmRefresh id="refreshSalesDMIncremental"
               dataMartName="SalesDatamart"
               incrementalDate="${header.lastRefreshDate}"/>

dmCustomers

Mirrors customer master data into the datamart. This loads customer data from the Pricefx Customer Master into the DM for use in datamart calculations and analytics.

Attributes

Attribute	Type	Required	Description
`id`	`xsd:ID`	Yes	Unique bean identifier.
`incrementalDate`	`xsd:string`	No	Date expression for incremental customer mirroring. Only customers modified after this date will be loaded.

Example: Full Customer Mirror

XML

<pfx:dmCustomers id="mirrorCustomers"/>

Example: Incremental Customer Mirror

XML

<pfx:dmCustomers id="mirrorCustomersIncremental"
                 incrementalDate="${header.lastCustomerSync}"/>

dmProducts

Mirrors product master data into the datamart. This loads product data from the Pricefx Product Master into the DM for use in datamart calculations and analytics.

Attributes

Attribute	Type	Required	Description
`id`	`xsd:ID`	Yes	Unique bean identifier.
`incrementalDate`	`xsd:string`	No	Date expression for incremental product mirroring. Only products modified after this date will be loaded.

Example: Full Product Mirror

XML

<pfx:dmProducts id="mirrorProducts"/>

Example: Incremental Product Mirror

XML

<pfx:dmProducts id="mirrorProductsIncremental"
                incrementalDate="${header.lastProductSync}"/>

Common Patterns

Full Datamart Reload Pipeline

A typical end-to-end pattern: truncate old data, load new data from CSV, flush to the feed, and refresh the datamart.

XML

<!-- 1. Define the mapper -->
<pfx:mapper id="salesMapper" type="loaddata">
    <pfx:body in="ProductID"    out="attribute1"/>
    <pfx:body in="Region"       out="attribute2"/>
    <pfx:body in="Revenue"      out="attribute3" converter="StringToNumber"/>
</pfx:mapper>

<!-- 2. Truncate the Data Feed -->
<pfx:dmTruncate id="truncateSales" targetName="SalesData_DMF"/>

<!-- 3. Load CSV into the Data Source -->
<pfx:dmLoad id="loadSales"
            dsUniqueName="SalesData_DS"
            mapper="salesMapper"/>

<!-- 4. Flush from Data Source to Data Feed -->
<pfx:dmFlush id="flushSales"
             dataSourceName="SalesData_DS"
             dataFeedName="SalesData_DMF"/>

<!-- 5. Refresh the Datamart -->
<pfx:dmRefresh id="refreshDM" dataMartName="SalesDatamart"/>

In the route, these would be called in sequence:

XML

<route id="fullDmReloadRoute">
    <from uri="direct:fullDmReload"/>
    <to uri="pfx-api:truncateSales"/>
    <to uri="pfx-csv:file:data/sales.csv?dataFormat=salesCsvFormat"/>
    <to uri="pfx-api:loadSales"/>
    <to uri="pfx-api:flushSales"/>
    <to uri="pfx-api:refreshDM"/>
</route>

Incremental Load with Customer/Product Mirror

XML

<pfx:dmCustomers id="syncCustomers" incrementalDate="${header.lastSync}"/>
<pfx:dmProducts  id="syncProducts"  incrementalDate="${header.lastSync}"/>

<pfx:dmLoad id="loadIncrementalSales"
            dsUniqueName="SalesData_DS"
            mapper="salesMapper"
            businessKeys="attribute1,attribute2"/>

<pfx:dmFlush id="flushIncrementalSales"
             dataSourceName="SalesData_DS"
             dataFeedName="SalesData_DMF"
             incrementalDate="${header.lastSync}"/>

<pfx:dmRefresh id="refreshDMIncremental"
               dataMartName="SalesDatamart"
               incrementalDate="${header.lastSync}"/>

Overview

dmLoad

Attributes

Example: Load CSV into a Data Source

Example: Load with Business Keys (upsert)

Example: Load into a Specific Partition

dmFlush

Attributes

Example: Flush a Data Source to a Data Feed

Example: Incremental Flush

dmTruncate

Attributes

Example: Full Truncate of a Data Feed

Example: Truncate with Filter

Example: Incremental Truncate

dmCalculate

Attributes

Example: Full Datamart Calculation

Example: Incremental Calculation

dmRefresh

Attributes

Example: Full Refresh

Example: Incremental Refresh

dmCustomers

Attributes

Example: Full Customer Mirror

Example: Incremental Customer Mirror

dmProducts

Attributes

Example: Full Product Mirror

Example: Incremental Product Mirror

Common Patterns

Full Datamart Reload Pipeline

Incremental Load with Customer/Product Mirror

See Also