Delta / Incremental Export

Overview

Exports only records modified since the last run. Uses a lastUpdateDate greaterThan filter with a dynamic timestamp value set via a message header.

Filter: filters/delta-filter.xml

The filter uses a dynamic value from a message header:

<filters>
    <filter id="deltaFilter">
        <and>
            <criterion fieldName="lastUpdateDate" operator="greaterThan" value="simple:${header.lastExportTimestamp}"/>
        </and>
    </filter>
</filters>

Route: routes/delta-export.xml

<routes xmlns="http://camel.apache.org/schema/spring">
    <route id="deltaExport">
        <from uri="scheduler://deltaExport?delay=3600000"/>
        <!-- Set the timestamp: now minus 1 hour in ISO-8601 format -->
        <setHeader headerName="lastExportTimestamp">
            <groovy>
                def cutoff = new Date(System.currentTimeMillis() - 3600000)
                return cutoff.format("yyyy-MM-dd'T'HH:mm:ss")
            </groovy>
        </setHeader>
        <to uri="pfx-api:fetch?objectType=PX&amp;filter=deltaFilter&amp;batchedMode=true&amp;batchSize=5000"/>
        <split>
            <simple>${body}</simple>
            <to uri="pfx-api:fetchIterator"/>
            <to uri="pfx-csv:marshal"/>
            <to uri="file:export?fileName=delta_${date:now:yyyyMMdd_HHmmss}.csv&amp;fileExist=Append"/>
        </split>
    </route>
</routes>

How It Works

1. A <groovy> expression computes the cutoff timestamp (one hour ago)

2. deltaFilter uses simple:${header.lastExportTimestamp} — the simple: prefix tells IM to evaluate the expression at query time

3. Only records with lastUpdateDate > cutoff are fetched

4. Each run creates a new timestamped file in the export/ directory

Common Pitfalls

• The simple: prefix in the filter value attribute is required — without it, the literal string ${header.lastExportTimestamp} is sent to Pricefx

• lastUpdateDate format must match Pricefx's expected ISO-8601 format: yyyy-MM-dd'T'HH:mm:ss

• For the very first run, use a far-past timestamp (e.g., 1970-01-01T00:00:00) to export all records