pfx-api:fetchIterator

pfx-api:fetchIterator

Continue a batched fetch iteration. Used after an initial pfx-api:fetch call with batchedMode=true to retrieve subsequent batches of data.

Overview

The fetchIterator method is a companion to pfx-api:fetch when using batched mode. After the initial fetch call retrieves the first batch and stores iteration state in the exchange, fetchIterator retrieves each subsequent batch.

URI Format

pfx-api:fetchIterator

No additional parameters are needed -- fetchIterator reads all state (object type, filter, batch size, current offset) from the exchange properties set by the initial fetch call.

How It Works

1. Call pfx-api:fetch with batchedMode=true and batchSize=N -- this returns the first batch and stores iteration state in the exchange

2. Call pfx-api:fetchIterator repeatedly to get subsequent batches

3. When there are no more records, fetchIterator returns null or an empty list

4. Use a loop construct (Camel loop with doWhile) to iterate until completion

Examples

Basic Batched Fetch with Iterator

<route>
    <from uri="direct:fetchAll"/>
    <!-- First batch -->
    <to uri="pfx-api:fetch?objectType=DMDS&amp;dsUniqueName=MyDS&amp;filter=myFilter&amp;batchedMode=true&amp;batchSize=10000"/>
    <to uri="direct:processBatch"/>
    <!-- Subsequent batches -->
    <loop doWhile="true">
        <simple>${body} != null &amp;&amp; ${body.size()} > 0</simple>
        <to uri="pfx-api:fetchIterator"/>
        <to uri="direct:processBatch"/>
    </loop>
</route>

Batched Fetch with Split Processing

<route>
    <from uri="direct:fetchAndSplit"/>
    <to uri="pfx-api:fetch?objectType=P&amp;filter=allProducts&amp;batchedMode=true&amp;batchSize=5000"/>
    <split>
        <simple>${body}</simple>
        <to uri="direct:processRecord"/>
    </split>
    <loop doWhile="true">
        <simple>${body} != null &amp;&amp; ${body.size()} > 0</simple>
        <to uri="pfx-api:fetchIterator"/>
        <split>
            <simple>${body}</simple>
            <to uri="direct:processRecord"/>
        </split>
    </loop>
</route>

Batched Fetch to File

<route>
    <from uri="direct:exportToFile"/>
    <to uri="pfx-api:fetch?objectType=DMDS&amp;dsUniqueName=SalesData&amp;filter=exportFilter&amp;batchedMode=true&amp;batchSize=10000"/>
    <to uri="direct:appendToFile"/>
    <loop doWhile="true">
        <simple>${body} != null &amp;&amp; ${body.size()} > 0</simple>
        <to uri="pfx-api:fetchIterator"/>
        <to uri="direct:appendToFile"/>
    </loop>
</route>

Common Pitfalls

1. Calling fetchIterator without an initial batched fetch -- fetchIterator reads state from the exchange. If you call it without first calling fetch with batchedMode=true, it will fail because no iteration state exists.

2. Forgetting the loop termination condition -- Always check that the body is not null and not empty before continuing. Otherwise you may loop indefinitely.

3. Modifying exchange properties between calls -- fetchIterator depends on exchange properties set by the initial fetch. Do not clear or modify exchange properties between iterations.

4. Memory with split -- When using split inside the loop, remember that each batch is loaded into memory. The batch size should be chosen to balance memory usage and API call overhead. A typical range is 5,000 to 50,000 rows per batch.