Handling Datamart Table Structure Changes (Agents)

This article outlines how Datamart table structure changes caused by agent configuration updates impact existing data. It highlights the differences between draft and approved agents and the user actions required to safely handle data retention and table recreation.

When Datamart Structure Changes

A Datamart structure change occurs when an implementation requires a new or updated Datamart column, for example:

• Agent-preserved column configuration changes

• New column added to the Datamart table

• Datamart table configuration changes (e.g., column name, label)

Agent Behavior

Non‑approved (Draft) Agents

For agents that are not yet submitted and approved (treated as draft agents):

• The Datamart table changes are automatically applied.

• The affected Datamart Tables are recreated.

• As part of recreation, all existing data in the changed Datamart table is cleared.

This default behavior does not apply to the Detected Items Datamart Table. As of Agents 1.7, whenever an agent's dimensions or measures change, the Detected Items table is automatically backed up instead of being cleared.

Approved Agents

For approved agents:

• The system does not automatically recreate the Datamart table.

• The user must manually remove the Datamart table so the system can recreate it based on the new structure.

• When the Datamart table is removed and recreated, all existing data in that Datamart table is cleared as a result of the removal.

This manual removal requirement applies to general Datamart tables that store agent output. It does not apply to the Detected Items Datamart Table, which is handled automatically for both draft and approved agents, as described below.

Automatic Backup of the DetectedItems Datamart Table

As of Agents 1.7, when an agent detects a structural change to its dimensions or measures, for example, after editing the Group By fields in the Definition step, the system no longer simply clears the Detected Items Datamart Table. Instead, it automatically backs up the existing data before recreating the table. This applies to both draft and approved agents and specifically protects the connection between existing Actions and the Detected Items used to create them, preventing Actions from becoming disconnected from their source data after a structure change.

The automatic backup process works as follows:

1. The system finds the highest runId in the current (old) DetectedItems table that has an associated actionId, meaning that run already produced an Action.

2. The system renames the current DetectedItems table to DetectedItems_<runId>, using the runId identified in the previous step, so the original data is preserved under a distinct table name.

3. The system creates a new DetectedItems table with the updated structure and continues its normal logic, adding newly detected items to the new table.

Rows moved into a backup table keep their original values. Any dimension that existed in the old structure but no longer exists in the new structure is preserved in a Previous Group By column, so the historical grouping context is not lost even though it is no longer part of the current table structure.

Before this change, Actions created prior to a structure change could appear disconnected from their originating agent, for example, showing Number of Items 0 on the Agent Action Review dashboard, once the Detected Items table was recreated and its data cleared. The automatic backup preserves this historical data so previously created Actions continue to display their original detected items and impact values.

Data Impact and User Responsibilities

Users must be aware of the following:

• When a Datamart table structure changes and the Datamart table is recreated, all existing data in that Datamart table is cleared.

• If users need the existing data for future reference, they must back up the data before the Datamart table is recreated or removed.

• Exception for Detected Items. As of Agents 1.7, the Detected Items Datamart Table is automatically backed up (renamed to DetectedItems_<runId>) whenever the agent's dimensions or measures change, so manual backup of this specific table is no longer required to preserve historical data linked to existing Actions. Manual backup guidance still applies to all other Datamart tables.

Recommended User Actions

For Non‑approved (Draft) Agents

• Draft agents are assumed to be in a design / test phase.

• No action is required from the user for structural Datamart Table changes, unless they want to keep the current data.

• If data retention is needed, users should export / backup the Datamart Table data before the change is applied.

• This guidance still applies to general Datamart tables. For the Detected Items table specifically, no manual export is needed to retain historical detected items, since the automatic backup mechanism described in Agent Behavior preserves that data under a DetectedItems_<RunId> table.

For Approved Agents

To safely handle Datamart Table structure changes for an approved agent:

1. Disable Automatic Execution

   • Remove any schedules from the agent so it does not run automatically via the scheduler during the transition.

2. Archive the Existing Agent

   • Duplicate the existing approved agent.

   • Use the duplicated agent as the new main agent going forward.

   • Keep the original agent as an archived agent (for reference or audit, as needed).

3. Handle Datamart Table Recreation

   • On the archived agent, manually remove the affected Datamart tables so that the system can recreate them with the updated structure.

   • Ensure that any required data backup has been performed before removal, as this process will clear all existing data in the Datamart table.

4. Detected Items Table – You do not need to manually remove or back up the Detected Items table as part of this process. When the agent's dimensions or measures change, the system automatically backs up the existing Detected Items data under a DetectedItems_<runId> table name before recreating the table with the new structure, so previously created Actions remain linked to their original detected items.

Summary

• Any Datamart table structure change (columns or config) results in Datamart table recreation and data clearing.

• Draft agents

  • Changes are applied automatically.

  • Data is cleared on recreation.

• Approved agents

  • User must manually remove Datamart tables.

  • Data is cleared on removal.

  • Users are responsible for backing up data they want to retain before Datamart Table removal or recreation.

• Detected Items Datamart Table (Agents 1.7+)

  • Exempt from the manual backup requirements above.

  • When an agent's dimensions or measures change, the table is automatically backed up (renamed to DetectedItems_<runId>) and recreated with the new structure, preserving old data, including a Previous Group By value for any dimension that no longer exists, so existing Actions do not become disconnected from their detected items.