Breadcrumbs

IntegrationManager 1.2.0


This document summarizes major improvements and fixes introduced in the Pricefx IntegrationManager release version.

Version

IntegrationManager 1.2.0

Release date

Sep 11, 2020


In this section:

Migration Steps

Dependency Changes

  • IM library dependency version 1.2.0
  • Spring boot version 2.2.7
  • Camel version 3.2.0

1. Camel Upgrade – Update Codebase

A new major version of Camel is used which resulted in major changes. 

Read Camel Migration Guides

Below are listed the official changes in Camel. Go through them and alter your code accordingly. 

Codebase Changelist

(warning) We strongly recommend to check your code against official guidelines and test changes properly before going live.

This changelist comes from our codebase changes:

Change Old Value New Value

Dependency xmllist was removed.

 xmllist

camel-jaxb
camel-jackson

Components json-to-list and rest-to-list were removed.

 json-to-list 
rest-to-list		 –

Exchanges method getOut() is deprecated.

 getOut()

getMessage()
Header name parameter renamed. <setHeader headerName="foo"> <setHeader name="foo">
Property name parameter renamed. <setProperty propertyName="foo"> <setProperty name="foo">
The http4 component has been renamed. http4  http
Simple language no longer supports constants. simple:Elasticity constant:Elasticity
Simple language function property renamed to exchangeProperty.
Source: https://camel.apache.org/components/latest/languages/simple-language.html		 simple:property.name simple:exchangeProperty.name
Quartz2 is now renamed to Quartz. <quartz2...> <quartz...>
Unmarshal ref tag now needs to be in a custom element. <unmarshal ref="xxx">... <unmarshal><custom ref="xxx"/>...

Other changes:

  • Source: The simple language uses ${body} placeholders for complex expressions or functions. (https://camel.apache.org/components/latest/languages/simple-language.html)
  • References to custom components must be prefixed.
    • All Bean methods (pfx:dsLoad, pfx:dsIntegrate, …) must be referenced as bean:xxx
    • All routes must be referenced as direct:xxx
    • Mocks are prefixed with mock:xxx
    • JdbcComponent are referrenced as jdbc:xxx
    • SqlComponent are referrenced as sql:xxx

    • Example of a Bean reference:

      Before 
      <pfx:dsLoad id="loadProduct" objectType="P" businessKeys="sku" mapper="dynamicPartitionMapper" />

<camelContext xmlns="http://camel.apache.org/schema/spring" id="camelContext">

<route>
	<from uri="direct://load-A"/>
	<setHeader headerName="loadDataMapperPfx">
		<groovy>exchange.context.registry.lookupByName("dynamicProductMapper-variantA")</groovy>
	</setHeader>
	<to uri="loadProduct"/>
	<to uri="mock:response-load"/>
</route>

</camelContext>
      After 
      <pfx:dsLoad id="test://loadProduct" objectType="P" businessKeys="sku" mapper="dynamicPartitionMapper" />

<camelContext xmlns="http://camel.apache.org/schema/spring" id="camelContext">

<route>
	<from uri="direct://load-A"/>
	<setHeader name="loadDataMapperPfx">
		<groovy>exchange.context.registry.lookupByName("dynamicProductMapper-variantA")</groovy>
	</setHeader>
	<to uri="bean:loadProduct"/>
	<to uri="mock:response-load"/>
</route>

</camelContext>

2. Use Maven Archetype for New Projects

There is a Maven archetype created. It is recommended to use this archetype for further development, as the archetype follows best practices, already has the basic configuration and will be regularly updated by Pricefx team.

(warning) You should also check your current code against the archetype.

Archetype documentation: Create Project from Maven Archetype

3. Connection Encryption

Connection passwords and clientSecrets will from now on be stored encrypted. Encryption password is managed by the integration.connections.encryption-password property. Encrypted fields are easily recognized by the {ENC} prefix. There is also an option to auto-encrypt current unencrypted properties by setting the integration.connections.auto-encrypt property to true.

All currently unencrypted connections will from now on appear in the logger with the WARN level.

(warning) If an encrypted connection cannot be read (e.g. changed encryption password), the connection cannot be used.

4. Component Changes

Component Change Old value New value
pfx-event The pfx-event component needs now to have the method specified.  <pfx-event?delay=5000/> <pfx-event:fetch?delay=5000/>
pfx-c4c The pfx-c4c component needs now to have the method specified as required param.  <to uri="pfx-c4c?method=fetchCustomers&username=..."/> <to uri="pfx-c4c:fetchCustomers?username=..."/>
pfx-api

The pfx-api component needs now to have the method specified. 

Implemented quote:save, massedit and fetch methods.

 <to uri="pfx-api?method=loaddata&objectType=C&mapper=myMapper"/> <to uri="pfx-api:loaddata?&objectType=C&mapper=myMapper"/>
pfx-rest There are several noteworthy changes on this component – see the changelog of the pfx-rest component.

pfx-dynamics This component was removed. 

pfx-filter This component was removed. 

pfx-hybris Marked as deprecated. Use pfx-rest.

pfx-google-shopping Marked as deprecated. Use pfx-rest.

pfx-odata2 Marked as deprecated. Use pfx-rest.

pfx-salesforce Marked as deprecated. Use pfx-rest.

5. IM Versioning

From now on, we switched to a more common versioning of IM. For more details see IM Roadmap (access for Pricefx only). 

6. Groovy Sandbox Changes

We were forced to narrow the scope of what Groovy sandbox can perform. If you used java.lang.System or java.lang.Class in your Groovy class, you are no longer able to do that. Please go through the updated documentation for more information how to handle it. You may also have to define whitelisted classes from PlatformManager.

7. Properties Changes

From now on, you should use only properties prefixed with integration.* when changing IM behavior. See the archetype documentation for required properties.

8. Secrets Stored Outside Application JAR

By default, the secrets (logstash and kafka connections, default passwords)  are now stored in Config Server. For more details read the ConfigServer specification (Pricefx employees only) and Use ConfigServer for Passwords Management (Optional).

9. Properties checks

There is a property check running on the startup now. It will log all resolved properties so you might check possible duplicates, problems etc.

10. Route error handling  - enableSendingMail vs enable-sending-mail

There is now a property check that checks wrong setup of property enable-sending-mail, which syntax wasn't supported. Please, use enableSendingMail. Mailing configuration now support both formats but the enable-sending-mail is deprecated.

11. Logging File Change

There is a new default location for logs - logs/main.log (instead of main.log). Your options:

  • If possible, ask someone to use salt deploy_im_standard_im_1_2_and_newer.
  • If not possible and you want the same behavior (to store into the root folder):
    Put -Dlogging.file.name=main.log into JAVA_OPTS and integration.logging.file=main.log into application.properties.
  • If the new location suits you, create a folder logs in the root folder.

Improvements

New Feature Description

ID

There is now a class which handles authentication and caching to Pricefx. The class is standalone and must be used as a bean.

PFIMCORE-55

The Maven archetype was created. The steps for a basic project setup were

updated

.

PFIMCORE-275

ITests were merged into Unit tests as external resources were mocked.

PFIMCORE-353

Authentication for the pfx-rest component is now cached. Frequency of the cache renewal can be set via the integration.schedulers.auth.cache.ms property and is set to 10 minutes by default. See the

documentation

.

PFIMCORE-431

There is now a possibility to update provisioned IM via

PlatformManager

.

PFIMCORE-439

There is now a possibility to start/stop IM instance via

PlatformManager

.

PFIMCORE-440

Routes can now be automatically removed along with the maper and connection via API.

PFIMCORE-441

Connection passwords will from now on be stored encrypted. Password encryption is managed by the integration.connections.encryption-password property. Encrypted passwords are easily recognized by the {ENC} prefix. There is also an option to auto-encrypt current unencrypted passwords by setting the integration.connections.auto-encrypt property to true.

All unencrypted connections will from now on appear in the logger with the WARN level.

If the encrypted connection cannot be read (e.g. changed encryption password), the connection cannot be used.

PFIMCORE-442

There is a new component for sending custom messages to PlatformManager. See 

pfx-info Component

. 

PFIMCORE-444

There is now a masking mechanism which prevents sensitive information to be stored into log files. See

 for more information.

PFIMCORE-446

Camel was migrated from 2.25 to 3.2. Follow the steps described above.

PFIMCORE-447

There is a new

pfx-config component

. 

PFIMCORE-448

Fetch is now implemented for the following object types:
DS
DM
LTV
PLI
PGI
CTLI (new)
DCR (new)
F (new)
MPL (new)
PBOME (new)
PCOMP (new)
PDESC (new)
PX (new)
PXREF (new)
QLI (new)
W (new)

PFIMCORE-450

pfx-odata2 now calls also GET on a resource.

PFIMCORE-454

`org.apache.camel:apt` was removed because it was unused and caused problems.

PFIMCORE-457

It is possible now to invalidate the PP cache.

PFIMCORE-464

SAP Commerce migrated to 1905.

PFIMCORE-467

pfx-rest now takes headers from exchange/connection. See the documentation for the

pfx-rest component

.

PFIMCORE-468

pfx-rest now handles path and query parameters in more standardized manner.  See the documentation for the

pfx-rest component

.

PFIMCORE-469

pfx-rest component shares context between component usages.

PFIMCORE-470

Code of IM now resides in Gitlab.

PFIMCORE-477

pfx-rest can set a list of acceptable HTTP responses.

PFIMCORE-489

pfx-rest auth is refreshed upon token expiration.

PFIMCORE-491

There are now more configurable options for the

pfx-rest component

.

PFIMCORE-492

pfx-event now needs to specify the method (e.g. pfx-event:fetch?...) and it creates archive directories if required.

PFIMCORE-502

Groovy sandbox now cannot be disabled.

PFIMCORE-506

Groovy sandbox now runs on every processor.

PFIMCORE-510

IM now provides API for creating typed connections.

PFIMCORE-511

Defined partitions are now sent to Kafka via the startup event.

PFIMCORE-519

There is now a banner in logs showing the IM, Spring and Camel versions and profile used.

PFIMCORE-521

There is now an endpoint which shows all supported convertors.

PFIMCORE-522

pfx-rest can now properly

decode

encoded response.

PFIMCORE-527

Connections are now testable via API.

PFIMCORE-531

The massedit method was added to the pfx-api component.

PFIMCORE-534

The IM Java Client now uses the correct version when contacting APIs of different IM versions.

PFIMCORE-538

pfx-rest component now fills CamelHttpResponseCode with the

return code

.

PFIMCORE-543

Sensitive configuration data are now stored outside the application. See 

for details.

PFIMCORE-544

pfx-api now takes the

connection

from the partitionPfxApi parameter.

PFIMCORE-546

pfx-api can now call the

save

method of a quote.

PFIMCORE-547

A

connection

'pricefx' added during runtime has a higher priority than the default 'pricefx' connection.

PFIMCORE-548

Java runner was added to the demo project.

PFIMCORE-550

It is possible now to loaddata into a PP without knowing its type beforehand. You have to specify objectType=LT and IM determines the table type based on valueType of the response. Currently, all MLTVs and JLTVs are supported for this functionality.

PFIMCORE-557

There are now methods:
- StringToNumberGER.convert(in)
- StringToNumberGER.convert(in, default)
- StringToNumberUS.convert(in)
- StringToNumberUS.convert(in, default)
available in Groovy expressions to convert a string number in a given locale.

PFIMCORE-563

Provisioned IM from version 1.2.0 will now run in docker containers.

PFIMCORE-564

Application properties are now listed on startup to check the setup.

PFIMCORE-572

Both forms 'enableSendingMail' and 'enable-sending-mail' are now supported in the configuration. There is also a check for the latter one which is deprecated.

PFIMCORE-573

Fixed Issues

Bug Description

ID

The data payload sent only with headers does not generate the report in the Records tab.

PFIMCORE-342

The cluster name is sometimes not recognized based on the pfx.partition parameter URL.

PFIMCORE-401

Default logback-spring.xml does not work with ELKINT.

PFIMCORE-475

The HTTP status 409 is marked as NonRecoverable and is not retried. 

PFIMCORE-478

OData component fails when using basic authorization for metadata get.

PFIMCORE-496

In IM data validations, the Quantity data type accepts just integers.

PFIMCORE-517

The IM artefacts are missing from maven-ext.

PFIMCORE-518

There is no possibility to retrieve metadata for the OBJECT and ARRAY types.

PFIMCORE-533

The pfx-api default connection gets violated in one route.

PFIMCORE-541

Kafka messages are masked and therefore IM instances are not monitorable.

PFIMCORE-542