The goal of this iteration is to give the logic a better structure, so it follows a best-practice pattern and is easier to read by others. We will also add some performance improvements.
Best-practice Logic Structure
This is the general structure of a logic which follows the best practices. The rules are not strict but dividing the logic into multiple logical parts helps the readability.
|
Element Name/Prefix |
Description |
Example |
|---|---|---|
|
Global |
Element with the |
- |
|
InputXXX |
Input elements starting with the Input prefix. |
InputCountry, InputDiscount, etc. |
|
AbortIfInputGeneration |
Special element with the input generation statement. The code after this element is not executed in the input generation mode. |
- |
|
GetXXX |
Elements with the Get prefix are meant to fetch and pre-process data. |
GetCostData, GetCountryData, GetMargin |
|
- |
All other elements go here. |
ProductCost, BasePrice, FinalPrice |
|
Errors |
It is a good practice to provide the users with some kind of indication if something went wrong (e.g., in a neatly formatted ResultMatrix). |
- |
We will update the structure of the logic according to this best practice in the following way:
|
Element Name |
Description |
|---|---|
|
Global |
This stays the first element of the logic. |
|
GetCountryData |
Here we fetch and cache the information about countries. Note: Contrary to the best-practice structure we place this element before the InputCountry element, but that is because we need the country data to populate the drop-down with countries. Therefore it is okay. |
|
InputCountry |
Generates the user input. |
|
AbortIfInputGeneration |
Aborts the calculation in the input generation mode. |
|
GetCostData |
Fetches and processes the data related to costs. |
|
ProductCost |
Calculates the product cost. |
|
ProductGroupAverageCost |
Displays the data calculated in GetCostData. |
|
AverageCostDiffPct |
Calculates the average cost difference. |
|
MarginAdjustment |
Fetches the margin adjustment for the product group. |
|
BasePrice |
Calculates the base price. |
|
CountryAdjustedBasePrice |
Adjusts the base price with the country factor. |
|
Errors |
Displays the errors. |
Global Cache Pattern, Caveats of the Find Method
Country Element
Let’s divide the Country element into the data-gathering part GetCountryData and the input part InputCountry as described above. And also let’s add some global cache.
Before
Country
// fetch the country factors
def countries = api.findLookupTableValues("CountryFactor")
// store the factors in the api.local space for further re-use
api.local.countries = countries
// transform the countries in a map so we can use it as labels for the drop-down
// and sort the results by the label alphabetically
def labels = countries.collectEntries {
[it.key1, it.attribute2]
}.sort {
it.getValue()
}
options = labels.keySet().toList()
// create the user entry
def selectedCountry = api.option("Country", options, labels)
// add input validation and set a default value if the input is not specified
if (!selectedCountry) {
api.yellowAlert("Country not selected, defaulting to AU")
api.local.errors << "Country not selected, defaulting to AU"
selectedCountry = "AU"
}
return selectedCountry
After
GetCountryData
// do this only once during the PL generation
if (api.global.countries == null) {
// fetch the country factors
def countries = api.findLookupTableValues("CountryFactor")
// transform the countries in a map so we can use it as labels for the drop-down
// and sort the results by the label alphabetically
def labels = countries.collectEntries {
[it.key1, it.attribute2]
}.sort {
it.getValue()
}
def options = labels.keySet().toList()
// cache the values
api.global.countries = countries
api.global.labels = labels
api.global.options = options
}
return null
InputCountry
// create the user entry
def selectedCountry = api.option("Country", api.global.options, api.global.labels)
// set default value if entry not specified
if (!selectedCountry) {
api.yellowAlert("Country not selected, defaulting to AU")
selectedCountry = "AU"
}
return selectedCountry
Step-by-step Analysis
if (api.global.countries == null) {
...
api.global.countries = countries
}
This is a caching pattern which ensures the block of code within the if-statement is run only once per the price list generation. For the line items other than the first one api.global.countries will be non-null and therefore the block will be skipped.
You need to set api.retainGlobal = true in the first element of the logic as shown earlier.
api.global.countries = countries
api.global.labels = labels
api.global.options = options
Here we cache the list of countries, the labels and options in the api.global space. This is done only once (when the first line item is processed) and we can re-use them later in the new InputCountry element.
def selectedCountry = api.option("Country", api.global.options, api.global.labels)
Here we re-use the previously cached options and labels for the country drop-down. Note this code is run for every line item.
ProductCost Element
Let’s give this element a small performance boost by filtering on the Valid From date directly in the database.
Before
ProductCost
def sku = api.product("sku")
// get the current or target date
def targetDate = api.calendar().format("yyyy-MM-dd")
// fetch the costs
// we could also do the filter on targetDate here but I couldn't demonstate the usage of a closure later
def data = api.productExtension("Product_Costs")
// find only records where the validity date is lower than the target date
def validRecords = data.findAll{ it.attribute4 <= targetDate }
if (!validRecords) {
api.redAlert("Cost not found for $sku")
api.local.errors << "Cost not found for $sku"
return null
}
// sort the costs by date in descending order and pick the most recent one
validRecords = validRecords.sort { a, b ->
return b.attribute4 <=> a.attribute4;
}
return validRecords[0].attribute1.toBigDecimal()
After
ProductCost
def sku = api.product("sku")
// get the current or target date
def targetDate = api.calendar().format("yyyy-MM-dd")
// set up the filters
def filters = [
Filter.equal("name", "Product_Costs"), // PX name
Filter.equal("sku", sku), // Sku
Filter.lessOrEqual("attribute4", targetDate) // Cost Date
]
// fetch the latest cost from the PX
// use the expansion operator to expand a list into a varargs array
def data = api.find("PX", 0, 1, "-attribute4", ["attribute1"], *filters)
if (!data) {
api.redAlert("Cost not found for $sku")
api.local.errors << "Cost not found for $sku"
return 0.0
}
return data[0].attribute1?.toBigDecimal()
Step-by-step Analysis
// set up the filters
def filters = [
Filter.equal("name", "Product_Costs"), // PX name
Filter.equal("sku", sku), // Sku
Filter.lessOrEqual("attribute4", targetDate) // Cost Date
]
// fetch the latest cost from the PX
// use the expansion operator to expand a list into a varargs array
def data = api.find("PX", 0, 1, "-attribute4", ["attribute1"], *filters)
Instead of fetching all product costs related to the product and then filtering the ones valid on the target date, we use api.find with a sortBy parameter. Because we filter only records which are less or equal to the target date and we sort by the Valid From date (attribute4) in the descending order (that is why there is a minus), we will get just what we want.
-
When using
api.find, only use thesortByparameter if you have a good reason for it. It has a negative impact on performance since it implies an extra sorting operation. -
Always specify the list of fields returned by the
api.findmethod. Only in very rare cases you will need all the fields. But in majority of the cases you will save a lot of bandwidth and therefore time when you only specify the fields you need. -
When you do not specify the limits of
api.findbecause you expect to load a potentially big number of records, useapi.streaminstead. It has some overhead cost but for large records it is quite fast.
Data Aggregation on the Database Level
ProductGroupAverageCost Element
We will divide this element into two parts. The first will gather the data and calculate the result, the other will just display the result.
We will therefore move everything from the ProductGroupAverageCost element into the new GetCostData element and make it more performance effective by adding the global cache pattern. Then we display the result in the original element.
Before
ProductGroupAverageCost
// let's create a reference to the library so the code is better readable
// especially useful when there are multiple libraries and/or long names
def Math = libs.DemoLib.MathUtil
def pg = api.product("Product Group")
// fetch the costs from the PX
// notice using the Product Master attribute to filter the PX
def cit = api.stream("PX", null,
Filter.equal("name", "Product_Costs"), // PX Name
Filter.equal("ProductMaster__attribute2", pg)) // Product Group
def costs = cit.collect{ it }
cit.close()
if (!costs) {
api.local.errors << "Missing costs for group $pg"
return null
}
def avgCost = costs.sum{ it.attribute1.toBigDecimal() } / costs.size()
return Math.round(avgCost, 4)
After
GetCostData
// let's create a reference to the library so the code is better readable
// especially useful when there are multiple libraries and/or long names
def Math = libs.DemoLib.MathUtil
// calculate the average costs only once and cache the result
if (api.global.productGroupAverageCosts == null) {
api.global.productGroupAverageCosts = [:]
// fetch the distinct product groups (there can be potentially a lot of them)
def productGroupsIt = api.stream("P", "sku", ["attribute2"], true)
def productGroups = productGroupsIt.collect { it.attribute2 }
productGroupsIt.close()
// remove empty product group
productGroups -= null
// for each product group except for empty, calculate the average cost
for (pg in productGroups) {
// fetch the costs from the PX
// notice using the Product Master attribute to filter the PX
def avgCosts = api.find("PX", 0, 1, "sku", ["attribute1": "AVG"], false,
Filter.equal("name", "Product_Costs"), // PX Name
Filter.equal("ProductMaster__attribute2", pg)) // Product Group
// extract the average cost
def avgCost = avgCosts ? avgCosts[0].attribute1.toBigDecimal() : null
if (avgCost == null) {
api.local.errors << "Missing costs for group $pg"
continue
}
// store the result in the Global cache
api.global.productGroupAverageCosts[pg] = Math.round(avgCost, 4)
}
}
ProductGroupAverageCost
// use the cached result on subsequent runs
def productGroup = api.product("Product Group")
return api.global.productGroupAverageCosts[productGroup] ?: null
Step-by-step Analysis
if (api.global.productGroupAverageCosts == null) {
api.global.productGroupAverageCosts = [:]
... rest of the code
}
Ideally we want to run the average group cost calculation only once for every group and then re-use it. This caching pattern ensures the inner block of code is run only once.
def productGroupsIt = api.stream("P", "sku", ["attribute2"], true)
def productGroups = productGroupsIt.collect { it.attribute2 }
productGroupsIt.close()
This code opens a stream to the Product Master and fetches only distinct product groups thanks to the last parameter set to true. We sort the results by sku because we do not really care. Sorting by attribute2 would be perfectly fine, but sku is ensured to be indexed, therefore it might be faster.
Thanks to Groovy we can call the collect closure directly on the stream iterator and collect the distinct group names. Don’t forget to close it after you’re done.
productGroups -= null
Because there might be products without a product group, the previous call could return also null as one of the product groups. Here we remove it from the list. This syntax is equivalent to calling productGroups.remove(null).
def avgCosts = api.find("PX", 0, 1, "sku", ["attribute1": "AVG"], false,
Filter.equal("name", "Product_Costs"), // PX Name
Filter.equal("ProductMaster__attribute2", pg)) // Product Group
This overload of api.find allows us to specify an aggregation on the database level. We want to get an average cost of all products of a group pg. We do that by providing a map of fields instead of the list, and an aggregation function ["attribute1": "AVG"]. Since we only aggregate one group, we limit the result to only one line.
There is currently a limitation that there is no equivalent to the group-by clause we are used to in SQL. The method aggregates all the data. Also, we cannot combine normal and aggregated fields. Everything is aggregated.
Aggregation works also with api.stream.
def avgCost = avgCosts ? avgCosts[0].attribute1.toBigDecimal() : null
if (avgCost == null) {
api.local.errors << "Missing costs for group $pg"
continue
}
// store the result in the Global cache
api.global.productGroupAverageCosts[pg] = Math.round(avgCost, 4)
Here we extract the cost from the first (and only) item returned by the previous api.find. If no cost is found, we display an error in the Errors element. Otherwise we store the average cost in the api.global.productGroupAverageCosts map with the product group pg as the key.
def productGroup = api.product("Product Group")
return api.global.productGroupAverageCosts[productGroup] ?: null
In the ProductGroupAverageCost element we only access the previously cached value from the api.global.productGroupAverageCosts field.
MarginAdjustment Element
You might also think about improving the performance of the MarginAdjustment element by caching the margins for each product group. Let’s do that.
Before
MarginAdjustment
def pg = api.product("Product Group")
def marginAdj = api.vLookup("MarginAdj", "Margin Adj", pg)
if (marginAdj == null) {
marginAdj = 0.0
}
return marginAdj
After
MarginAdjustment
if (api.global.margins == null) {
// if the key is not found, the map returns 0.0
api.global.margins = [:].withDefault{ 0.0 }
def margins = api.findLookupTableValues("MarginAdj")
margins.inject(api.global.margins, { map, it ->
map[it.name] = it.attribute1
return map
})
}
def pg = api.product("Product Group")
return api.global.margins[pg]
Step-by-step Analysis
if (api.global.margins == null) {
// if the key is not found, the map returns 0.0
api.global.margins = [:].withDefault{ 0.0 }
}
def pg = api.product("Product Group")
return api.global.margins[pg]
If this is the first execution of the logic on the first line item, we initialize api.global.margins with a map. Notice withDefault{ 0.0 }. If a key is not found in the map, the value 0.0 is returned instead of null.
If this is the next execution of the logic, we just retrieve the margin from the global map.
def margins = api.findLookupTableValues("MarginAdj")
margins.inject(api.global.margins, { map, it ->
map[it.name] = it.attribute1
return map
})
We fetch the complete contents of the MarginAdj PP. Then we use the fancy inject closure which helps us populate the previously created map with the entries. The map key is the product group name and the value is the margin percentage.
Depending on the data, this optimization is not that necessary if there are just a couple of product groups. The vLookup function is optimized and automatically uses a cache of its own. Therefore a call to the function with a given key accesses the database only once and repeated calls with the same key retrieve the value from a cache. Given we have a price list of 100,000 products but just 10 product groups, there will be just 10 accesses to the DB.
It is not necessary to cache the call to the the vLookup function because it is cached automatically on the server. Every value is accessed only once for any given key combination.