If you choose the Reduce type in a transform process, you can specify the following operations (or actions) on one or more fields of data. It reduces a set of records down to one summary record, which means corresponding rows will be merged into one row. Before starting this transform, make sure the fields have been sorted first, which minimizes memory use. This is because comparing each record with many other unsorted records requires a significantly large memory if there are huge volumes of data. However, if we only have to compare each record with the previous record, we can run through massive data without needing huge amounts of memory.
The following table shows an example of the input shared by all the operations under Reduce Transform:
This operation calculates the values from the specified fields into the average.
If you choose the Average action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 2.0 | 2.9 |
| D | E | 4.5 | 5.4 |
This operation merges the values from the specified fields into an unordered list separated by comma.
If you choose the Comma Separated List action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 2.0,1.0,3.0 | 2.9,1.9,3.9 |
| D | E | 5.0,4.0 | 5.9,4.9 |
This operation merges the values from the specified fields into an ordered list separated by comma.
If you choose the Comma Separated Set action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 1.0,2.0,3.0 | 1.9,2.9,3.9 |
| D | E | 4.0,5.0 | 4.9,5.9 |
This operation counts how many values are in the specified fields.
If you choose the Count action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 3 | 3 |
| D | E | 2 | 2 |
This operation extracts the first value from the specified fields.
If you choose the First action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 2.0 | 2.9 |
| D | E | 5.0 | 5.9 |
This operation extracts the last value from the specified fields.
If you choose the Last action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 3.0 | 3.9 |
| D | E | 4.0 | 4.9 |
This operation merges the values from the specified fields into the maximum.
If you choose the Max action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 3.0 | 3.9 |
| D | E | 5.0 | 5.9 |
This operation merges the values from the specified fields into the median value. If there are only two values in a field, it returns the average directly.
If you choose the Median action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 2.0 | 2.9 |
| D | E | 4.5 | 5.4 |
This operation merges the values from the specified fields into the minimum.
If you choose the Min action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 1.0 | 1.9 |
| D | E | 4.0 | 4.9 |
This operation calculates the values from the specified fields into the product.
If you choose the Product action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 6.000 | 21.489 |
| D | E | 20.00 | 28.91 |
This operation calculates the values from the specified fields into the standard deviation.
If you choose the Standard Deviation action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 1 | 1 |
| D | E | 0.7071* | 0.7071* |
*This is an approximate value of the output for the convenience of illustration.
This operation calculates the values from the specified fields into the summary.
If you choose the Sum action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 6.0 | 8.7 |
| D | E | 9.0 | 10.8 |
This operation calculates the values from the specified fields into the variance.
If you choose the Variance action for Field 3 and Field 4, the following table shows the output of Table 6.1, “Shared input”:
| Field 1 | Field 2 | Field 3 | Field 4 |
|---|---|---|---|
| A | B | 1.00 | 1.00 |
| D | E | 0.50 | 0.50 |