# Outlier detection and handling recipe steps Use these recipe steps to work with outliers in your data and perform advanced transformations on them.. **Topics** + [FLAG\$1OUTLIERS](recipe-actions.FLAG_OUTLIERS.md) + [REMOVE\$1OUTLIERS](recipe-actions.outliers.REMOVE_OUTLIERS.md) + [REPLACE\$1OUTLIERS](recipe-actions.REPLACE_OUTLIERS.md) + [RESCALE\$1OUTLIERS\$1WITH\$1Z\$1SCORE](recipe-actions.RESCALE_OUTLIERS_WITH_Z_SCORE.md) + [RESCALE\$1OUTLIERS\$1WITH\$1SKEW](recipe-actions.outliers.RESCALE_OUTLIERS_WITH_SKEW.md) # FLAG\$1OUTLIERS Returns a new column containing a customizable value in each row that indicates if the source column value is an outlier. **Parameters** + `sourceColumn` – Specifies the name of an existing numeric column that might contain outliers. + `targetColumn` – Specifies the name of a new column where the results of the outlier evaluation strategy is to be inserted. + `outlierStrategy` – Specifies the approach to use in detecting outliers. Valid values include the following: + `Z_SCORE` – Identifies a value as an outlier when it deviates from the mean by more than the standard deviation threshold. + `MODIFIED_Z_SCORE` – Identifies a value as an outlier when it deviates from the median by more than the median absolute deviation threshold. + `IQR` – Identifies a values as an outlier when it falls beyond the first and last quartile of column data. The interquartile range (IQR) measures where the middle 50% of the data points are. + `threshold` – Specifies the threshold value to use when detecting outliers. The `sourceColumn` value is identified as an outlier if the score that's calculated with the `outlierStrategy` exceeds this number. The default is 3. + `trueString` – Specifies the string value to use if an outlier is detected. The default is "True". + `falseString` – Specifies the string value to use if no outlier is detected. The default is "False". The following examples display syntax for a single [https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeAction.html](https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeAction.html) operation. A *recipe* contains at least one [https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeStep.html](https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeStep.html) operation, and a recipe step contains at least one recipe action. A *recipe action* runs the data transform that you specify. A group of recipe actions run in sequential order to create the final dataset. ------ #### [ JSON ] The following shows an example `RecipeAction` to use as member of an example `RecipeStep` for a DataBrew [Recipe](https://docs.aws.amazon.com/databrew/latest/dg/API_Recipe.html), using JSON syntax. For syntax examples showing a list of recipe actions, see [Defining a recipe structure](recipes.md#recipes.structure). **Example in JSON** ``` { "Action": { "Operation": "FLAG_OUTLIERS", "Parameters": { "sourceColumn": "name-of-existing-column", "targetColumn": "name-of-new-column", "outlierStrategy": "IQR", "threshold": "1.5", "trueString": "Yes", "falseString": "No" } } } ``` For more information on using this recipe action in an API operation, see [https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html) or [https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html). You can use these and other API operations in your own code. ------ #### [ YAML ] The following shows an example `RecipeAction` to use as member of an example `RecipeStep` for a DataBrew [Recipe](https://docs.aws.amazon.com/databrew/latest/dg/API_Recipe.html), using YAML syntax. For syntax examples showing a list of recipe actions, see [Defining a recipe structure](recipes.md#recipes.structure). **Example in YAML** ``` - Action: Operation: FLAG_OUTLIERS Parameters: sourceColumn: name-of-existing-column targetColumn: name-of-new-column outlierStrategy: IQR trueString: Outlier falseString: No threshold: '1.5' ``` For more information on using this recipe action in an API operation, see [https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html) or [https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html). You can use these and other API operations in your own code. ------ # REMOVE\$1OUTLIERS Removes data points that classify as outliers, based on the settings in the parameters. **Parameters** + `sourceColumn` – Specifies the name of an existing numeric column that might contain outliers. + `outlierStrategy` – Specifies the approach to use in detecting outliers. Valid values include the following: + `Z_SCORE` – Identifies a value as an outlier when it deviates from the mean by more than the standard deviation threshold. + `MODIFIED_Z_SCORE` – Identifies a value as an outlier when it deviates from the median by more than the median absolute deviation threshold. + `IQR` – Identifies a values as an outlier when it falls beyond the first and last quartile of column data. The interquartile range (IQR) measures where the middle 50% of the data points are. + `threshold` – Specifies the threshold value to use when detecting outliers. The `sourceColumn` value is identified as an outlier if the score that's calculated with the `outlierStrategy` exceeds this number. The default is 3. + `removeType` – Specifies the way to remove the data. Valid values include `DELETE_ROWS` and `CLEAR`. + `trimValue` – Specifies whether to remove all or some of the outliers. This Boolean value defaults to `FALSE`. + `FALSE` – Removes all outliers + `TRUE` – Removes outliers that rank outside of the percentile threshold specified in `minValue` and `maxValue`. + `minValue` – Indicates the minimum percentile value for the outlier range. Valid range is 0–100. + `maxValue` – Indicates the maximum percentile value for the outlier range. Valid range is 0–100. The following examples display syntax for a single [https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeAction.html](https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeAction.html) operation. A *recipe* contains at least one [https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeStep.html](https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeStep.html) operation, and a recipe step contains at least one recipe action. A *recipe action* runs the data transform that you specify. A group of recipe actions run in sequential order to create the final dataset. ------ #### [ JSON ] The following shows an example `RecipeAction` to use as member of an example `RecipeStep` for a DataBrew [Recipe](https://docs.aws.amazon.com/databrew/latest/dg/API_Recipe.html), using JSON syntax. For syntax examples showing a list of recipe actions, see [Defining a recipe structure](recipes.md#recipes.structure). **Example in JSON** ``` { "Action": { "Operation": "REMOVE_OUTLIERS", "Parameters": { "sourceColumn": "name-of-existing-column", "outlierStrategy": "Z_SCORE", "threshold": "3", "removeType": "DELETE_ROWS", "trimValue": "TRUE", "minValue": "5", "maxValue": "95" } } } ``` For more information on using this recipe action in an API operation, see [https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html) or [https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html). You can use these and other API operations in your own code. ------ #### [ YAML ] The following shows an example `RecipeAction` to use as member of an example `RecipeStep` for a DataBrew [Recipe](https://docs.aws.amazon.com/databrew/latest/dg/API_Recipe.html), using YAML syntax. For syntax examples showing a list of recipe actions, see [Defining a recipe structure](recipes.md#recipes.structure). **Example in YAML** ``` - Action: Operation: REMOVE_OUTLIERS Parameters: sourceColumn: name-of-existing-column outlierStrategy: Z_SCORE threshold: '3' removeType: DELETE_ROWS trimValue: 'TRUE' minValue: '5' maxValue: '95' ``` For more information on using this recipe action in an API operation, see [https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html) or [https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html). You can use these and other API operations in your own code. ------ # REPLACE\$1OUTLIERS Updates the data point values that classify as outliers, based on the settings in the parameters. **Parameters** + `sourceColumn` – Specifies the name of an existing numeric column that might contain outliers. + `outlierStrategy` – Specifies the approach to use in detecting outliers. Valid values include the following: + `Z_SCORE` – Identifies a value as an outlier when it deviates from the mean by more than the standard deviation threshold. + `MODIFIED_Z_SCORE` – Identifies a value as an outlier when it deviates from the median by more than the median absolute deviation threshold. + `IQR` – Identifies a values as an outlier when it falls beyond the first and last quartile of column data. The interquartile range (IQR) measures where the middle 50% of the data points are. + `threshold` – Specifies the threshold value to use when detecting outliers. The `sourceColumn` value is identified as an outlier if the score that's calculated with the `outlierStrategy` exceeds this number. The default is 3. + `replaceType` – Specifies the method to use when replacing outliers. Valid values include the following: + `WINSORIZE_VALUES` – Specifies using the minimum and maximum percentile to cap the values. + `REPLACE_WITH_CUSTOM` + `REPLACE_WITH_EMPTY` + `REPLACE_WITH_NULL` + `REPLACE_WITH_MODE` + `REPLACE_WITH_AVERAGE` + `REPLACE_WITH_MEDIAN` + `REPLACE_WITH_SUM` + `REPLACE_WITH_MAX` + `modeType` – Indicates the type of modal function to use when `replaceType` is `REPLACE_WITH_MODE`. Valid values include the following: `MIN`, `MAX`, and `AVERAGE`. + `minValue` – Indicates the minimum percentile value for the outlier range that is to be applied when `trimValue` is used. Valid range is 0–100. + `maxValue` – Indicates the maximum percentile value for the outlier range that is to be applied when `trimValue` is used. . Valid range is 0–100. + `value` – Specifies the value to insert when using `REPLACE_WITH_CUSTOM`. + `trimValue` – Specifies whether to remove all or some of the outliers. This Boolean value is set to `TRUE` when `replaceType` is `REPLACE_WITH_NULL`, `REPLACE_WITH_MODE`, or `WINSORIZE_VALUES`. It defaults to `FALSE` for all others. + `FALSE` – Removes all outliers + `TRUE` –Removes outliers that rank outside of the percentile cap threshold specified in `minValue` and `maxValue`. The following examples display syntax for a single [https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeAction.html](https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeAction.html) operation. A *recipe* contains at least one [https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeStep.html](https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeStep.html) operation, and a recipe step contains at least one recipe action. A *recipe action* runs the data transform that you specify. A group of recipe actions run in sequential order to create the final dataset. ------ #### [ JSON ] The following shows an example `RecipeAction` to use as member of an example `RecipeStep` for a DataBrew [Recipe](https://docs.aws.amazon.com/databrew/latest/dg/API_Recipe.html), using JSON syntax. For syntax examples showing a list of recipe actions, see [Defining a recipe structure](recipes.md#recipes.structure). **Example in JSON** ``` { "Action": { "Operation": "REPLACE_OUTLIERS", "Parameters": { "maxValue": "95", "minValue": "5", "modeType": "AVERAGE", "outlierStrategy": "Z_SCORE", "replaceType": "REPLACE_WITH_MODE", "sourceColumn": "name-of-existing-column", "threshold": "3", "trimValue": "TRUE" } } } ``` For more information on using this recipe action in an API operation, see [https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html) or [https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html). You can use these and other API operations in your own code. ------ #### [ YAML ] The following shows an example `RecipeAction` to use as member of an example `RecipeStep` for a DataBrew [Recipe](https://docs.aws.amazon.com/databrew/latest/dg/API_Recipe.html), using YAML syntax. For syntax examples showing a list of recipe actions, see [Defining a recipe structure](recipes.md#recipes.structure). **Example in YAML** ``` - Action: Operation: REMOVE_OUTLIERS Parameters: sourceColumn: name-of-existing-column outlierStrategy: Z_SCORE threshold: '3' replaceType: REPLACE_WITH_MODE modeType: AVERAGE minValue: '5' maxValue: '95' trimValue: 'TRUE' ``` For more information on using this recipe action in an API operation, see [https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html) or [https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html). You can use these and other API operations in your own code. ------ # RESCALE\$1OUTLIERS\$1WITH\$1Z\$1SCORE Returns a new column with a rescaled outlier value in each row, based on the settings in the parameters. This action also applies Z-score normalization to linearly scale data values to have a mean (μ) of 0 and standard deviation (σ) of 1. We recommend this action for handling outliers. **Parameters** + `sourceColumn` – Specifies the name of an existing numeric column that might contain outliers. + `targetColumn` – Specifies the name of an existing numeric column that might contain outliers. + `outlierStrategy` – Specifies the approach to use in detecting outliers. Valid values include the following: + `Z_SCORE` – Identifies a value as an outlier when it deviates from the mean by more than the standard deviation threshold. + `MODIFIED_Z_SCORE` – Identifies a value as an outlier when it deviates from the median by more than the median absolute deviation threshold. + `IQR` – Identifies a values as an outlier when it falls beyond the first and last quartile of column data. The interquartile range (IQR) measures where the middle 50% of the data points are. + `threshold` – The threshold value to use when detecting outliers. The `sourceColumn` value is identified as an outlier if the score that's calculated with the `outlierStrategy` exceeds this number. The default is 3. The following examples display syntax for a single [https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeAction.html](https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeAction.html) operation. A *recipe* contains at least one [https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeStep.html](https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeStep.html) operation, and a recipe step contains at least one recipe action. A *recipe action* runs the data transform that you specify. A group of recipe actions run in sequential order to create the final dataset. ------ #### [ JSON ] The following shows an example `RecipeAction` to use as member of an example `RecipeStep` for a DataBrew [Recipe](https://docs.aws.amazon.com/databrew/latest/dg/API_Recipe.html) operation, using JSON syntax. For syntax examples showing a list of recipe actions, see [Defining a recipe structure](recipes.md#recipes.structure). **Example in JSON** ``` { "Action": { "Operation": "RESCALE_OUTLIERS_WITH_Z_SCORE", "Parameters": { "sourceColumn": "name-of-existing-column", "targetColumn": "name-of-new-column", "outlierStrategy": "Z_SCORE", "threshold": "3" } } } ``` For more information on using this recipe action in an API operation, see [https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html) or [https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html). You can use these and other API operations in your own code. ------ #### [ YAML ] The following shows an example `RecipeAction` to use as member of an example `RecipeStep` for a DataBrew [Recipe](https://docs.aws.amazon.com/databrew/latest/dg/API_Recipe.html) operation, using YAML syntax. For syntax examples showing a list of recipe actions, see [Defining a recipe structure](recipes.md#recipes.structure). **Example in YAML** ``` - Action: Operation: REMOVE_OUTLIERS Parameters: sourceColumn: name-of-existing-column targetColumn: name-of-new-column outlierStrategy: Z_SCORE threshold: '3' ``` For more information on using this recipe action in an API operation, see [https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html) or [https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html). You can use these and other API operations in your own code. ------ # RESCALE\$1OUTLIERS\$1WITH\$1SKEW Returns a new column with a rescaled outlier value in each row, based on the settings in the parameters. This action works to reduce distribution skewness by applying the specified log or root transform. We recommend this action for handling skewed data. **Parameters** + `sourceColumn` – Specifies the name of an existing numeric column that might contain outliers. + `targetColumn` – Specifies the name of an existing numeric column that might contain outliers. + `outlierStrategy` – Specifies the approach to use in detecting outliers. Valid values include the following: + `Z_SCORE` – Identifies a value as an outlier when it deviates from the mean by more than the standard deviation threshold. + `MODIFIED_Z_SCORE` – Identifies a value as an outlier when it deviates from the median by more than the median absolute deviation threshold. + `IQR` – Identifies a values as an outlier when it falls beyond the first and last quartile of column data. The interquartile range (IQR) measures where the middle 50% of the data points are. + `threshold` – Specifies the threshold value to use when detecting outliers. The `sourceColumn` value is identified as an outlier if the score that's calculated with the `outlierStrategy` exceeds this number. The default is 3. + `skewFunction` – Specifies the method to use when replacing outliers. Valid values include the following: + LOG – Applies a strong transformation to reduce positive and negative skew. This is a natural logarithm (2.718281828). + ROOT (with `value = 3` ) – Applies a fairly strong transformation to reduce positive and negative skew. (Cube root) + ROOT (with `value = 2` ) – Applies a moderate transformation to reduce positive skew only. (Square root) + SQUARE – Applies a moderate transformation to reduce negative skew. (Square) + Custom transform – Applies the specified `LOG` or `ROOT` transform using the custom number provided in the `value` parameter. + `value` – Specifies the value to use for the custom transform. If `skewFunction` is LOG, this value represents the base of the log. If `skewFunction` is ROOT, this value represents the power of the root. The following examples display syntax for a single [https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeAction.html](https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeAction.html) operation. A *recipe* contains at least one [https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeStep.html](https://docs.aws.amazon.com/databrew/latest/dg/API_RecipeStep.html) operation, and a recipe step contains at least one recipe action. A *recipe action* runs the data transform that you specify. A group of recipe actions run in sequential order to create the final dataset. ------ #### [ JSON ] The following shows an example `RecipeAction` to use as member of an example `RecipeStep` for a DataBrew [Recipe](https://docs.aws.amazon.com/databrew/latest/dg/API_Recipe.html), using JSON syntax. For syntax examples showing a list of recipe actions, see [Defining a recipe structure](recipes.md#recipes.structure). **Example in JSON** ``` { "Action": { "Operation": "RESCALE_OUTLIERS_WITH_SKEW", "Parameters": { "outlierStrategy": "Z_SCORE", "threshold": "3", "skewFunction": "ROOT", "sourceColumn": "name-of-existing-column", "targetColumn": "name-of-new-column", "value": "4" } } } ``` For more information on using this recipe action in an API operation, see [https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html) or [https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html). You can use these and other API operations in your own code. ------ #### [ YAML ] The following shows an example `RecipeAction` to use as member of an example `RecipeStep` for a DataBrew [Recipe](https://docs.aws.amazon.com/databrew/latest/dg/API_Recipe.html), using YAML syntax. For syntax examples showing a list of recipe actions, see [Defining a recipe structure](recipes.md#recipes.structure). **Example in YAML** ``` - Action: Operation: RESCALE_OUTLIERS_WITH_SKEW Parameters: outlierStrategy: Z_SCORE threshold: '3' skewFunction: ROOT sourceColumn: name-of-existing-column targetColumn: name-of-new-column value: '4' ``` For more information on using this recipe action in an API operation, see [https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_CreateRecipe.html) or [https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html](https://docs.aws.amazon.com/databrew/latest/dg/API_UpdateRecipe.html). You can use these and other API operations in your own code. ------