Keying is a powerful tool that works in two basic steps:
A kind of transfer operation, Keying can be helpful in many tasks such as map regularization, color replacement, feature tracing and more.
Let's see how Keying works by trying it in a map. All map types except Normal support Keying, so open a map into one of those map editors.
Select the Guide map from the drop menu, which in our example will be the map you just opened.
You can use another image or a map open in another editor as a guide, which also has its uses for transferring features from a map to another. Imagine for example, transferring a metallic shape contained in a Base Color map, selected by its color, into the Metallic map, so that the shape's pixel values are set to 1 (fully metallic) - this way you can transfer features from one map into another.
After the Guide is selected, new controls appear below the drop menu, divided in two sections: "Pick Mask" and "Replace With".
Move the mouse cursor over the map and click in an area of contrasting color. When you click, you are picking a source color value that will be replaced by what you define in the "Replace With" section.
The mask of the selected areas in the Guide image is briefly displayed when you click to pick it and also if you click the "Show Mask" button. Try clicking and holding it down to see the mask.
This mask is created based on color/value match between pixels, controlled by the similarity amplitude set in the Range slider: increasing range causes a larger area to become selected and vice versa.
If the Guide image has color, you can change whether the match is done By Color or By Brightness, and a trade-off between the two. If for example, some image features are more distinct in terms of color, move the slider into the By Color side.
Softness controls the blur applied to the boundaries of the mask. This helps softening hard contours and promotes better blending of the replacement color.
The mask is set and we need to choose what to replace it with. To do this choose a Color/Roughness/Height value: all areas previously selected in the mask will turn into this color.
The Guide Mix settings controls how much is the Guide's source brightness blended with the flat Color, which is very useful to add texture to the selected Color. Setting this to 0% will use flat Color only, increasing it will add texture taken from the Guide brightness, creating variety.
Since clicking to pick a mask will forget the previous picking, you can use the "Apply and Pick Another" button to commit the current state. Or, you can press Clear to clear the current mask.
When you click to pick a mask, it is applied to the whole map, which can be problematic for local features.
To restrict the mask to a selected area in the map, press Ctrl and click in the editor area: this will define a selection area which will limit where the mask is applied. Add points to the selection area (while keeping Ctrl down) and finally click in the initial point to close - only after closing will the selection limit the mask.
You can later drag the selection handles to modify the area. To clear, click outside, away from the selection area.
For contextual help, leave the mouse next to the Keying tool's header: an ( i ) button will appear which you can click for tool information.