How to Replace a Photo's Background with Photoshop's Refine Mask Tool

By Darrin Koltow

Using Photoshop to replace a fuzzy-edged subject's background once involved painstaking pixel-level work. But the evolved edge-detection technology of Refine Mask has put those bad ol' days behind you. Refine Mask looks for sharp color changes within an outline surrounding your target subject. It then renders the changes onto a layer mask. Layer masks fitted with Refine Mask potentially boost your photos' storytelling power, a power this upcoming tutorial discusses. Consider a photo whose drama is obscured by foreground elements. Layer masks tailored with Refine Mask let you keep the drama in focus so you can blur just the foreground.

Load into Photoshop a photo with a subject you want to cut out. Pick a subject whose edges are difficult to discern so you can see how effective Refine Mask is. Possible subjects include a person with frizzy hair, foliage from a landscape photo, or an object emitting smoke. If you want to work with the same images as those in this tutorial, see the image credits at the end of the tutorial. The following are the beginning and ending images:

Duplicate the Background layer by pressing Control or Command-J, then click the upper color swatch of the Tools palette to choose a colour that contrasts sharply with the subject you're cutting out. Click Edit>Fill to fill the background with the colour.

                       Fill the background with a colour that contrasts with your subject

                               Make a rough tracing of the subject

Click Window>Mask to display the Mask panel, then click the Pixel Mask button to make a layer mask. Click the Mask Edge button to display the array of controls for fitting the mask. We'll first use automated tools for this task, then manual tools.
 

                           Create the mask                                                               The Refine Mask window

Automated Fitting

Click the Show Radius checkbox to display the Radius, the area where Photoshop will look for the separation between the subject and its background. This area doesn't exist until you tell Photoshop to make it, so the canvas appears black.

Drag the Radius slider a bit to the right. A thin outline appears around the subject, indicating that Photoshop is expanding the Radius.

Controls and appearance of the Radius

Uncheck Show Radius, then drag further to the right to close the gap between the rough outline you traced and the actual outline of your subject. Check the Smart Radius checkbox to make the Radius larger for soft areas of the subject and smaller for the hard-edged areas. This doesn’t always improve the mask, so uncheck this control if your mask looks worse. To see different representations of the layer mask, which can help you refine it, click different views within the View control. For example, click the Marching Ants view to display the mask border as marching ants.

Increase the Radius to refine the mask

Drag the Radius slider right until the bright Background layer starts intruding into the subject. Drag left until the subject is restored, and the gap between the foreground and background is as small as possible. Drag the Shift Edge slider left to reduce the gap even further. This control widens and shrinks the subject and background borders that Photoshop detected within the Radius.

Manual Mask Fitting
Up to now you've been using Refine Mask's automated tools for fitting the layer mask. Now you'll use its manual tool, Refine Radius, to further improve the mask. This tool is turned on by default, which the circular mouse cursor indicates.

Press "[" or "]" to size the Refine Radius brush until you can easily straddle the brush across your subject's outline. Drag over that outline to manually define the Radius, in which Photoshop will seek your subject's exact outline. Photoshop will add to the mask when you release the mouse button. If the mask intrudes onto the subject, press and hold Alt or Opt when you drag the brush to restore a portion of the original, unfitted layer mask.

Drag over the subject's outline to refine the layer mask

If refining the mask is difficult because the background's colour has bled into the foreground, check the "Decontaminate colors" checkbox. Photoshop will replace the color tinge with nearby colours that are closer toward the subject's center.

Click "OK" when you've made the layer mask as tight as you can. If there are still gaps in the mask, paint them away by using the Brush tool directly on the layer mask. Or, delete the layer mask, then refine a new one with Refine Mask. When you're done, you're ready to paste your masked subject onto a new background.

Open the new background image on which you want your subject to appear. Click Window>Arrange>Tile to view the foreground and background canvases together. Drag the layer holding your cutout subject onto the background's canvas to insert the layer onto the background.

Use the Move and Free Transform tools to arrange the subject or background into a composition you like.

Photo Credits

• Heather Blush at the Canada Day 2007 concert, courtesy Robert Thivierge
• Crowd at the City of Manchester against Hull City in the Premiership, courtesy AGilhooley
• Both photos are available at Wikimedia Commons