For each pixel in the selection, Photoshop examines the pixels around it, to the distance you specify in the radius setting. If more than half of these surrounding pixels are selected, the pixel remains in the selection, and the unselected pixels around it are added to the selection. If less than half the surrounding pixels are selected, the pixel is removed from the selection. The overall effect is to reduce patchiness and smooth sharp corners and jagged lines in the selection.