I think that is to the bottom of the pillow curve where the flat begins.
How far oversize did you go on the plugs? If you get a tight fit it'll push out the hole to fit tightly with the plug, I think 1/64" is common (and a slight chamfer on the base to ease entry).