Thanks Pottz. 
First I have to drill a 3/8” hole. Then I place the 3/8” mortise bit housing over that hole. I get it dead over the hole. I even use a square on it to square it. Then I hit the mortise. Sometimes it moves a 1/64” over the hole. The grain in the wood sometimes draws it. 

And now on one side of the square a little part of the 3/8” drill hole is showing. 

Now I have to straighten that section out with a chisel and the hole has grown 1/64”or more. 
It doesn’t happen all the time.

That’s why I make the holes first and make the ebony plugs last. 
I hope I’m making sense. If not let me know. 

James McIntyre