Thinking a bit more, if you can get a straight hole drilled with reasonable accuracy, start point to end point, and have your material a bit thicker/wider (1/8"?), you could then use a drum sander or planer with one end wedged up so the top surface is exactly parallel with the hole. Repeat for the adjacent side. The wedging could be a simple wedge or you could insert a steel rod with a slip fit and have the end protrude a bit to place over a raised support on one end.
You would need to be sure the wood was fully supported so it didn't bow down from the cutting so maybe a thick chunk-o-something underneath and the wedge support under that.
Think of it as a tilting table.
Once two 90 degree surfaces are parallel to the hole, you can just lay it flat to surface the opposing sides to the correct offsets.
Look at "lunch box planer sleds" for a better description.