My “shop” is my garage, and we actually park our cars in it, so I put wheels on everything. It’s not even a debate.
It’s interesting that you see such a difference between 3” and 4” wheels. My workbench has flip-up casters, and I’m pretty sure they’re 3” too, and I haven’t had any problems.