You are also experiencing the effect of torque steer, thrown in with deformation of the standard lower front locating bushes.
As has been said earlier replacing those front bushes with Powerflex examples will help, you should then consider that any widening of the front track will increase torque steer and its effects, that may be as a result of fitting spacers to widen the track or indeed a thicker centre bell from some big brake kits, needed to gain caliper clearance with standard wheels.
Regards, Don.