The recoil force exerted back toward the shooter by the gun is centered on the barrel axis, the bullet pushing forward down the barrel creates the "Push-back" in line with it.  However the grip, which is resisting that force to keep the gun from flying back in our face, is centered about halfway down the handle.  (The exact distance down the handle is irrelevant since the whole hand is below the bore axis anyway.)

So the resulting torque - twisting force - comes from the push-back along the axis of the bore wanting to twist the gun around the handle's center point (or a point close to it).  If the gun is held sideways, then that twisting force will be sideways as well and the bullets will tend to come out in a horizontal line.

Is this an oversimplification? To an extent, sure, but those are still the main factors.  Others are how secure the shooter's grip is and whether the wrist is locked - things like that.

Does that work for you?

Stu