Yeah but fielding is such a huge part of the game, and I actually like doing it. So that is not an option here.
Anyway, I searched the web and saw people complaining about this last year. So it's not a bug from this year's version only.
Anyway, the explanation as to why this occurs seems to be that if you keep pressing the thumbstick after the camera changes, when the hitter hits the ball, then your fielder will go into some automatic animation and as soon as it finishes, he will go to the direction you had the analog stick pointed right after the camera changed, no matter if you have long pressed it to the opposite direction.
So the "fix" is to let go of the analog stick right after the pitch.
I do seem to have a habit of keep pressing it after the pitch is made.