Fatigue is a factor - pitches lose velocity and probably movement as well as location as a pitcher hits tired and eventually exhausted status.
Not to mention the confidence meters. As a pitcher gets pushed around, his confidence falls (both in pitches and overall), tending to cause things to continue to spiral out of control. It works both ways (i.e. you can do the same to the CPU pitchers), but I think it's overdone and it's slanted against the user (even with 0 consistency for the CPU, the user's pitchers will lose confidence much more sharply and regain confidence much more slowly than the CPU pitchers).
L/R match-ups can matter too, but that's more considering the splits of the hitters since pitchers don't have splits for some reason.