Part of the issue as you said is you controlling them. It is similar to when you control a QB, outside of forcing errant throws you the human are making the reads which is the most difficult part. As a HB they could make you fall down easier as a back up I guess, unless that is one of the attributes your HB is good at, but you the player are controlling finding the running lanes which tends to be the toughest thing for a HB.