Sure you can. Just hit the Start button to pause play, go to coaching options and select "challenge play". I challenge and win on fumble calls quite often and have won a couple of challenges on out of bounds plays as described by the original poster. Bad calls are a part of football and the fact that the challenge option is built into Madden is a nice feature, IMO. Of course, if you lose, you lose a time out, etc.
Note - sometimes you have to select the next play call before the "challenge play" option shows up in the coaching menu.