How much of a factor? Enough of a factor where the obvious scenarios simply don't play out. A small guy should not be able to run over a significantly larger guy. It should pass the eye test, it doesn't have to be overly complicated.
If I ran into Dontari Poe full speed in real life, I'd expect that 99.99% of the time, I'd fall flat on my back afterwards. Unless he was sleepwalking. Then I might have a 0.01% chance.
Because, you know, physics and other magical stuff.