Try lowering the Layup Success slider to around 30. If it's too hard, go up by increments of 2 or 3 until you find a sweet spot. I'm going to edit the Layup ratings in a future release because some players who don't finish at a very high rate at the rim (especially in traffic) have a very high rating in that category, making it very easy to score on layups when you learn how to use the moves correctly.
If you watch an NBA game, there will be multiple times in a game where a player will get all the way to the rim, try to finish the layup in traffic and will miss without a block or foul. This can be replicated in-game but both the Layup Rating and Layup Success sliders need to be edited for best results.