Normally their Potential isn't effected, and if it is, it's only by a point or two.
I've generally found though that the earlier they join the MLB club the faster they progress.
I simmed a couple of test seasons with the Phillies and tested this on JP Crawford (their top prospect). When I left him down in AAA during the entire 2017 season, he finished the season with an 81 OVR both times. When I moved him up to the MLB in early July around the all-star break, his overall would consistently be an 85 or an 86.
I'm playing the entire 2019 season right now and he's still currently an 85 OVR but he struggled last year (batted .250 and change), so he didn't really progress last year at all.
It could be that moving them to the MLB helps in the short-term and hurts in the long-term, but I would just test it a few times to make sure. Most of the progression I feel is based on performance so each scenario is probably case by case anyway.