The main issue is that, as an observer we are really only paying attention to 1 on 1 basketball. Curry "overall" is an above average defender but if we're talking about strictly on ball defense or 1 on 1 defense then yes he is slightly below average and the data shows that. If you look at defensive iso stats curry is like in the 47th% for guards. Where he makes up for this though is his off-ball and team D. On ball D is only one aspect of defense and unless you're playing against westbrook, there's only a handful of situations that the point guard is going to be isolating 1 on 1. Ability to guard the pick and roll, denying your man, recognizing and making the right rotations are all apart of defense and even more important and you don't have to be very athletic to do so you just need high IQ. Aka Tim duncan being an elite defender despite being the most physically ungifted bigman. Curry is definitely a below average - average at best on ball defender due in part to his athleticism and size, and klay is a solid on ball defender so it makes sense for him to guard the best opposing guard.
Where curry makes up for it though is his team defense IQ.
People literally only think about on ball D when talking about defense for guards which is a huge problem. Someone like chris paul, is an elite on ball defender but then if you follow him off-ball he often loses his man and overplays the passing lanes. And the data shows where CP3 is #1 among PG's in Defensive ISO PPP but not the case for his overall defensive body of work.