Yep, same here. I logged tons of hours playing Madden 2000 on N64, but the controller always irked me. That tiny freaking analog stick seems so crappy when you look at analog control now, and became obsolete when Sony came w/ Dual Shock controls.
I'd have to say the PS2 would be my #1, with 360 at #2. Even though now I can't even use PS2/PS3 controller, I've become so accustomed to the 360.
Never really was a big fan of the Dreamcast controller either.
N64 was ok for stuff like Zelda & Golden Eye, but I too felt like it was inadiquate for sports games. I remember playing QB Club and just hating it because of the controller, and i would get so frustrated playing games like 1080 because that little joystick made doing the tricks so difficult.
I despised the N64 controller as well. I would always try to hold it from the outsides of the controller and stretch my thumb to the analog stick (which was a disaster) and I could never get access to the Z button. I'd always be like "How do you do ____" and he'd be like "Press the Z button." Then I'd have to look all over the controller to figure out it was underneath it.
I'd agree with the list except for #1. The NES can't win all debates on sentimentality alone as it always seems to do, but for whatever reason it does.
With that said, I feel the 360 controller is the best of the bunch right now in terms of response, size, and ergonomics. The PS3 controller has been the exact same since PSOne except add-ons but in the end it's still far too small and uncomfortable after even a short while.