Ping is not everything. Packet loss is a huge contributing factor also. One of the reasons I asked op if he was playing over wireless because the majority of people playing over wireless have not optimized their wireless router configuration, it could have interference from nearby routers on the same channel causing excessive retransmits of data.
What is packet loss?
Much as it sounds, if you have anything less than complete success in transmitting and receiving "packets" of data then you are experiencing this problem with your Internet connection. It can mean much slower download and upload speeds, poor quality VoIP audio, pauses with streaming media and what seems like time warping in games -- your connection may even come to a total standstill! Packet loss is a metric where anything greater than 0% should cause concern.
Test your ping connection
http://www.pingtest.net/index.php
One thing to remember this test is from your PC to the router. It might say everything is fine however the connection from your Xbox to your router might be the problem. If you are using an ethernet connection try using another Ethernet cable (pets chewing on them, crushed by furniture etc), wireless guys check your signal strength. If you have an older router make sure your firmware is up to date. Also check to see if it is upnp compatible, if not you may have to setup port forwarding.