No, you can icon switch to anyone at any time. It's because the opponent has their icons up at the same time preventing you from seeing yours.
You can test this out in a game with both controllers. On an inbound activate the defending team's switch icons and you won't be able to activate the inbounding team's pass icons, and vice versa. Use this knowledge responsibly hahah.