Is it possible to 'unbroaden' shoulders from exercising?

Question

I have been swimming for 4 years now and I've noticed that my shoulders have broadened and clothes don't fit me properly around the shoulders. If I stop swimming, will my shoulders 'unbroaden' or are the changes permanent? Is there any better sport I can do that does not broaden shoulders and just tones the body without excessive bulky muscle?