I wasn't told to avoid swimming at any point but waited til after 12 weeks mark just in case... we all know those pregnancy paranoias!
Now I do an aquanatal class twice a week and it is FABULOUS!! Just being in the water is great let alone the exercise... oh yes, and all the chat with other expectant mummies-to-be! Weightlessness is certainly a wonderful thing during pregnancy!
So long as you have the support of your doc/mw I'd recommend swimming as often as you're able.
Congrats and good luck with it all.
PS if you're still worried about absorbing chemicals through your skin it's helps reduce this if you lather on body lotion (a pregnancy-safe one of course! ) before you go to the pool, and then soak yourself under the showers just before getting in the pool - this seems to reduce how much your skin wants to suck up the pool water - plus you don't get that lingering chlorine smell afterwards. I always have a long shower directly after swimming too.