To be honest, he can't stop you on your own website however much he complains. It's your portfolio, plus it actually might increase his traffic/
seo (minor but still).
I'd wait till you're paid and then update your folio, but you don't need to mention this to your client since this is a given. The only time you might is if you're a freelancer for a client of a client. I've seen the same piece of work on 4 people's sites that's been done at some agency - it's fairly unlikely that all 4 had equal parts in it's design but they all featured it as if they had...
The designed by link is another issue. Technically he agreed to it soooooo......