The official big answer on Community Wiki at What are "Community Wiki" posts? says:
Moderators can remove it, when necessary. Once removed, the post will never automatically be given the CW status again.
But, it appears that our ongoing contest (Weekly Featured Image: Ongoing Contest) keeps getting converted. This appears to be a bug, given the quote above.
Alternately, I note that some sites have increased limits for the auto-cw conversion. If fixing the problem isn't easy, could we instead have the auto-cw limit bumped up to something large (1000?) for this site?