I downvoted on an answer few days ago, explaining carefully in a comment why I think the answer is incorrect. My votes are 72 up / 2 down. Today, I noticed my reputation on the site has gone down by 80. I noticed there were 40 downvotes (maximum permitted in a day by a single person), and wondered why. No downvote contained a single comment, and all were within 1 hour or so.
I suspected the person who I downvoted is the downvoter, and checked the profile of that person. Sure enough, this person got "Vox Populi", "Suffrage" and "Critic" badges at the same time my posts were systematically downvoted. The "all time" downvotes was also 40; there were only 5 upvotes. Votes cast month/week/day were 40/40/40. Thus, I have very strong grounds to believe that this person is systematically downvoting me.
Is this kind of activity permitted on the site? Sure, a downvote takes 1 of your reputation, but then again it takes 2 of the downvoted person's reputation so it's not a zero sum game. Thus, I believe downvotes should be used sparingly.