Lately I noticed, when I downvote a answer(Which according to me is poor) my reputation is reduced by 1.
Is this a standard procedure or something wrong with my downvotes?
Downvotes on questions don't give -1, to promote well researched questions which are not easily found elsewhere in the net. Downvotes on answers imply a small negative effect, so that you value them well. It is much smaller than the amount you get when you are upvoted, so it shouldn't pose a problem.
This was slightly different in the past, the current situation was explained some months ago by SE staff. So, the observed behaviour is normal.
Ps: You can also flag for attention problematic answers or questions, and it doesn't affect your rep.
This is one of the key reasons Stack Exchange works. On Facebook Questions, for example, everyone voted down all competing answers, basically making the thing useless. There are other ways to solve that, but this is very elegant.
By the time you have enough reputation to go on a downvoting rampage, you're usually engaged enough with the site that you won't.