What is the proper response to someone who says "Did you down vote me?".
I know I'm supposed to write a comment when I down vote anyone, but I don't know how should I respond to the down vote question.
Lets address this first "I know I'm supposed to write a comment when I down vote anyone" - simply not true. Downvoting does not require or need a comment. If you choose you may offer an explanation, but it is certainly not the expectation nor something you should feel guilty about not doing.
Second, if somebody asks you - either answer truthfully or don't answer, that part is up to you. But its generally poor taste of the first person to ask.