Someone threatening to sue me for $100. I agreed to return the said money but he does want to collect it. how to defend?

I was looking for a roommate and took the deposit of 100 from him. Did a backgriund check and found him to be unstable. So I refused the apartment and offered to return his deposit. A long series of emails back and forth confirm this. He did not come to collect it and about a week later is calling me and saying I make arrangements to return the money. Such a silly guy, I said its your money and u shud come and take it. He now threatens to send a notice(dont know of what kind) . If he goes to small claims court how to defend myself? I never refused to return the money, but he is arrogant and wants me to remove time and come to his home and return. I appreciate your responses. How to defend myself?