Can I be forced to testify in a divorce case if I just talked to a married woman?

If you receive a subpoena to testify, then you have to go to court and answer the questions. Otherwise, you cannot be compelled to testify.

I would agree with my colleague. It isn't clear what state you're in, either. So, it may not even be relevant (whether or not you are or are not seeing this person). In KY, unless there is a claim for spousal support, whether or not a party is unfaithful doesn't really have much relevance in a divorce proceeding as KY is a no fault state.