I never signed a non compete agreement at a tax office. I opened my own office - can I solicit previous clients?

The tax biz was bought out of BK proceedings and I was hired by the buyer to work at the tax office. I did tax prep and representation there. I also paid desk rent to solicit clients for insurance sales (my license they are not licensed). The person who bought the tax company sold it back to the original owner and loaned him the money to buy it. I am being told by the old owner I cant solicit the clients from the tax office - even though I paid for the client base I have (I am not mailing all the tax office clients - just mine I acquired there) and he no longer owns it (he did loan money to old owner). I opened my own tax and accounting and insurance firm and I believe I can contact clients I paid for. My office is 13.8 miles away from his so some may or may not come see me for my services.