I have two ex husbands. I have two children with behavioral disorders. 1st is father of my children and I divorced because he was abusive and controlling. I divorced the 2nd because my children claimed and told therapist one time that he was instigating behavioral episodes when I wasn't around. I believed it because 2nd and I were fighting so much. 2nd and I rekindled our friendship after the divorce and our children are still friends. After divorce 1st made me believe that all reasons I left was no longer there and I believed him. 1st never had a problem with my friendship with 2nd. Shortly after moving in with him, I found out he lied, big time. Now that I am moving out again, he is convinced that I am moving back with or going back to the 2nd. at this time that is not true however, who knows what the future has. To be frankly honest, if I could, I would love nothing more than to fix the problems with my 2nd. He is trying to say that I may never take my kids around the 2nd regardless of whether we or the kids are friends or not or he will seek legal action. also, the therapist is no longer seeing the children. we have joint physical and I have sole legal. Does he have grounds?
The short answer is "No, but there are exceptions." An obvious exception would be if you were taking the kids around a registered or known sex offender. He certainly could object to his children being around that person, but ultimately, a Court would have to make a final determination.
Here is the speech: you have children involved. They are important, or should be. Get an attorney. If you cannot afford one try legal aid, the local law school, local organizations that assist with children's issues, or the bar association.
Former spouses often get certain notions in their noggins, and then get convinced the notion is perfectly sound ... even when it isn't.
He is correct that he could seek to alter the custody/parenting agreement to include keep children away from certain people. A court would need to be convinced that notion was in the best interest of the children, and not just something the ex thinks is a right fine idea from his perspective.
He is incorrect about being able to simply call shots in your life and you must adhere to his wants, whims and wishes.