28 U.S. Code § 455 provides:
(a) Any justice, judge, or magistrate judge of the United States shall disqualify himself in any proceeding in which his impartiality might reasonably be questioned.
(b) He shall also disqualify himself in the following circumstances:
(1) Where he has a personal bias or prejudice concerning a party, or personal knowledge of disputed evidentiary facts concerning the proceeding;
(2) Where in private practice he served as lawyer in the matter in controversy, or a lawyer with whom he previously practiced law served during such association as a lawyer concerning the matter, or the judge or such lawyer has been a material witness concerning it;...
If your judicial officer ruled on your case when subsection (2) was applicable and recusal was necessary -- then according to the statute you have a legal basis on appeal to set aside the ruling and the accompanying agreement.