An affair with a married man who now doesnt want to lose me.?

Sounds like he wants his relationship with his wife more than he wants to be your boyfriend. She comes first, and you have solidly been put in your place as second.