I think he meant “while she’s with me” thinking she’s going to leave him and break HIS heart. Remember how panicked he was when he woke up and she wasn’t in bed.
This is what Jessica is always talking about he allows the women to get too close. If he had kicked Serena out soon as he saw her he wouldn't be having this problem.