The Emperor returned to the room, sat on the bed and watched the sleeping Lei Xing. Of course, the Emperor did not believe the rumors. First of all, he had already investigated Lei Xing's past. Second, he always had people watching his brothers. Therefore, he would know about said secret affair.
But the rumors still greatly annoyed him. She is to be his Empress, so it would not be good to let this sort of false rumor damage her reputation. Clearly, she was the target of the rumors, which meant someone was purposely trying to ruin her.
The Emperor initially suspected his mother, but this was not how she handled things. If it were her, she would have waited until a later time and simply dealt with Lei Xing directly, she did not need to spread a rumor to get it done. So he turned his suspicion to the other Imperial wives, but it was not like he could torture all of them to get answers, that would just exacerbate the situation.