He could react with anger. After all, if he has feelings for Beckett, seeing her with someone else like Vaughn might make him furious. He might lash out at both Beckett and Vaughn without really understanding the situation.
Castle would probably be shocked at first. He'd be thinking 'Why is my daughter doing this?'. Then he'd start trying to talk to her, using his charm and wit to figure out what's going on.