Love is a subjective word here. The feeling involved are a little complicated here than love. The harem is mostly for mutual benefit for now where they help each other out. So no. Not everyone will travel together all the time, just like not everyone will reminder in the MC's harem forever if they find something or someone better for them.