Gremory knew the Oracle probably sent the Giant Worm since the Witch was attacking Zimirt at the same time. So when he discovered the Witch was one of her servants, Gremory thought she sent her to destroy Zimirt.
The Oracle was always acting to destroy all the living beings in the world, so it was undoubted the loyalty to the plan even though she lies a lot and doesn't want to go back into the Abyss.
So the Oracle might have planned to get Gremory involved somehow, and she accomplished that with him diving into madness.
Being used as a tool was not a problem because, in the end, everyone is just pawns in a chess game where the fate of the world is decided; Gremory had no problem in being used like that but being manipulated like that was too much.
How would Gremory know if everything he was fighting for was even true if all this can be only the Oracle desires planted inside his head. There was too much to think about, but this was not the time.