The Goblin Summon understood Adam's orders and respected and revered Adam as its master, so was planning to follow them to the best of its abilities.
However, as soon as they neared the castle, it clearly seemed afraid and intimidated by the castle or whatever is inside it.
It still followed closely after Adam, but what it was feeling was clear to Adam and it surprised him, but also confirmed to him that there really was more to the castle than traps.
The traps were going to be activated without him needing to tip them off, so it was best if he protected his Goblin Summon while skipping through the traps as quickly as he could.
He could sense that the Goblin Summon was sending out some sort of radar that was similar to a spiritual sense, but also very different at the same time.