Silva didn't throw a punch back; he got up and started walking away—another segment of his plan to really sink it in that he was only defending himself.
And the four elves bit the bait and attacked him all together. That was when Silva moved faster than their eyes could follow and got behind one of the elves.
"You know, you shouldn't seek out trouble when you do not know what you are dealing with," Silva said. The elf froze in fear. Silva's fist slammed into his spine, the elf arched backward in pain, and then fell to the ground.
The remaining three elves were stunned after seeing this. How fast did one have to be to do what Silva just did? They started retreating, but Silva wasn't going to let them walk away that easily.
You see, all he was looking for was his ticket to do what he wanted, and they gave him that ticket when they attacked him, so he wasn't going to waste it.