In any fight there are two sides the attackers and the defenders.
The attackers would have been an army on the march. The defenders would have been inside a castle in their way.
The attackers could not march past a defending castle for fear the garrison would come out as soon as they were past, and attack them from the rear.