From the English point of view, the biggest threat was from Scotland and vice versa.
Some of the biggest and strongest castles were placed near the Scottish-English border
Berwick and Carlisle are two of the most fortified and strongest castles in Britain.