The walls of the bailey were the biggest part of the castle, and they contained the most stone. This made them a target for people scavenging stone. Many had also been reduced in height at the end of the Civil War.
The walls you see are often very much lower than they were in medieval times. But little pieces can usually still be seen at their full height, especially where they meet buildings such as the keep, a tower or a gatehouse. Just use your imagination to rebuild the rest and don't forget the crenellations!