The castle was a personal fortified home, and it needed a garrison. For much of the time the garrison's job was to act like (rather rough) policemen and see the lord's laws were kept. Most were not knights, even though some were on horseback. Others were archers, pikemen and so on.
The lord did need special fighters, and these were made of lesser lords – knights – who had to come and help in times of need, but who otherwise lived on their own lands. It was cheaper this way.
