The Roman empire was vast. It needed strong rulers to hold it together.
The 'barbarians' to the north were also able warriors. So when there was trouble back in Rome, the barbarians were quick to attack and capture the lands of the empire. They even burned Rome to the ground in 410.
Within a century, most of the huge empire had been lost. Rome would never be great again.