When you look at a wadi, as in this picture, you will see a gorge with a sandy floor. Sometimes, when there is heavy rain, the rain will run off of the rocky mountains and fill the valleys and make a flash flood. That will wash much of the sand away.

But rain is very rare in a desert, and so the flood of water quickly disappears and winds blow sand off the mountains and begin to replace to the sand on the wadi floor.