At the top of the Fences tab, you can choose a different layout and it will move all the available fences where you can get at them.
After you apply, you can drag them where you want.
Did you have more than one monitor when you set up the fence?
And can you post a screenshot before you change the layout.