I made a few more tests and it is definitely because of the icon count.
I started deleting various kinds of shortcuts and fences, until I just realized that 0 icons = 0 delay, and every new icon I put inside a fence added a very tiny slight delay that eventually starts to mount up.
But I still don´t get why is this happening now.
Just in case. I revised every windows update (I had none the day it started failing) and new software, and none (should!) cause this issue. Im not on the Windows 10 Creator update yet. I also uninstalled Fences but this time deleted all settings folders and registry to start completely anew, with no results.
RAM memory and cpu's are mostly unused, with only a small <5% spike in explorer.exe and dwm.exe whenever I click on an icon, fenced or not, though only the icons inside fences have the delay.