This is done in Skinstudio in: "Edit Titlebars and WindowFrames" in the specific subsection.
If you want to edit the buttons for the PerPixelFrames for instance, you change them in "Edit PerPixelButtons".
You'll have to edit the graphics to make the buttons smaller. Use transparency on .png files, or "Magic Pink" (255,0,255) on bmp files, to set parts of the existing images transparent. Or create new images, which are smaller by default.
Hope this helps. I'm a beginner in skinning SKS.