If you just want rounded corners on the top title bar you can do that by making the titlebar image with rounded corners (transparent around the corners) and checking the "Replace default frames" box here:
If you want the rounded corners on the bottom, you'll probably need to use the "Shadows" images for it. You can see how he did that on this skin (Studio by Gilbou):
https://www.wincustomize.com/explore/curtains/272/
Just doing the titlebar is much easier.
