Unfortunately, I don’t know about making changes to the size of subsystem, but whenever I intend to use small subsystems, I copy one small subsystem in Optisystem samples and use that in my layout.
However, as for what you have mentioned about the limited size of layout, you can change its size by using “layout size” icon on top of the page below menu bar (with a blue icon) to change the height and width of the layout. I hope this helps.
Regards