To calculate the link margin budget of a system, first duplicate the layout. Then add an Attenuator component before your photodetector. While increasing the Attenuator’s loss monitor the BER of your system. Once the BER is below your desired threshold, the Attenuator’s loss should signify the link margin budget of your system.
For the system power penalty, you can perform the same protocol. For example, if you are interested in the dispersion power penalty. You duplicate your layout, remove the cause of dispersion and add an Attenuator before the photodetector. Then increase the loss until the BER matches that of your dispersion-including-layout.
For your multi-mode question, do you have a design layout I can look at? (OptiSystem 12)