Hi Fernando,
Our DSP Components are designed such that you can use our algorithms or insert any of your own at each stage of the process, the only restriction is that you should use our downsampling. See the section “Inserting your own MATLAB algorithm” in the help documentation of the “DSP for 16QAM” and the “DSP for QPSK” for further information.
In OptiSystem 13, which will be released in June or July, we expand the capability to include non-linear compensation as well as BPSK,8PSK,16PSK and 64QAM modulation formats.
Can you expand on your second question?