According to matlab usage, DSP Components are designed such a way that you can either use build-in algorithms or insert any any others (i.e., your own) at each stage of the compensation process. However, the restriction is that you should use downsampling from optiSystem. As an example, you can take a look at the section “Inserting your own MATLAB algorithm” in the help documentation of the “DSP for 16QAM”.