From my understanding of the LMS algorithm it can only compensate for dispersion effects from fibers. Have you tried disabling all of the other nonlinear effects? And even attenuation?
Furthermore, the signal generated by the Matlab link creates a very structured constellation diagram. It seems like every point is on some sort of grid. The signal does not look truly random.