First off, the BER is ~0.5 because the data is being read incorrectly. Even guessing at the bit every time you would be right half the time. On average the worst BER you can get is around 0.5.
The constellation diagram I am getting is rotated, yet still distinct levels. The rotation comes from phase delay of the signal with respect to the local oscillator. So I believe the delay is causing the majority of the problem.
I think the next step is to try setting the delay compensation manually (no DSP) using the Decision component. The multimode fiber report in the component properties will explicitly tell you the delay calculated.
The Matlab component is definitely an option but I’m not certain if it would be easier than trying the step above.