The increase in error when power reaches 10 dB is expected due to nonlinearities. I reduced the bandwidth of the filter at receiver side to 0.7*Bitrate and now the plot seems OK. I believe it can be reduced even further. So I think maybe the reason for the rise in plot at -5 dB laser power has to do with noise that is not filtered.
Regards