This issue is explained by the first few lines after the figure in the paper. OSNR refers to signal to crosstalk ratio. So the author has once measured BER without crostalk, and then adding crosstalk of different levels, they have measured BER again, which became worse this time. In order for this BER to become equal to when no crosstalk was there, they have added more input power. The value of the added input power for which the system with crosstalk acts as good as the system without crosstalk (in terms of BER) is the power penalty of the system.
Regards