Hi Muhammad,
I recommend setting your Bessel filter to order “4”. This should give you steeper drop off. I also suggest setting the cut off frequency to around 0.7*bit rate.
I also noticed that the y-axis for the RF spectrum analyzer is in dBm (not dBm/Hz). To set the y-axis to spectral density units please check the parameter “Power spectral density” under the Graph tab of the Component properties panel. Your power levels may be OK! Cheers.