OptiSystem can do direct detection of the RF signal, as shown in the examples attached. If you want to do transistor level design of RF receivers our product OptiSpice is perfect for this application. To measure the BER, you can send the optical signal to OptiSpice from OptiSystem, simulate your desired RF receiver and then return it to OptiSystem to finally calculate the BER.