Dear Awanish,
the best option to model Nakagami fading channel in OptiSystem is to use the Matlab component where you need to identify the number of input and output ports and define the port signal type. This can be done in the component. then you could modify the Matlab code to allow proper communication with OptiSystem. There are many examples on using the Matlab component in OptiSystem Example library found at the following location C:UsersUSER NAMEDocumentsOptiSystem 20.0 SamplesSoftware interworkingMATLAB co-simulation.
Regards,
Ahmad