first of all you must have a clear idea about about the fading channel which you are going to implement in MATLAB which is must more dependent on the distance of transmission (i.e. if you consider the mobile environment what should be the size of the cell) the go accordingly for the criteria such as if there is line of sight path or not or both may exist. my suggestion would be to implement in MATLAB first and then go for the respective modification for optisystem. Sorry to say that I do not have any kind of sample file for such a channel i will let you know when I will design for the same. RoF is one of my topic of work at present.