you need to write the Matlab code for the SQRT functionality and load it to the Matlab component in OptiSystem. It should be easy. The incoming signal needs to be read by the Matlab code, process then pass it to the output port of the Matlab component. There are few examples in OptiSystem Example Library about the commands used for that located at the following link.
C:UsersUSER NAMEDocumentsOptiSystem 18.0 SamplesSoftware interworkingMATLAB co-simulation