If you want to study reflections then OptiFDTD would be the right choice. With OptiFDTD you can define the fiber by its index of refraction profile and simulate an incident wave reflecting from the surface. This same functionality is not possible with OptiSystem.
Cheers