Here are diagrams of the the two networks. They look very similar to the diagrams in your attached paper. It is hard for me to say why they are getting better dispersion compensation with the compensating fiber than the fiber bragg grating. I don’t see a design using gratings.
On the other hand, it is probably because you are using the Ideal Dispersion Compensation on FBG component, which is probably not the model the referenced authors used in their FBG design. If you have access, look at the referenced paper given below, I believe it is the paper the authors of your paper have referenced. They mention they used an original apodization technique. Once you find out what it is you can try to model it with the Fiber Bragg Grating component. For this component, you can set the “Apodization function” to user defined to get a better match.
Sa, L.M.; Nogueira, R.; Andre, P., “Simplified technique for the design of multichannel dispersion compensation FBG,” EUROCON – International Conference on Computer as a Tool (EUROCON), 2011 IEEE , vol., no., pp.1,4, 27-29 April 2011