Hi Zheng,
I have checked you project,
I think I got the error.
note that the output of PIN photodetector is time domain signal. so you can not multiply time domain with transfer function in frequency domain!!
So the solution, you have to
convert the output of PIN to frequency domain then multiply by H(omega).
hope that helps,
thanks
Mohamed Abouseif