You are right, FDTD will calculate the transmittance along the detector plane in the near-field. To account for specular transmittance you need to calculate the far-field transform (you can do this by exporting your detector plane result into f3d and then by using the far-field transform tool of the OptiTools toolbox). You can then use the far-field angular distribution to determine the power transmitted.
A similar approach was used in the following (a bit outdated) example :
To simulate an unpolarized light source, the only possibility here is to run 2 simulations:
– 1 simulation with X polarization (or TE in 2D)
– 1 simulation with Y polarization (or TM in 2D)
and then calculate the average of both simulations.