The Mode solver can be very reliable! Remember though sometimes the user may need to make some changes to the simulator to give a more accurate result much quicker.
Scalar mode solvers will approximate that the two transverse components are independent and thus will find less modes. For example in a optical fiber a scalar mode solver will calculate the LP modes but not the EH and HE modes that constitute the LP modes. These will be calculated by a vector method. Generally if polarization effects are important you should use a vector method.
Whether a complex or real option is chosen will determine whether the mode solver will calculate a complex propagation constant which includes modal loss information.
Since a mode solver generally just solves an eigenvalue equation, it is hard for it to know the number modes that a waveguide should support. This option will tell the solver to assume that waveguide will have x number of modes and try to solve for all of them.
Regards