The results you are showing are the results of a divergent FDTD simulation. Typical reasons for this is mesh that is too large or a material defined such that it has gain.
1. I know you say you are using less than lambda/10 but that statement is based on your central wavelength. How broadband are we talking? What is your smallest wavelength?
2. What materials are you using in your structure? If you defined a dielectric with real and imaginary components what is the sign of your imaginary component? In OptiFDTD materials are defined as ~n = n – jk. Positive k is loss so the imaginary component must be negative for loss.
Scott