Please see the examples provided in OptiSystem 12 Samples\Advanced modulation systems\QAM systems. We have several examples of 16-QAM systems for long distance.
For 64-QAM, we are planning to include examples in our next release (OptiSystem 13) for June/July 2014.