Heitor,
The issue here then must be with the amount of memory your computer has. 1.7 GB is pretty low, but there might still be away to simulate the project. If you get rid of all the extra visualizers and Data Monitors on the output ports of each component only keeping those that you need you should be able to simulate it. Use the Monitor Tool on the right toolbar of OptiSystem to delete the data monitors.
I tried you base project and it takes around 3 GB of memory.
Regards
