If you are not referring to Shannon limit (which I don’t know if actually can show the limit), then I don’t think there is any way to determine any limits for this. Of course, you can find papers regarding comparison of these modulations performances in different systems, but this doesn’t mean limit I think.
Regards