In Inter satellite communications, antennae aperture diameters have always been an important factor as it decides the cost effectiveness of system. System payloads decrease with smaller size of antennae reducing its weight and hence launching cost.
There have been a lot of work in this arena.
There is no such constraint limiting the size of antennae.It depends upon beam divergence, whether your system is unidirectional or bidirectional, and the distance the signal is traveling as size of antenna at receiving side is usually taken more than transmitting side(due to beam divergence reasons).
A lot of work is done in arena. As per my knowledge the results achieved till now is 15 cm or 150mm. Rest it depends upon how long you want link to operate.
In normal system it have reduced by various parameters and trade-offs from 50cm in earlier researches to 15 cm in latest ones. Find the attached links.