Hi aadilqau,
As they mention before, there is no any formula or criteria that shows how can we choose or calculate sequence length or samples per bit. As far as I know, for the sequence length, you have to increase it as much as you can to get good performance in terms of BER for example. and for the sample per bit you can increase until the sample rate can cover the spectrum of your design (for example: if you have RF in range of 40 GHz, you need to increase the sample per bit until the sample rate be more than 80 GHz). however the RAM of your computer is also a factor for increasing both of them, so there is a limit of increasing these two parameters in order to avoid system shut down.
Kind regards
Mohamed