Thank you for your query. Unfortunately we are not very familiar with the techniques you describe below. We did however find a paper (attached) which discusses using FIR filters in combination with a low pass filter to generate duo-binary signals. Possibly this will help you with your design query.
Sorry for not being able to respond in greater detail.
We hope the enclosed information helps!