Trellis coding modulation technology can improve the signal-to-noise ratio, reduce the transmission power, and lower the bit error rate without changing the bandwidth of the information transmission spectrum. The realization method is to divide the constellation map to form a subset and gradually increase the minimum Euclidean distance between the signal points in the constellation map. In this paper, the demodulation performance of 8PSK and 16QAM-TCM modulation techniques is simulated on the simulation platform of a relay satellite communication system developed by the project team. The simulation curve shows the relationship between the required bit error rate and signal-to-noise ratio. The simulation results demonstrate: ① under the channel conditions of an ideal channel, I/Q amplitude and phase imbalance, amplitude and frequency characteristics, group delay, phase noise, power amplifier saturation, if the required BER is 1E-7, compared to 8PSK modulation technology, the signal-to-noise ratio of 16QAM-TCM technology saves 8.85 dB, 9.04 dB, 8.45 dB, 10.2 dB, 8.5 dB, and 14.6 dB, respectively; ② under the channel conditions of non-linear, correspondingly along with increasing of the signal-to-noise ratio, the BER of 8PSK modulation technology fluctuates around the order of 1E-3; ③ if the required BER is 1E-7, compared with the ideal channel SNR simulation result, the non-linear channel SNR simulation result of 16QAM-TCM technology loses 4.8 dB.