We are taking a live rtsp input (720p 5fps) and transcoding to 2 resolutions - 720p (with KeyFrameInterval different than input) and 360p. I have tried encoding with mainconcept encoder and NVENC encoders (on NVIDIA grid K520) at sufficiently high bitrates, but get very poor quality video output. I observed video artifacts at exact same locations when encoded with both mainconcept and NVENC encoders, which makes me believe the decoded video output (just before encoding) must be poor.
Is there anyway i can tweak the video decoding parameters to improve the video quality?
Do you see any messages like this in the access log?
TranscodingSession.updateBehindFilter[transcoder/_definst_/myStream]: Video behind filter state change. New state: SKIP1FRAME
TranscodingSession.updateBehindFilter[transcoder/_definst_/myStream]: Video behind filter state change. New state: ALLFRAMESON
If so, this means that machine is not keeping up with transcoding. Take a look at the Transcoder Benchmarks to get an idea of server capabilities and limitations.