When publishing H264 using the flash player, and then streaming out with RTP I am seeing some very strange packets. Generally I see STAP-A packets which for the most part I am not expecting. Within these STAP-A packets are some unexpected and invalid NALU types. For instance 6, 9, 38. When publishing using FMLE I will only see NALUs of 1 or 28 (FU-A) (Or 1 and 5 once depacketized).
It is not so much the STAP-A packet that I am concerned with. It is the data in the STAP-A that does not correspond to a NALU of type 1 or 5 (I doubt a key frame would be placed in a STAP-A). On occasion our decoder will not like the NALU from the data in the STAP-A (Just ignoring the NALU type). On other occasions it is fine. I am guessing at this point that it could be AMF data in the RTP packets (the flash clients are using shared objects). I have not taken a hard look at the data in the NALUs that are invalid or 'strange'. Generally they are very small (2 to say 24 bytes).
Also, if I ignore these packets then our decoder never works (only did a few tests). Any ideas you might have as to where to look, or how to approach looking at the RTP payload for issues would be appreciated.
-- EDIT 1 --
I will have to do a better analysis. If I ignore STAP-A and dump any packets with less then 64 bytes it works fine. Obviously not an ideal solution. I will post back again with more information.
-- EDIT 2 --
Here is what the data looks like.
All FU-A packet data is fine. The flash player seems to generate fairly large P frames which get placed in FU-A's. The same for the key frames. Anything in the STAP-A packets is "garbage". Is it possible that you are sending AMF/Shared Object data in STAP-A?