10Gbits server , blok on 1,2Gbits

[marko1101]

hi,

we setup server with 10 Gbits card, and 10Gbits link.

linux centos 6.3 64bit
i7 950
24GB rama

wowza 3.1.2
java 1,7

only one publish live (RTSP)
for streaming live used only RTMPT port 80

we used tunning metode from wowza artical, also tunning linux kernel for 10gbit traffic
also from wowza forum , some sugestion.

server are testing and can reach 6Gbps traffic, test with test tool like iperf, netperf

we also do testing wowza tool , and reach 5.9 Gbits traffic.


but, we have problem when used real production, when traffic reach 1,2Gbps ... stream begine to blocking

if we calculate not reach over 1,2gbps, stream work OK ... 2000 vievers, with 200k stream .. all wok ok

but, is we have stream 1000kbps, when we reach 1200 vievers, stream begin to blocking.

i was see, in this moment, when streem blocking, i also can not access to woza stremamanager page, on port 8086 ... is to slow to open ... ALL other services like iptraf over SSH, webmin, conecting to another SSH ... all on same machine , work

only we have problem with WOWZA, probably JAVA service for WOWZA proces.


did anyone have/see , somthing like this ?

how to find , where is bottleneck ?


thanks,
marko

[rrlanham]

Marko,

I assume your test with using the Wowza tool that reached 5.9gbs was the Wowza load test tool and that the client machines were nearby on the same LAN. So it is definitely, as you said, a bandwidth bottleneck somewhere, not Wowza configuration and tuning. So it must be further up, some switch perhaps or where your network connects to the internet. Search for "find bandwidth bottleneck" for some ideas.

Richard

[marko1101]

hypothetical ...
if I start two WOWZA instance on same machine, and reach 2,4gbps (2 x 1,2gbps) ... what can be problem ?

btw, if it is bandwidth bottleneck some where on network, and if i start two instacne of WOWZA on this machine, it should not have to cross 1,2gbit total, right ?


btw, we have many WOWZA server with 1Gbps card/link, with 4000 viewers/stream 200k = 800mbps , all work ok.

we want to move to a larger bandwidth per server, and we came across on this problme.

[rrlanham]

Marko,

Are you saying you don't think it is bandwidth bottleneck now? Do you think it is a problem with Wowza? You report a load test that yielded 5.9gbs output from Wowza, which is outstanding.

If you put two instances of Wowza on a server with 10gbs nic, you can get better utilization of that network capacity, but if there is an upstream bottleneck that will just be twice as much data that is not getting through, so it won't solve that problem.

Richard

[marko1101]

yes, i dont think this is bandwidth bottleneck ... we can create 5,9gbps with wowza tool

i think, wowza tool make diferent hardware load ... becouse of this i think this is WOWZA problem , or JAVA problem.

i think tis becouse, only WOWZA blocking ... btw, on same server, apach on port 81, work ok, also FTP work ok.

web/ftp work ok, when wowza makeing 1,2gbps, and blocking

if this is bandwidth bottleneck , then web/ftp , also may need to have problems ... i do not have problem with web/ftp on same server.

[marko1101]

btw, like i sad ...

we used only RTMPT port 80

on 1500 ESTABLISH connection , we have 3500 Time_wait:

netstat -ant | grep 80 | awk '{print $6}' | sort | uniq -c | sort -n
3 LISTEN
7 SYN_RECV
19 LAST_ACT
52 FIN_WAIT1
607 FIN_WAIT2
1345 ESTABLISHED
3571 TIME_WAIT

this is not good ?!



now we are make oniy RTMP 1935 in player and on wowza, and we will wait to see, what will happend

now we have this:

netstat -ant | grep 1935 | awk '{print $6}' | sort | uniq -c | sort -n
1 FIN_WAIT1
1 LISTEN
1 SYN_RECV
129 ESTABLISHED

[rrlanham]

Actually the Wowza load tool creates actual traffic and load on the server and network.

Maybe the difference between load test results and now is RTMPT. Why RTMPT?

Richard

[rrlanham]

It's a stretch that RTMPT would be that much of a difference. The load test results being that good on lan still points to bottleneck upstream, as you said before. Look in that direction

Richard

[marko1101]

problem are RTMPT, port 80 tunneling , this is problem in wowza or TCP seting on server.

now we used RTMP port 1935, all work fine ... 2000 vievers, stream 1,2mbps, total 2,2-2,5gbps ... CPU 12% ... ALL WORK OK on port 1935

netstat -ant | grep 1935 | awk '{print $6}' | sort | uniq -c | sort -n
1 CLOSING
1 LISTEN
1 SYN_RECV
4 LAST_ACK
11 FIN_WAIT1
18 TIME_WAIT
2052 ESTABLISHED

there is not to many TIME_WAIT, like we have on port 80.


why we have to many TIME_WAIT on RTMPT ? ... this is right quesiton.

if you have answare on this question , then you have solution for this problems.

[marko1101]

wowza test tool work with RTMP ... becouse of this we was able to make 5,9gbps with wowza tool.