DDos on wowza

[Sergey_Zykov]

Good afternoon.

Guys, we have serious problems. A lot of time our wowza server is under ddos attack. We don’t know how protect media server. The attack focus only on hls(http) streaming. And few weeks ago I understand technology. It is http slow read ddos. On wowza we use token for streams and limit on server for users (700 connections). When ddos start in logs i saw that a lot of wrong connection (with incorrect token) “fly” on victim stream. And after few seconds I saw in this stream max hls connection (on server this number was 700 of course). And after in logs we see a lot of incorrect sessions. After few hours java heap (it on our production 10gb) full 100% and wowza down!

I try to combat with this problem, but anything didn’t help.

I used iptables. Used some rules which limit no more than 2 request from 1 ip address. It got better but in any case wowza down. ddos request pass on wowza. I used java module for wowza which reject all hls sessions if it’s more then 50 for example. But it didn’t save. All ddos request successfully passed.

I didn’t know how protect wowza. What should we do? Thank you.

[Soeren_Steinmann]

Hi, I can not help, but have similar experiences.

One problem in our situation was (and is), that the live stream is probably listed in some ‘live stream tv apps’. Some Apps with a large user base seems to shuffle the startstream or guide the users on the startscreen sometimes to ‘recomended’ streams… so on some days for sereveral hours they guide 100,000 users to a singe live stream. And the customer DO NOT invite or allow any of the Apps to do this!!!

We program a user connection limit. And eg all users over 1000 at the same time get

public void onHTTPCupertinoStreamingSessionCreate(HTTPStreamerSessionCupertino httpSession) {

httpSession.rejectSession();

}

But the Apps do a refresh/reconnect on the rejected connection! So this is like a DDOS attack in result.

We get over 1000 connection requests per second and Wowza show fast 50,000 connections for the server vhost, although any connection is simply rejected in the module !

[Soeren_Steinmann]

Thank you!

[Paul_Shields]

Hi,

Personally I’d recommend managing such attacks at the OS or network level via a firewall. For example, if on a Linux server then iptables can be used to block or throttle connections to defined ports - e.g. the following rule would allow up to 10 concurrent connections on port 1935 (Wowza default streaming port) from a single IP address, after which they would be rejected

iptables -A INPUT -p tcp --syn --dport 1935-m connlimit --connlimit-above 10 -j REJECT --reject-with tcp-reset

Alternatively (or in combination) you can mitigate the effects of clients who continually attempt connections in rapid sequence by setting a rate limit (this also protects against general DOS or DDOS attacks),

iptables -A INPUT -p tcp --dport 1935 -i eth0 -m state --state NEW -m recent --set 
iptables -A INPUT -p tcp --dport 1935 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 -j DROP

The above example will drop incoming connections for an IP address that makes more than 10 connection attempts to port 1935 within 60 seconds on interface eth0. Using iptables in this way rejects any unwanted connections before they reach Wowza and so is much more efficient than dropping them within Wowza itself.

Regards,

Paul

[Paul_Shields]

Hi,

I’m not really sure what else to suggest in that scenario. When receiving that kind of volume per second then iptables would seem to be the logical way to block or rate limit them before they reach Wowza.

Paul

[Sergey_Zykov]

Hi, I can not help, but have similar experiences.

One problem in our situation was (and is), that the live stream is probably listed in some ‘live stream tv apps’. Some Apps with a large user base seems to shuffle the startstream or guide the users on the startscreen sometimes to ‘recomended’ streams… so on some days for sereveral hours they guide 100,000 users to a singe live stream. And the customer DO NOT invite or allow any of the Apps to do this!!!

We program a user connection limit. And eg all users over 1000 at the same time get

But the Apps do a refresh/reconnect on the rejected connection! So this is like a DDOS attack in result.

We get over 1000 connection requests per second and Wowza show fast 50,000 connections for the server vhost, although any connection is simply rejected in the module !

Stoneman, thank you for quick reply!

Limit on application didn’t help too.

[Sergey_Zykov]

Maybe guys from Wowza can help? It’s not good if the software doesn’t have any protection. And any person can send a lot of request and down media server.

[Sergey_Zykov]

Hi,

Personally I’d recommend managing such attacks at the OS or network level via a firewall. For example, if on a Linux server then iptables can be used to block or throttle connections to defined ports - e.g. the following rule would allow up to 10 concurrent connections on port 1935 (Wowza default streaming port) from a single IP address, after which they would be rejected

iptables -A INPUT -p tcp --syn --dport 1935-m connlimit --connlimit-above 10 -j REJECT --reject-with tcp-reset

Alternatively (or in combination) you can mitigate the effects of clients who continually attempt connections in rapid sequence by setting a rate limit (this also protects against general DOS or DDOS attacks),

iptables -A INPUT -p tcp --dport 1935 -i eth0 -m state --state NEW -m recent --set 
iptables -A INPUT -p tcp --dport 1935 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 -j DROP

The above example will drop incoming connections for an IP address that makes more than 10 connection attempts to port 1935 within 60 seconds on interface eth0. Using iptables in this way rejects any unwanted connections before they reach Wowza and so is much more efficient than dropping them within Wowza itself.

Regards,

Paul

Thank you, Paul. But few messages later, i wrote that this rules didn’t help. Yes, I used rules by conection limit from 1 ip and no more conn from 1 ip till minute. But it didn’t help. I don’t know why, but attackwas from various ip and it was not so fast. Finally, in wowza i saw +50 connection every second. If I use limit module on server, all bad and good connections after limitation was dropped else I saw count of viewers for example 10k, 20k, 64k! And wowza down. With limitation module it’s work little more (several minutes) and in any case down!

This attack is only on hls url.

[Alexnader_Obuhov]

We had also got in such situation… Has anybody found solution?