From what you are describing, it sounds like the server hardware may be getting overloaded. When comparing Windows server editions to Linux, there shouldn't be a lot of difference unless there is a lot of other processes running in the background. Running Wowza from the desktop on either OS will use more resources because the desktop consumes a lot of resources
The following could cause what you are describing and would be the same for each OS as well as for Wowza compared to FMS:
Not enough network bandwidth available for the number of connections x stream bandwidth.
What is the stream bandwidth and how many connections before the problem occurs.
What bandwidth is available? Is the network adaptor running at full capacity? It has been know for network adaptors to auto negotiate to a lower value if there is a problem with the cable or switch.
Not enough CPU cores available to spread the load.
How many cpu cores does the server have? Is it a dedicated real server or a shared server or virtual server?
Not enough memory available or allocated which will cause more frequent garbage collection to occur.
How much memory is available? How much is allocated to Wowza?
A badly written module could also cause problems if it is delaying worker threads too much. It might not be noticeable with a few connections during testing however lots of connections could reveal problems.
Is there any 3rd party or addon modules being used?
Answers to these questions may help to identify the cause of the problems.