Proper benchmarking has always been hard – it is trivial to get some numbers, but not so if you are trying to receive relevant ones, that is, results describing the behaviour of the system under test rather than some of auxiliary setup.
One example is testing ASP.NET web sites performance on a developer machine running Windows. It is very easy to achieve pretty low results compared to other systems like Apache, and the outcome is tempting to make far-reaching conclusions about platform’s performance. For a less obvious example, blog post I’ve stumbled upon recently compared the performance of Web API project using synchronous and asynchronous code, having threaded solution blown out of water by the magnitude of ten.
However, such results do not signify a weakness of Windows/IIS/ASP.NET platform in general due to the simple fact that IIS running on non-server Windows is limited to pretty small number (3 or 10) of concurrent web requests, which naturally limits throughput of a web application quite heavily if the request processing time is not too small. It might not be a bottleneck if your requests are done in 50 ms or so, but if the idea is to simulate a long call to an external web service, then the results stop to be realistic very soon.
How to get the real data then? Of course one way is to use Windows Server machine for performance testing. However, it might present less than optimal feedback cycle when you want to work in a measure – adjust – measure loop. Then another solution might be to use self-hosting capabilities present for a long time in Web API and recently included in ASP.NET MVC as well. In my experience, such hosts are quite quick and do not have any artificial limitations even on desktop Windows.