Wednesday, 16 May 2007

Performance Tests

Every now and then, it rears its head on one or other IRC channel. Someone trots out one of a number of web sites that purports to compare like with like, hammering the crap out of various implementations of a web app in Rails, various Perl frameworks and not-frameworks, J2EE, PHP, etc etc, and attempts to draw meaningful conclusions from the results.

Setting aside, for the moment, brian d. foy's tongue-in-cheek demolition of benchmarking as a valid tool, what do we learn from these?

Actually, mostly, we learn that a single instance of almost any web app written in just about any language on semi-decent hardware, pounded on by JMeter, ab or whatever will cough up well over 100 hits/second without actually breaking into much of a sweat.

Let's stop right there for our first remedial maths class, OK?

100 hits/second.

That's 6,000 hits a minute. 360,000 hits an hour.

Eight point six million hits a day. And change.

And that's page views - c'mon, you don't serve your static images out of of your webapp framework. That's a dickens of a lot of traffic. To give you an idea,'s record month while I was there was a quarter of a billion page views... that's about 8.3 million a day on the biggest single-sport website on the net.

You're expecting that amount of traffic? Man, do you have delusions of grandeur! Paying for the bandwidth is going to be a much bigger worry than whether your web app framework can cope.

Stick an ad on every page, from Y!, Google, whoever. A week's traffic, tops, and you can afford another server. Hell, you can afford two and a load-balancer. And quite frankly, your rate-determining step is almost certainly NOT how fast your underlying development framework can generate the pages, but how well-written your code and your DB are, how cacheable the pages are, how big they are, and how laggy your average client is. One missing database index can turn a 0.02s query into a 3s query.

Get over it, folks.

[composed and posted with ecto]

Technorati Tags: , ,


brian d foy said...

For some reason, you say to set aside my comments on benchmarking, but then use the same arguments I make to say to ignore them. People don't understand benchmarking or the results they get, and then draw inappropriate conclusions from the numbers they see, finally making incorrect decisions.

You seem to be saying the same thing.

So, why bring me up at all just to dismiss what I said?

Mike Whitaker said...

Fair comment. :)

I was distinguishing between your (excellent, I might add) talk which seemed to me to be largely covering the problems with tests on a more micro scale (why your tests are not valid for me, why last Tuesday's are not necessarily meaningful today), as opposed to benchmarks which are frankly irrelevant on just about any scale.

Dan Dascalescu said...

I've seen this argument before: "100 hits per seconds means 8 million hits a day".

The flaw is that traffic is not evenly distributed. You get 500 hits per second in the peak morning hours and 10 in the afternoon. Does your framework handle 500 hits per second?