The benchmarking scheme has been devised now and is giving good results that are not biased by the use of front-end caching proxies that some hosts are using on demo stores.

Whilst we do not disagree with the methods of using a front-end proxy, we find that hosts should only be demonstrating their demo store on a platform exactly the same as what they are willing to offer.

Hosts will be able to modify their profiles and demonstrate various Magento packages they offer – but must submit a demo store for each to suit, so we can accurately benchmark each to show the end user what benefits can be seen in performance (if any).

The benchmark

Following the advice of users on the Magento Commerce forum, a method has been devised to build a good performance profile of participating hosts.

Based on a sample fixed list of 50 urls:

20% search requests 40% homepage requests 15% category requests 25% product requests

All the requests are non-sequential or repeated, so a different page is accessed on each request.

As testing will be based from a dedicated server of ours based in Manchester, England – to ensure fairness to international hosts, 3x ping requests will be made to the demo store and the double the latency observed multiplied by the number of requests will be subtracted from the overall time. This will remove the requirement for overseas testing.

Based on levels of 1, 2, 5, 10, 20 and 30 concurrent connections, there will be two tests. Both tests are custom written and have been designed to reflect real-world load not artificial testing such as siege/ab. The first test visits 15 urls and downloads all content on the page (images/css/js) with the above concurrency variants. The second test is to gauge the PHP processing speed by only requesting HTML, this is carried out over 50 urls with the above concurrency variants. All tests will be performed at 05:00 am GMT.

All participants will be contacted prior to benchmarking to ensure their willingness to participate and to add our farming server to their allowed IP lists.

Each day the submitted url will also be pinged 4 times, an averaged formed and logged. This will allow an uptime record and show periodic latency relative change.

The tests will be performed every 4 times per month, per host.

Based on the above, a profile will be auto-generated to represent the results, such as the MageBenchmark example . The host will be able to modify certain fields to market their product.

We do encourage all hosts to participate!