The basics — Making Magento run fastThere are numerous Magento speed recommendations on the Internet and in whitepapers, however these are of widely varying value. Our extensive in-house testing and experience with thousands of Magento stores both lead to the clear conclusion that there are only several factors that make a measurable (let alone significant) difference. It’s important to distinguish between factors that halve page load times, and those that shave an average of 2ms off a 100ms HTML generation time. The 2ms is important when serving millions of requests a month, but won't make a slow site fast.
Magento speed — the Big Five (what works)
- CPU: Choose a fast webhost
Slow CPUs are the reason why cheap hosts run Magento so badly. Without fast CPUs, the store is going to be slow. CPUs can be compared on a benchmarking site to see how fast they are (at the time of writing M.D.G. IT uses Xeon E5-2680 v2). GHz values are not an indicator of absolute or relative speed — a 3 GHz processor from 2005 is going to be incomparably slower than a modern 2.6 GHz processor. The critical factors are:
- having fast CPU cores
- having access to cycles on those CPU cores. This is the problem with oversubscribed shared servers, for example, which are designed for lightweight applications like Wordpress: individual websites don't have access to enough CPU cycles to generate pages quickly enough. This is also an issue with AWS, where an instance's access to CPU cycles is restricted.
- Code: Don't slow down the site with bad extensions and bad development
Magento, on a fast CPU, without any other optimisations, runs well. Tweaks can make it run faster, but never to the extent that bad code makes it run slower. The most significant factor in Magento speed problems after CPU speed is problems introduced by developers or extension installers during site customisation or updates. See our separate Development Guidelines article.
- Caches: Never disable caches on production
If a store is in production, every cache must be green in the Magento backend (System — Cache Management). If a Rewards Points extension or a new menu block don’t work properly with the caches on, it is the extension that should go, not the caches. Magento is not designed to run in production (where multiple simultaneous requests are likely) without caching, and disabling any of caches in the backend is asking for trouble.
- Configuration: Run PHP and MySQL fast.
This means mod_php (not suexec, no open_basedir.) or FastCGI, using APC (PHP 5.3), OPCache (PHP 5.4+) or another opcode cache, and setting up sane query cache and buffer pool sizes in MySQL. There are lists of 20+ other MySQL variables that are touted as improving Magento page load performance; however we have yet to find any measurable performance increases that these represent on fast hardware. In many cases suggested MySQL configuration values actually made sites run slower than previously.
- More caching: Move the load off Apache
This isn't a substitute for the above (see more on Varnish below), however Varnish is the difference between a fast site and a site that runs as quickly as possible. See here for a Varnish demonstration site. Varnish caches as much as possible of each page when it is first loaded, only fetching personalised content (e.g. cart contents) from Apache. Likewise, a CDN caches images and other static content, and delivers these directly to visitors from remote servers.
What doesn't work
- Focusing on server settings as a response to problems caused by code
No amount of caching or tweaking will return normal performance to a site that has significant development issues. Problems introduced in code need to be fixed there. The following is unfortunately a common pattern:
- A client finds a developer, often offshore to reduce cost, who is likely to be inexperienced in Magento, whether or not they admit the fact. There are always exceptions, however good Magento developers are not likely to be looking for work on freelancing websites for $20 / hour. The problem is that while it's likely that the first Wordpress site a PHP developer builds is going to work well, it's even more likely that the first several sites a PHP developer builds in Magento are going to have significant problems.
- This developer typically works alone, without access to senior developers who can give advice about how to build in Magento to keep the site fast. No source control is used.
- Speed is not tested and overall performance is not mentioned at all during the site build.
- Speed issues become obvious shortly after going live, when the site can't handle moderate traffic, or when real products start to be created in the catalogue.
- When faced with the tedious process of removing customisations one by one until the cause of the speed issues is identified, developers start suggesting PHP and MySQL configuration values, using Nginx, or using grossly overpowered servers, in the hope that these will restore normal performance
- Session store in database. The theory is that the database can cache this in RAM. The reality is that unless you have incredibly slow disk IO, the access time to store and retrieve cache information is irrelevant to most installations.
- Session store on TMPFS. As above, the idea being that all cache info is stored in RAM. Other than wasting RAM that can be better used elsewhere, this made no measurable difference in our tests. Writing and retrieving data to and from cache isn’t the problem - the problem is the CPU churning to generate the page / cache contents in the first place.
- MySQL configuration. The bottleneck in Magento page loads isn't database access, it's PHP. The two configuration options listed above do speed things up, but even these are single digit percentage increases on frontend pageloads (effects on indexing and other backend processes tend to be more significant)
- Nginx. Benchmarks comparing Nginx favourably to Apache are plentiful, however these invariably compare serving high volumes of static pages. It's not Apache causing a CPU core to run at 100% for 100ms while a Magento page is generated, but PHP. Nginx still needs to make a request to PHP FPM for the actual PHP to run. Our tests have shown quite conclusively that properly configured Apache / mod_php or PHP-FPM runs as fast as Nginx / PHP-FPM.
- Considering images. The number and size of images on a site has no relation whatsoever to how long it takes to generate HTML. HTML loads first, the page then start rendering, and then images are loaded. If it takes too long for the page to start loading this is not because of image size. See here for testing HTML response times.
- Blaming catalogue size. Larger catalogues tend to highlight problems in code. While code problems can cause larger categories to load significantly slower than small ones, the problem is still in the code. We've tested catalogues up to 1 million SKUs on PowerBase VPSs, without any noticeable performance issues in the front end when using the default Magento theme, without extensions.
- Blaming the theme. There is no reason why a good looking, well designed theme should be any slower than the Magento default theme. If switching to a custom theme slows down the site, this is because the implementation of the theme has problems, not because a particular design would by its nature have performance implications.
Why can't we just put the site on a bigger or faster server?
PHP is single threaded. This means that even on a server with 40 CPU cores, a single Magento pageload request can only run on one of them. If the CPUs are already fast, adding extra server resources allows a site to handle more requests at once, but it does not speed up the time taken for an individual page to be generated (unless the server was previously so overloaded that individual page load performance was affected as a result). While upgrading from shared to VPS hosting does make a significant difference to single page load times, upgrades to larger or multiple VPSs are about adding total capacity and reducing server load, not about making individual pages load faster.
Can all problems be fixed with Varnish
Full page caching is great, but it shouldn't be considered a fix for bad development. The underlying uncached performance must be acceptable, otherwise the store will still have problems:
- Checkout is critical, and needs to be fast. Because checkout involves significant interactions that can't be cached, the underlying performance must be fast.
- The cache will be invalidated by indexing or product changes — at some point it needs to be rebuilt. The first pageload for each page (i.e. each category, each product) will in the first place be completely uncached. This load time should be as fast as possible because a human will likely be waiting for this. Caches are also per browser type, so someone loading a page in IE won't charge the cache for a Firefox or Chrome user. Even Varnish warm scripts don't normally deliver completely cached first page loads.
- Varnish won't help with crawlers, which tend to request thousands of pages in a short period of time, many of which are likely to not be cached
A quick and precise way to identify performance bottlenecks is to investigate and profile the website code during execution.
Profiling website PHP code
This level of diagnosis can be accomplished by your developers using a profiling tool like NewRelic. To start making use of this tool, you will need to register an account with NewRelic for their Application Performance Management (APM) service. Once registered, M.D.G. IT can install and configure NewRelic on request by opening a support ticket through the Client Area and providing us with your NewRelic license key.
Once NewRelic is configured and installed on your server, the NewRelic PHP agent will begin to send data to their APM service, populating your NewRelic dashboard with PHP / MySQL runtime information including transaction times, resource usage, error rates, and the specific processes involved in these. This granular view will provide your website’s developers with clear indications of what code needs review. As PHP profiling can add a ~30% increase in page load times, M.D.G. IT recommend performing these tasks during non-peak times and disabling the NewRelic APM service once usable data has been collected.
For a thorough guide on what to look for in the NewRelic APM overview, please refer to the following.
Summary and tips:
- Fast hosting is critical, CPUs must be fast. Core speed is critical, PHP is single threaded. This means 2 fast cores are better than 6 slow ones.
- Don't slow Magento down in the first place. Hire experienced developers, and convince yourself that changes to the site aren't causing performance issues.
- Benchmark regularly during development.
- Extensions are not your friend: Magento doesn't perform better as more extensions are installed, it gets slower. Shoppers want a fast, streamlined shopping experience, not functionality at the expense of one second increases to pageload times. Benchmark extensively before and after extension installation.
- Magento caches must be on in production, without exception. Code must work with caches on, disabling them to allow for badly written extensions / code is devastating to performance.
- Get a VPS if you can afford it. VPSs run PHP in the fastest possible way, without sacrificing speed for security as on multi-tenant systems (aka shared hosting). In addition to this VPSs can run opcode caches, which helps dramatically, and can also run Varnish, taking advantage of full page caching.
- Ignore generic tips to use memcached, SQL variables beyond query cache and innodb buffer, session and cache on TMPFS, until your HTML generation times are around 100ms. Once at this level, any server side changes need to be benchmarked to ensure that they're actually helping.
- Minimising individual page load times is the first step to scaling.
- Don't set indexes to update on save, run these as a cron overnight instead.