PHP CMS comparisons #3 - Size and performance
Some people in the Statamic community have commented that my previous tests were unfair (or rather; dumb and crap).
Some arguments pointed towards 1: file size doesn’t matter, and 2: Statamic had «no caching and file watcher was on».
I'm not out here to talk any system down, and only try to fairly make comparisons for my own sake. But in any case, lets dive into the arguments:
1: File size doesn’t matter?
This is not about technical advantages or disadvantages, and people might see this differently than me, but my perspective is like this:
While file size might seem irrelevant to privileged users (often us developers), sitting on fairly new hardware and unlimited fiber-speed internet, its easy to forget that this is not the case for the whole world.
As a simple example comparison of the Kirby install of 6mb vs Statamic at 86mb: At 1 million installs it’ll be 6TB vs 86TB. This will likely be downloaded on local machines, but also on a server. Lets say 2 places (12TB data transfer vs 172TB) Maybe you even have multiple environments and separate installs.
Google estimates roughly 810 million wordpress websites as of 2023. Of course there aren’t as many Statamic, Craft, or Kirby sites. But still, why not lead by example? Data centres alone are responsible for 2.7% of the EU’s electricity demand. By 2030, their consumption is expected to rise by 28%.
The way I see it, at scale, filesize most definitely matters.
2: How about caching?
Just to make that clear from the bat: None of the CMSs in my previous test used caching. It was all fair game on even terms.
But ok, an argument could be made that one would anyway be caching, so therefore the tests were not realistic.
Even though I don’t personally agree to this, and don’t believe the majority of sites use caching, I still got interested to see how it looks if every system uses caching.
Some notes before results
Wordpress was only tested uncached and cached with a plugin. https://www.boldgrid.com/w3-total-cache/. I didn’t find a "native" way to activate caching (without a plugin). Probably just me being dumb. Let me know if you know how!
The three states tested on all the other systems were 1. uncached, 2. cached with the native simple caching mechanism: Craft CMS with {% cache %} tags. Statamic with file watcher off and cache strategy "half". And in Kirby with simply activating cache in the config.php. 3. Full static caching with NGINX rewrites.
The "best case" caching serving static files (Statamic full cache, Craft CMS with Blitz, Kirby with Staticache and Wordpress with w3 total cache" are almost identical, and kind of useless data. All it does at this point is serve html-files without touching php, so they are just included to see the performance gains of that vs uncached and "simple" caching in each system.
All numbers have been averaged based on three tests runs.
I've done tests with OPcache activated and deactivated, apart from on the statically cached examples.
Takeaways:
OPCache makes a big difference.
Kirby cached and uncached are less different than other systems. But regardless, with OPcache turned on Kirby is getting even close to the statically cached sites. Pretty impressive.
Statamic uncached is quite bad. Seeing that it's the default, I wonder how many web pages are running in production like this.
Craft CMS would really benefit from merging Blitz to core, to be able to offer what Kirby and Statamic does without third party plugins.
Using Laravel Forge, performance was «as if» OPcache was active before actually activating it on the server. Turning it on and thereafter off substantially worsened the results. Is OPcache on by default even though it says it isn’t?
This is all just one aspect of CMS differences, and doesn't at all mean that one system is overall better than the next. It just measures how much baggage each system brings along.