Static Page Caching with Craft CMS

This web­site just got a whole lot faster, because pages are now sta­t­i­cal­ly cached via FastC­GI Cache. If you want to take your web­site from annoy­ing­ly slow to fast & respon­sive, noth­ing will get you there quick­er than sta­t­ic page caching.

And this arti­cle is going to explore how you can do it, too. From response times that make peo­ple smile, to being able to han­dle the ​“thun­der­ing herd” of mas­sive traf­fic, noth­ing beats sta­t­ic page caching.

If you look at the response in Chrome dev tools, you’ll see the x-cache: HIT header:

This means that the page was served up by the FastC­GI Cache, and PHP, Craft, and our tem­plates were nev­er even touched to make it happen.

Caching in gen­er­al is just keep­ing data around that we need quick access to. We’ve dis­cussed Craft’s built-in tem­plate caching in the The Craft {% cache %} Tag In-Depth arti­cle. Craft’s tem­plate cache is a frag­ment cache, where chunks of data are stored in a data­base for quick retrieval.

A full page sta­t­ic cache by con­trast stores the entire ren­dered page as a flat file on disk. 

So why would you want to have a full page static cache? Performance, but more importantly, concurrency.​

Full page sta­t­ic caching is incred­i­bly per­for­mant because when a web­page is request­ed, the HTML is just returned. There’s no com­pu­ta­tion involved in build­ing the page.

When a thun­der­ing herd hits your web­site, it can han­dle the traf­fic eas­i­ly, because the cost of each trans­ac­tion is very low. For a dis­cus­sion of why you should care about per­for­mance, check out the A Pret­ty Web­site Isn’t Enough article.

Sta­t­ic site gen­er­a­tors are pop­u­lar for this rea­son, but you often have to sac­ri­fice the flex­i­bil­i­ty that data­base-pow­ered CMS sys­tems such as Craft CMS offer you. A sta­t­ic caching lay­er in front of a data­base-dri­ven CMS offers you the best of both worlds.

One pop­u­lar solu­tion for full page sta­t­ic caching is Var­nish; but while it offers a flex­i­ble solu­tion, there are some down­sides in terms of no native https sup­port (you need to set up a proxy), and it involves installing and con­fig­ur­ing anoth­er lay­er of software.

FastC­GI Cache by con­trast is avail­able by default in any Nginx set­up, is easy to con­fig­ure, and actu­al­ly per­forms bet­ter than Var­nish. It’s not the right solu­tion for every case, but it works remark­ably well for most.

There are also some plu­g­ins avail­able for Craft CMS that offer sta­t­ic page caching, but it’s a tricky thing to get right, and in gen­er­al I pre­fer to use dae­mon-lev­el ser­vices for this type of thing for depend­abil­i­ty reasons.

Not every caching sys­tem is right for all sce­nar­ios, though; pick your tech­ni­cal solu­tion based on the prob­lem you’re try­ing to solve. For more infor­ma­tion on alter­na­tive, check out the Thoughts on full page caching in Craft arti­cle from Josh Angell.

Link Should you use a full page static cache?

It might sound like a sta­t­ic page cache is only for very high-end web­sites that need to han­dle mas­sive amounts of traf­fic. While it’s true that this is a great appli­ca­tion for a sta­t­ic page cache, it is also an excel­lent thing to use when resources are sparse.

A small, inex­pen­sive VPS can han­dle a sig­nif­i­cant amount of traf­fic if a sta­t­ic page cache is used. This is because the cost in serv­ing a web­page is very low, and thus the hard­ware pow­er­ing it can be as well.

This makes static page caching a fantastic choice for websites with budgetary constraints as well.

Here’s a sim­pli­fied view of what the flow looks like when a request comes in for a web­page and FastC­GI Cache is active:

As you can see, if there’s a cache hit (mean­ing the request­ed page is in the FastC­GI Cache), the whole process gets short-cir­cuit­ed, and the page is imme­di­ate­ly returned.

The whole cas­cade of PHP exe­cut­ing the com­piled Twig tem­plate, issu­ing a num­ber of MySQL queries, and so on just nev­er hap­pens. And that’s the lengthy and CPU/​disk inten­sive part of the process.

So this is great, but there are some caveats. Here’s what you can’t have if you want a full page sta­t­ic cached site:

• Serv­er-ren­dered dynam­ic con­tent that’s dif­fer­ent on a per-user basis
• Serv­er-side cook­ies, such as logged in ses­sions, Craft Com­merce carts, etc.

Remem­ber, all that is deliv­ered to every per­son who vis­its your site is the same exact raw HTML page. Your Twig tem­plates (real­ly, the com­piled PHP ver­sion of them) are ren­dered out once and cached.

So if you want dynam­ic con­tent that’s dif­fer­ent on a per-user basis, you’ll need to use a fron­tend JavaScript frame­work to pro­vide it, as per the Lazy Load­ing with the Ele­ment API & Vue­JS article.

Also, because serv­er-side cook­ies are includ­ed in the actu­al HTTP Response as a head­er like:

It’d obvi­ous­ly be a dis­as­ter if some­thing like the CraftSessionId or Craft Com­merce Cart cook­ies were part of the cached HTML, because every­one vis­it­ing the site would get the same cook­ie. Check out the The Case of the Miss­ing PHP Ses­sion for more than you ever want­ed to know about ses­sion cookies.

You can, how­ev­er, still user client-side cook­ies set via JavaScript when using full page sta­t­ic caching.

If you can live with these restric­tions, you can get some absolute­ly fan­tas­tic per­for­mance out of full page sta­t­ic caching with FastC­GI Cache. The sta­t­ic pages are cached on disk in a direc­to­ry with a hashed name like this:

The top lev­el direc­to­ry is the first let­ter of the hash, and the sec­ond-lev­el direc­to­ry is 2^nd two let­ters of the hash, and the file name is the remain­der of the hash. So if the full hash is e0beeedb43e78c8042f38b33f66fdc060be then it will be stored as e/0b/eeedb43e78c8042f38b33f66fdc060be on disk.

Let’s peek at the file contents:

The gib­ber­ish char­ac­ters are just some bina­ry data, then we have the KEY for the cached file, and then the HTTP Response head­ers, and then the raw HTML of the web­page itself. When a request comes in, FastC­GI Cache sees if it has the file in the cache, and if so, it just returns it. You can also set how long pages should be cached, and how long they should be kept around if they are inac­tive (not accessed).

That’s it, that’s all there is to it. It’s a very sim­ple but effec­tive page caching mechanism.

Link Some Timing Stats

This web­site was already quite per­for­mant before I imple­ment­ed FastC­GI Caching, but let’s have a look at the per­for­mance gains real­ized from mov­ing it to using sta­t­ic page caching.

The first thing I did was I removed all of the {% cache %} tags from my tem­plates. If we’re doing full page caching, there’s real­ly no rea­son for us to both­er with using Craft’s tem­plate caching as well.

Then I did some tim­ings using curl, first with caching disabled:

Next up, I turned the FastC­GI Cache on, and repeat­ed the same tim­ing test:

So that’s pret­ty good, we went from 1.087s to 0.354s to deliv­er the blog index page. That’s a bit over 3x faster, quite a gain!

The impor­tant stat to look at here is time_starttransfer. There’s a cer­tain amount of over­head involved with any HTTPS con­nec­tion, and as you can see the stats for every­thing else is about the same between the two sam­ples. This amounts to the (most­ly) fixed over­head; but time_starttransfer—which rep­re­sents the time it takes for the web serv­er to deliv­er the page — went from 0.872 uncached to 0.140.

Wow. That’s about 6x faster using a static page cache.

The fun thing to remem­ber is that even if the page orig­i­nal­ly took about 4s to load, it’d still only take about 0.35s to load if it is sta­t­i­cal­ly cached. This is because no mat­ter how long it takes to ren­der the page with­out caching, once it’s cached, it’s just text that’s being sent back.

The impor­tant thing to keep in mind is that your web­pages won’t just load faster — which is worth it alone — but also there is sig­nif­i­cant­ly less load on your serv­er. This helps with con­cur­ren­cy, for when your web­site receives mas­sive traffic.

And this is on a $40/m VPS. This can easily get down to .08s or lower on a beefier setup/connection

Here’s a tim­ing for the home­page from Web​PageTest​.org before, when we used Craft’s tem­plate caching:

And here’s the same page now that it’s a full page sta­t­i­cal­ly cached via FastC­GI Cache:

So our Time To First Byte (TTFB) went down 0.209s (almost 50% low­er), and the load time went down 0.344s. Not too shab­by, espe­cial­ly con­sid­er­ing the site was already quite fast.

Again, if the site wasn’t performant to begin with, the gains would be even larger.

There’s a temp­ta­tion to sim­ply use a full page sta­t­ic cache as a way to, well, var­nish over a poor­ly per­form­ing web­site. Don’t do this, folks.

Do your best to make the web­site rea­son­ably per­for­mant with­out caching, and then imple­ment caching to help with scal­a­bil­i­ty & concurrency.

If you don’t, then non-cache hits will be unnec­es­sar­i­ly slow, caus­ing more strain on your serv­er. It also is just slop­py to make site that per­forms like a dog, and mask it with a caching lay­er. It’s anal­o­gous to being too lazy to remove your wall­pa­per, and instead just paint­ing over it with many coats of paint.

Yeah, it’ll work, but you’re set­ting your­self up for prob­lems down the road.

Link Implementing FastCGI Cache

Alrighty, if you made it this far, I’ll assume you’re pumped to get FastC­GI Cache imple­ment­ed. Here’s how we do it.

You’re going to need to be run­ning Nginx; my serv­er is pro­vi­sioned by Lar­avel Forge, so I’m run­ning PHP 7.1 as well.

First in your virtualhost.conf file, add the fol­low­ing line at the top of the file, out­side the server {} block:

fastcgi_​cache_​path spec­i­fies that the sta­t­ic page cache files should be stored in /var/run/nginx-cache/nystudio, how many levels deep the direc­to­ry names should be, the keys_zone name & size, and inactive spec­i­fies how long since a cached item key has been accessed before it should be removed from the cache. You can name the zone what­ev­er you want, it’ll be used again in the .conf file.

The rea­son we’re stor­ing the FastC­GI Cache files under /var/run/ is that on Ubun­tu, this is loaded as a RAM disk via tmpfs, so the file sys­tem actu­al­ly exists in mem­o­ry. This will make it extra fast when access­ing the cache files. When you cre­ate the nginx-cache direc­to­ry in /var/run/nginx-cache, make sure that you chown and chgrp it to the user your web serv­er runs as (forge in my case).

Note that when the serv­er is restart­ed, every­thing in /var/run/ will be delet­ed, so the direc­to­ry will need to be recre­at­ed, and the sta­t­ic cache will be cleared. You can have it cre­ate this direc­to­ry for you auto­mat­i­cal­ly on reboot by cre­at­ing a file like this in /etc/cron.d/:

You can have more than one fastcgi_cache_path on your serv­er. For exam­ple, if you’re host­ing mul­ti­ple web­sites on the same VPS, can you cre­ate a sep­a­rate FastC­GI Cache for each, by giv­ing them their own fastcgi_cache_path. You may have to ini­tial­ly cre­ate the cache direc­to­ry your­self, depend­ing on the file per­mis­sions (it needs to be write­able by the web server).

Next, inside of the server {} block, add:

fastcgi_​cache_​key spec­i­fies the com­bi­na­tion of Nginx vari­ables that should be used to cre­ate a unique hash for each page in the cache. Most con­fig­u­ra­tions will use just $scheme$request_method$host$request_uri that will look some­thing like httpsGETnystudio107.com/. How­ev­er, you can use any­thing you want; we added the vari­able $abtest to the mix as per the A/B Split Test­ing with Nginx & Craft CMS article.

Note that the $request_​uri vari­able is the URI pre-nor­mal­iza­tion, and will include the query string (includ­ing any Google Ana­lyt­ics refer­rer params, etc.). If you want just the nor­mal­ized path with­out the query string, you can use the $uri variable.

add_header just sets a head­er that indi­cates the sta­tus of the X-Cache head­er in the HTTP Response (either HIT if it’s in the cache or MISS if it’s not), and then sets a $no_cache vari­able to 0 (false) by default, but sets it to 1 (true) if we don’t want to cache this response.

I have it set to not cache the response if the request is POST or if the URI match­es any of the spe­cial Craft seg­ments that are used for the AdminCP. You can add in oth­er con­di­tions, too, like if a par­tic­u­lar cook­ie is set, or what have you.

Spe­cif­ic to Craft CMS, we also have it not cache if the request has the X-Debug-Tag head­er set, which is what the Yii2 Debug Tool­bar sets if it is visible.

Next in the server {} block, add:

What we’re doing here is ensur­ing that any access­es to the AdminCP goes through our spe­cial @phpfpm_nocache loca­tion. This is impor­tant, because for all sta­t­i­cal­ly cached pages, we’re going to strip all cook­ies from the HTTP Response (more on this lat­er), but we don’t want to do this for AdminCP requests, or we can’t login in. Which is bad.

Note these these loca­tion blocks should be the first loca­tion blocks in your Nginx con­fig, to ensure that they match first.

Note that the fastcgi_param SCRIPT_FILENAME and fastcgi_param PATH_INFO are dif­fer­ent here than for our nor­mal PHP han­dler, and we’re spec­i­fy­ing that requests han­dled at this loca­tion are nev­er cached.

We also aren’t strip­ping cook­ies off of any requests at this loca­tion either, so if you have cer­tain pages that need serv­er-side cook­ies (such as, say, a Craft Com­merce store), you can send them to the @phpfpm_nocache loca­tion as well.

If you’re won­der­ing what all of the CRAFTENV_ set­tings are, check out the Mul­ti-Envi­ron­ment Con­fig for Craft CMS arti­cle for details.

Final­ly, here is our default loca­tion han­dler for .php files:

If you’re using a PHP ver­sion oth­er than 7.1, change the fastcgi_pass set­ting as appropriate.

The fastcgi_ignore_headers set­ting tells FastC­GI Cache to ignore the Cache-Control, Expires, and Set-Cookie head­ers that it would nor­mal­ly look at to decide whether or not it should cache the request.

The fastcgi_hide_header set­ting tells Nginx to strip out all cook­ies from the HTTP Response. We do this so that no serv­er-side cook­ies are stored in the page cache, because we don’t want things like the CraftSessionId to be cached and thus deliv­ered to and shared by every vis­i­tor to the site!

fastcgi_cache tells it which key_zone to use, as spec­i­fied above.

The fastcgi_cache_valid set­ting tells FastC­GI Cache what HTTP Response codes should be cached (200 = ​“OK”) and for how long the pages should be cached, in our case, 1 day.

And that’s it, save the set­tings, restart Nginx via sudo nginx -s reload and away you go! You can see all of the set­tings avail­able to you on the Mod­ule ngx_​http_​fastcgi_​module website.

Link Ch-ch-ch-changes

We’re very used to hav­ing a dynam­ic site gen­er­at­ed by Craft, so we have to make some changes to our mind­set as well as our code when work­ing on a full page sta­t­i­cal­ly cached site.

As men­tioned above, I stripped out all of the {% cache %} tags from my tem­plates. There’s just no point in hav­ing two lay­ers of caching if we’re plan­ning to have the FastC­GI Cache last for a rea­son­ably long time.

Tan­gent: An argu­ment could be made for using the Craft tem­plate caching on top of full page FastC­GI caching if you want­ed to lever­age Craft’s more intel­li­gent cache-bust­ing tech­niques to speed up cache miss­es. For exam­ple, we could glob­al­ly {% cache %} the blog archives sec­tion on this site, and depend on Craft’s intel­li­gent cache bust­ing to han­dle when it needs to be regen­er­at­ed. But in gen­er­al, you can like­ly eschew mul­ti-lay­er caching, espe­cial­ly if both caches are like­ly to be bro­ken at the same time. Because then we end up doing more work than if we had a sin­gle lay­er of caching, as both caches need to be rebuilt and saved.

Next, any­where I used con­di­tion­als based on cook­ies cour­tesy of my Cook­ies plu­g­in, such as the Crit­i­cal CSS described in the Imple­ment­ing Crit­i­cal CSS on your web­site arti­cle, I changed to use Nginx Serv­er Side Includes (SSI):

You have to make sure you set ssi on; inside of your server {} block, but once you have, you can lever­age SSI to have con­di­tion­als in your sta­t­i­cal­ly cached pages.

I also wrapped my entire base _layout.twig tem­plate in {% minify %} tags cour­tesy of my Mini­fy plu­g­in, so that the cached pages are mini­fied by strip­ping white­space. Might as well, since they are going to be cached!

Just take care not to wrap our SSI direc­tives in the {% minify %} tags, or they will be stripped out of the result­ing HTML (as all com­ments are, by default). I just use mul­ti­ple {% minify %} / {% endminify %} tags to exclude the SSI direc­tives, while still mini­fy­ing the entire page.

The big change, though, is men­tal­ly adjust­ing to the idea that the page peo­ple are receiv­ing is sta­t­ic, and han­dling dynam­ic con­tent via a fron­tend JavaScript frame­work or Nginx SSI conditionals.

Link Cache Busting

Final­ly, what do we do to bust the cache if we make changes to our web­site? Well, we could just wait until the page caches expire (after 1 day in our set­up), but patience has nev­er been my thing.

First, if you use Craft-Scripts as described in the Data­base & Asset Sync­ing Between Envi­ron­ments in Craft CMS arti­cle, you can have the clear_caches.sh script auto­mat­i­cal­ly delete the FastC­GI Cache on deploy by set­ting LOCAL_FASTCGI_CACHE_DIR in your .env.sh

This way, when­ev­er you deploy code changes, you can ensure cache coheren­cy by just clean­ing the entire FastC­GI Cache. This is real­ly the only safe way to do it when we’re talk­ing about a code change deploy (whether to Craft, or your tem­plates, or your plugins).

For bust­ing the cache due to chang­ing things in the Craft CMS AdminCP (such as edit­ing or adding an Entry), I cre­at­ed a sim­ple FastC­GI Cache Bust plu­g­in. Install it, put your FastC­GI Cache path (e.g.: /var/run/nginx-cache/nystudio ) into the plu­g­in Set­tings page, and any time you save any­thing in the AdminCP, it’ll bust the entire FastC­GI Cache.

If you’re using Craft 3.x, there’s a ver­sion of the FastC­GI Cache Bust plu­g­in for Craft 3.x as well.

This is a bit of a brute force approach, and there are ways that it could cache bust only spe­cif­ic URLs, but it suits my needs for now. In the future, I’ll like­ly make it a lit­tle less ham-handed.

So that’s it, folks. Hap­py caching!

Link The Full Monty

To aid you if you’re try­ing to set this up your­self, here’s the full Nginx virtualhost.conf file used on this web­site (which is based on the Nginx-Craft set­up, and also uses the Craft-Mul­ti-Envi­ron­ment set­up). This is the con­fig from Lar­avel Forge.

Remem­ber that in Nginx, some­times the order in which direc­tives appear can affect how things work.

Also, in your general.php you’ll want to have:

Here’s the detailed expla­na­tion from Craft docs: Enabling PATH_INFO

And now, onto the config: