Robust queue job handling in Craft CMS

N.B.: This arti­cle was updat­ed to no longer rec­om­mend using the Async Queue plugin.

Queu­ing in line is some­thing few peo­ple are fond of, but it allows each per­son to be ser­viced in order, in an effi­cient and fair manner.

Craft CMS also uses a queue, too. Any lengthy task is pack­aged off into a job, and jobs are put into a queue to be exe­cut­ed in order.

In com­put­er sci­ence terms, this is a First In, First Out (FIFO) data struc­ture that ser­vices things in what we’d con­sid­er to be a nat­ur­al order.

Here are some lengthy tasks that end up as jobs in the Craft CMS queue:

• Updat­ing entry drafts and revisions
• Delet­ing stale tem­plate caches
• Gen­er­at­ing pend­ing image transforms
• Re-sav­ing Elements
• Updat­ing search indexes
• Plu­g­in cre­at­ed queue jobs

The queue exists to allow these often lengthy jobs to run in the back­ground with­out block­ing access to the Craft CMS Con­trol Pan­el (CP) while they execute.

Link See How They Run

So now that we know what type of jobs will end up in the queue, let’s have a look at how these jobs are run. This ends up being a cru­cial­ly impor­tant bit of infor­ma­tion to under­stand about queue jobs.

Giv­en that Craft CMS is built on top of the Yii2 frame­work, it makes sense that it lever­ages the exist­ing Yii2-queue by lay­er­ing a QueueIn­ter­face on top of it.

It does this so that it can pro­vide a nice friend­ly sta­tus UI in the CP. The default Craft set­up uses a data­base (Db) dri­ver for queue jobs, stor­ing them in the queue table.

Although there are oth­er queue dri­vers avail­able, you’re unlike­ly to see any gains from using any­thing oth­er than the Craft CMS Db dri­ver. And you’d lose the afore­men­tioned sta­tus UI in the CP.

Where queue jobs are stored and how they are run are completely separate things.

But how do the queue jobs run? Here’s how:

• When some­one access the CP
• Via web request

Here­in lies the rub with the default queue imple­men­ta­tion in Craft CMS. No queue jobs will run at all unless some­one logs into the CP, and even then, they are run via web request.

Here’s an excerpt from the Craft CMS CP.js JavaScript that kicks off the the web-based queue running:

And here’s the queue/​run con­troller method that ends up get­ting exe­cut­ed via AJAX:

They are doing what they can to make sure the web-based queue jobs will run smooth­ly, but there’s only so much that can be done. Bran­don Kel­ly from Pix­el & Ton­ic even not­ed ​“Craft 2, and Craft3 by default, basi­cal­ly have a poor-man’s queue”.

Running lengthy operations via web request rarely ends well

Web requests are real­ly meant for rel­a­tive­ly short, atom­ic oper­a­tions like load­ing a web page. For this rea­son, both web servers and PHP process­es have time­outs to pre­vent them from tak­ing too long.

This can result in bad things happening:

• Queue jobs stack up until some­one logs into the CP
• Queue jobs can fail or get stuck if they take too long and time out
• Queue jobs can fail or get stuck if they run out of memory
• Lengthy queue jobs can impact CP or fron­tend per­for­mance as they run

For­tu­nate­ly, we can alle­vi­ate all of these prob­lems, and it’s not hard to do!

If you do have tasks that have failed due to errors, check out the Zen and the Art of Craft CMS Log File Read­ing arti­cle to learn how to debug it.

Link Queue Running Solutions

For­tu­nate­ly Yii2, and by exten­sion Craft CMS, comes with a robust set of com­mand line tools that I detail in the Explor­ing the Craft CMS 3 Con­sole Com­mand Line Inter­face (CLI) article.

The queue com­mand gives us the abil­i­ty to shift our queue jobs into high gear by run­ning them direct­ly. PHP code that is run via the con­sole com­mand line inter­face (CLI) uses a sep­a­rate php.ini file than the web-ori­ent­ed php-fpm. Here’s an exam­ple from my server:

The rea­son this is impor­tant is that while the web-based php-fpm might have rea­son­able set­tings for memory_​limit and max_​execution_​time, the CLI-based PHP will typ­i­cal­ly have no memory_limit and no max_execution_time.

This is because web process­es are exter­nal­ly trig­gered and there­fore untrust­ed, where­as things run­ning via the CLI is inter­nal and trusted.

So run­ning our queue jobs via PHP CLI gives us the fol­low­ing benefits:

• Can be run any time, not just when some­one access­es the CP
• Won’t be ter­mi­nat­ed due to lengthy exe­cu­tion time
• Are unlike­ly to run out of memory

The queue con­sole com­mands we use are:

• queue/run — Runs all jobs in the queue, then exits
• queue/listen — Per­pet­u­al­ly runs a process that lis­tens for new jobs added to the queue and runs them

There are sev­er­al ways we can run these com­mands, dis­cussed in the Yii2 Work­er Start­ing Con­trol arti­cle, and we’ll go into detail on them here.

Link Solution #1: Async Queue Plugin

I no longer rec­om­mend using the Async Queue plugin.

While it is tempt­ing to use a plu­g­in as a sim­ple solu­tion, the Async Queue plu­g­in has had too many unre­solved issues.

Instead, I rec­om­mend using one of the options below.

Link Solution #2: Forge Daemons

If you use the won­der­ful Lar­avel Forge to pro­vi­sion your servers (dis­cussed in detail in the How Agen­cies & Free­lancers Should Do Web Host­ing arti­cle), Forge pro­vides a nice GUI inter­face to what it calls Dae­mons.

Go to your Serv­er, then click on Dae­mons, and you’ll see a GUI that looks like this:

Here’s the com­mand we’ve added for the dev​Mode​.fm website:

Let’s break this down:

• /usr/bin/nice -n 10 — this uses the Unix nice com­mand to run the process using a low­er pri­or­i­ty, so it won’t inter­fere with the CP or frontend
• /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose — this starts a PHP CLI process to lis­ten for any queue jobs, and run them

You should swap in your own path to the craft CLI exe­cutable; ours is /home/forge/devmode.fm/craft

We tell it that we want to run this as the user forge and we want 2 of these process­es run­ning. Click Start Dae­mon and you’ll be greet­ed with:

Then ensure that you’ve set the run­QueueAu­to­mat­i­cal­ly Craft CMS general.php con­fig set­ting to false for your live pro­duc­tion environment:

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

The fol­low­ing is an entire­ly option­al peek under the hood at how this works with Lar­avel Forge. Feel free to skip it if you’re not inter­est­ed in the gory details.

What Forge calls Dae­mons are actu­al­ly com­mands run via the Super­vi­sor com­mand. When we cre­at­ed the above dae­mon in the Forge GUI, it cre­at­ed a file in /etc/supervisor/conf.d:

The con­tents of the file look like this:

And indeed, this is exact­ly one of the meth­ods rec­om­mend­ed in the in the Yii2 Work­er Start­ing Con­trol arti­cle. Huzzah!

You might be won­der­ing what to do when you deploy changes to your web­site; we obvi­ous­ly don’t want the queue run­ner sit­ting there run­ning old code! So sim­ply do this as part of your CI:

This will cause any run­ning queue jobs to grace­ful­ly ter­mi­nate, and then supervisord will restart your queue run­ner. To ensure that you can sudo to run this com­mand, edit the /etc/sudoers.d/php-fpm file to add a line like this:

This ensures that the sudo -S supervisorctl restart all can exe­cute with­out need­ing to enter a sudo password. 

Link Solution #3 Using Heroku

If you’re using Heroku for your host­ing needs, you can use Work­er Dynos to run your queue/listen command:

Then ensure that you’ve set the run­QueueAu­to­mat­i­cal­ly Craft CMS general.php con­fig set­ting to false for your live pro­duc­tion environment:

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

Link Solution #4: Utilizing systemd

If you’re using a Lin­ux serv­er, but you’re not using Forge or Heroku, or per­haps you just want to con­fig­ure things your­self, we can accom­plish the same thing using sys­temd.

systemd is a way to start, stop, and oth­er­wise man­age dae­mon process­es that run on your serv­er. You’re already using it whether you know it or not, for things like your web serv­er, MySQL, and so on.

Here’s what I did for dev​Mode​.fm to use systemd to run our queue jobs. You’ll need sudo access to be able to do this, but for­tu­nate­ly most mod­ern VPS host­ing ser­vices give you this ability.

First I cre­at­ed a .service file in /etc/systemd/system/ named devmode-queue@.service via:

Here’s what the con­tents of the file looks like: 

Let’s break this down:

• /usr/bin/nice -n 10 — this uses the Unix nice com­mand to run the process using a low­er pri­or­i­ty, so it won’t inter­fere with the CP or frontend
• /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose — this starts a PHP CLI process to lis­ten for any queue jobs, and run them

You should swap in your own path to the craft CLI exe­cutable; ours is /home/forge/devmode.fm/craft

We tell it that we want to run this as the user forge , we want it to restart the process on fail­ure and that it depends on both the network and mysql ser­vices being available.

The strange @ in the .service file name allows us to have mul­ti­ple instances of this ser­vice run­ning. We can then use the range syn­tax {1..2} to spec­i­fy how many of these process­es we want.

We’ve cre­at­ed the file, but we need to start the ser­vice via:

And then just like our oth­er ser­vices such as nginx and mysqld we want it to start up auto­mat­i­cal­ly when we restart our server:

Then ensure that you’ve set the run­QueueAu­to­mat­i­cal­ly Craft CMS general.php con­fig set­ting to false for your live pro­duc­tion environment:

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

Now you can start, stop, enable, dis­able, etc. your Craft CMS queue run­ner just like you would any oth­er sys­tem ser­vice. You can also mon­i­tor it via the jour­nalctl command:

The above com­mand will tail the systemd logs, only show­ing mes­sages from our devmode-queue service.

You can learn more about using journalctl in the How To Use Jour­nalctl to View and Manip­u­late Sys­temd Logs article.

Link Understanding TTR

If you’ve land­ed on this page as the result of a des­per­ate Google search, because despite doing all of the won­der­ful things, your queue jobs some­times still time­out and fail, you’re in luck!

I have an answer for you, and it involves a lit­tle some­thing called TTR aka, Time To Reserve.

This is the amount of time that is reserved for a Yii2 queue job to run in the queue job. It does­n’t mat­ter what your PHP max_execution_time is set to, it does­n’t mat­ter that you are run­ning your queue jobs via a queue run­ner via the CLI.

Your queue jobs will always be terminated if the queue job exceeds the TTR

Here’s what the Yii2 doc­u­men­ta­tion has to say about TTR:

“The ttr (Time to reserve, TTR) option defines the num­ber of sec­onds dur­ing which a job must be suc­cess­ful­ly com­plet­ed. So two things can hap­pen to make a job fail:

1. The job throws an excep­tion before ttr is over
2. It would take longer than ttr to com­plete the job (time­out) and thus the job exe­cu­tion is stopped by the worker.

In both cas­es, the job will be sent back to the queue for a retry. Note though, that in the first case the ttr is still ​“used up” even if the job stops right after it has stared. I.e. the remain­ing sec­onds of ttr have to pass before the job is sent back to the queue.

The attempts option sets the max. num­ber of attempts. If this num­ber is reached, and the job still isn’t done, it will be removed from the queue as com­plet­ed.

These options apply to all jobs in the queue. If you need to change this behav­ior for spe­cif­ic jobs, see the fol­low­ing method.”

Thank­ful­ly, the TTR is set to some­thing fair­ly rea­son­able, at 300 sec­onds (5 min­utes). We can see that here:

This val­ue is then used to cre­ate a Sym­fony Process object:

Which is passed into the Process object con­struc­tor as the $timeout parameter:

And as the Sym­fony Process doc­u­men­ta­tion states:

If the timeout is reached, a ProcessTimedOutException is thrown.

You’ll see an error like this in your storage/logs/queue.log as well as in the Util­i­ties → Queue Manager:

So if you some­how do real­ly have queue jobs that need to run for more than the default 300 sec­onds (5 min­utes), you can con­fig­ure the queue com­po­nent in your config/app.php to have a longer ttr:

The above will allow queue jobs to run for 10 min­utes; you can make this as long as you need to.

If you want there to be no time­out at all, you can pass in null, and the process will nev­er time­out.

To make things more fun, there’s an open issue stat­ing that TTR does­n’t work as adver­tised, which is true if the queue job exe­cu­tion isn’t done via a work­er process.

Link Getting Dotenvious

This is an entire­ly option­al sec­tion you can skip if you’re not inter­est­ed in learn­ing how to not use .env files in pro­duc­tion, as rec­om­mend­ed by the php­dotenv pack­age authors:

php­dotenv is made for devel­op­ment envi­ron­ments, and gen­er­al­ly should not be used in pro­duc­tion. In pro­duc­tion, the actu­al envi­ron­ment vari­ables should be set so that there is no over­head of load­ing the .env file on each request. This can be achieved via an auto­mat­ed deploy­ment process with tools like Vagrant, chef, or Pup­pet, or can be set man­u­al­ly with cloud hosts like Pagod­abox and Heroku.

You do not need to do any of this unless you do not use .env in production

Addi­tion­al­ly some setups have envi­ron­men­tal vari­ables intro­duced by alter­nate means. I made a Doten­vy pack­age that makes it easy to gen­er­ate .env key/​value vari­able pairs as Apache, Nginx, and shell equivalents.

In this case, we’ll need to mod­i­fied the com­mand we run slight­ly, so that we’re defin­ing our envi­ron­men­tal vari­ables in the shell before run­ning Craft via the CLI.

All we need to do is change our com­mand to this:

All we’re doing is exe­cut­ing the .env_cli.txt file (cre­at­ed by Doten­vy) that con­tains our envi­ron­men­tal vari­ables we export to the shell environment:

The rest is the same as the com­mands we’ve dis­cussed above for run­ning queue/listen, and then we wrap it all in /bin/sh -c so it appears as a sin­gle com­mand that can be executed.

Link Deeply Queued

Here’s some queue job triv­ia that you may find interesting.

We men­tioned ear­li­er that the only way queue jobs are run nor­mal­ly is if you vis­it the CP. This is most­ly true.

There’s actu­al­ly a case where queue jobs can be run via fron­tend page load. If you have gen­er­ate­Trans­forms­Be­forePage­Load set to false (which is the default), and there are pend­ing trans­forms that need to be generated.

What hap­pens is the Assets::getAssetUrl() method push­es the image trans­form queue job, which calls Queue::push():

So if this is a site (fron­tend) request, and it’s not an AJAX request, Craft will add a call to Queue::handleResponse() that gets fired when the Response is about to be sent back to the client.

In this case, Craft will inject some JavaScript into the fron­tend request that pings the queue/run con­troller via AJAX:

While tech­ni­cal­ly this could hap­pen for any queue job that is Queue::pushed()​‘d in the queue dur­ing a fron­tend request, asset trans­forms are the only case I’m aware of that this hap­pens in practice.

For this rea­son, if you’re post­ing any XHR requests to Craft con­trollers or API end­points, make sure you add the header:

Craft looks for this head­er to deter­mine whether the request is AJAX or not, and will not attempt to attach its queue run­ner JavaScript to the request if the request is AJAX.

This head­er is set auto­mat­i­cal­ly by jQuery for $.ajax() requests, but it is not set auto­mat­i­cal­ly by pop­u­lar libraries such as Axios.

Link Wrapping Up

So which method do I use? If I’m spin­ning up a VPS with Forge (which I nor­mal­ly am), then I’ll use Solu­tion #2: Forge Dae­mons. If it’s not a Forge box, but I do have sudo access, then I’ll use Solu­tion #3: Uti­liz­ing sys­temd.

For fur­ther read­ing, check out Ben Cro­ker’s arti­cle Queue Run­ners and Cus­tom Queues in Craft CMS.

Replac­ing the default web-based queue job sys­tem in Craft CMS with a more robust CLI-based sys­tem will pay dividends.

Your queue jobs will run more smooth­ly, fail less often, and have less of an impact on the CP/​frontend.

It’s worth it. Do it.

Hap­py queuing!