A question we sometimes field from Conductor users is “Why won’t my website update?” You go into your Conductor admin, update a page, and then view it on the live site. The old content is still showing.
This concern (and the underlying issue of performance) is common to every content management system, be it Conductor, Wordpress, Drupal, or any other. Search “performance” and the name of any CMS, and you will find article after article about how to improve the performance of your pages. Within those articles, page caching will be among the first recommendations.
We, the Web Team, measure page load times in milliseconds. If a page in Conductor takes one full second to be generated, that is 5 times too long for our taste. We have have very high performance standards and our sites, built on Conductor, consistently deliver at a higher level than similar systems.
In this blog post, I will be provide some insight into what caching is, why we chose to do things the way we are doing them, what can go wrong, and what you should do if things do go wrong.
From the Database to the Page
Like most content management systems, Conductor stores all your pages in a database. When a page is requested, Conductor looks up the page (or news item, calendar event, etc.), based on the site’s domain and URL. Once Conductor has identified the page, it has to collect all the page data (content, custom fields, images, etc.) from the database and piece it all together into a web page in the user’s browser.
If that sounds like a lot… it is. Depending on how complicated a web page is, we are talking about as many as a thousand database queries to collect all the data needed to create one page.
Enter the Caching Proxy Server
A screen capture of a performance monitoring system showing the database queries needed to create a single page on Conductor.
When we moved Conductor to Amazon Web Services (AWS), we had a new set of tools available to address performance.
Our goal is to have the page delivered to the browser in less than 200 milliseconds. If you consider the process described previously for receiving a request and delivering a complete web page to the browser, 200 milliseconds probably sounds unrealistic. In fact, it is often unrealistic. But, what if we skip all that backend magic, all those queries?
That is what Varnish allows us to do. Varnish is a server that sits in front of Conductor, receiving requests, and passing them on to Conductor. It also saves that fully formed web page that is created, so the next time that page is requested, Varnish already has a copy of it. It can serve the page to the next user without executing all those database queries.
This means the first time a page is requested, it can be a little slow. In most cases, you won’t notice a significant lag but there are certain pages across Conductor that are slow to load. But, in any case, the second time, the third time, or the hundredth time a page is requested, it will load quickly.
Cache Busting
Varnish Logo
But what if the page is updated? How does Varnish, this other server, know what to do? How can we be sure the cached page is the latest version of the page?
You may already see a principle conflict between, on one hand, being fast and on the other hand, being accurate.
One way to update cached pages is to put an expiration date on the cache. Using this method, Varnish could keep a copy of a page for 7 days (or even a couple of months) before saving a new copy. Obviously that would be a problem for sites that are updated regularly. Conductor applies this technique, but always in conjunction with another technique: Telling Varnish directly to delete a page cache from its collection of cached pages.
When any page, news item, or event is created or updated a request is sent to Varnish to delete all the cached pages for the site from its cache. Almost every change you make in the Conductor admin, will trigger this action. So, every time you save a change to your site, the entire page cache for your site is cleared out and the process of creating a all new cache starts.
The Queue
The last bit of technology you need to know about before this all comes together is the queue.
There are a bunch of things Conductor does that don’t have to be done immediately. “Immediately” meaning something that has to be completed before a user can continue.
For instance, when you upload an image to Conductor, it uploads and you immediately see the file in the admin.
There are other things, however, going on with that image and those requests are added to the queue. For example, uploaded images are sent to a third party service to be optimized (to minimize the file size). The image is saved at a variety of file sizes and dimensions. Recently, we added a process to automatically resize images that are larger than 2400px. The queue is the place we send all these requests while the user moves on with their business.
In the background, Conductor is constantly looking for new requests in the queue. When it finds one, it completes it in the background.
A request for Varnish to delete the cached pages for a site is another example of something that is sent to the queue.
What Can Go Wrong?
Generally, it’s one of two things that can cause a delay in a site not being updated as expected.
The Queue Is Backed Up
For the most part, the queue chugs away and Conductor hums along, processing everything that comes in within a few seconds. But, on occasion there can be so many requests hitting the queue that a backlog develops. Sometimes one of the requests takes a very long time and the requests in line must wait for it to finish. This backup will fix itself once the long running request is done or the amount of requests getting sent to the queue slows down. Generally, such a backup last no more than a minute or two.
The Queue Is Down
On rare occasion, the queue quits working. Requests can be added but the queue has stopped processing them completely.
It only takes a minute, but we have to manually restart the queue in these cases. Once the queue is restarted, all pending requests will get processed within a few minutes.
What Does This Mean for Users?
The reality of maintaining hundreds of sites, tens of thousands of pages, and hundreds of thousands of assets is that it takes time to create, load, save, or serve that content millions of times everyday. So, sometimes Conductor (and every other CMS) is going to take a bit longer than 200 milliseconds to load the latest version of a page. It's rare that it takes more than 1 or 2 seconds, but, if it does, here are the things you can do:
- Refresh the page in your browser. Updates to pages should be reflected on the live site by the time you can load or preview the page.
- Wait a minute or so and refresh the page. A page should reflect changes within seconds, but, as described, sometimes there is a lag of a minute or two if there is a backlog of requests in the queue.
- Click “Refresh site content” button in the Conductor admin. When you click this button, Conductor sends a request to the Varnish server to delete ALL the page caches for your site. (Note that repeatedly clicking the “Refresh Site Content” will not speed up the site refresh as it is adding more requests to the end of the queue.)
- If those things fail, the queue maybe down. Contact webhelp@nd.edu.
If you have other concerns about performance or caching on your site, or a specific page of your site, we are always working on new ways to make Conductor run smoother, so feel free to contact us.