Multi-stage Templating as Progressive Denormalization

One of the perennial rallying cries of content management has always been that “content is separate from presentation.” We store content in a raw, pure form then when it comes out, we muck it all up with templating. This is the standard paradigm.

But is all templating equal?

We usually think of templating as the final step to create HTML out of content. In our heads, content makes the jump from “raw” to “presentation” in one step – it’s raw content then…BOOM…it becomes presentation-ready content in the blink of an eye.

But perhaps we need to expand this definition to include a layer of “transformation” at the repository level? Furthermore, maybe we need to look at the journey from raw to presentation as a series of progressive denormalization.

I’m borrowing the word “transformation” from XSLT. That technology was intended to turn XML into other XML formats to use for different purposes (or, more commonly, as part of a pipeline). Since no one reads raw XML, the assumption was that the XML would eventually get “templated” into a consumable form. (Of course, then we all started “transforming” XML directly into XHTML, which wasn’t the original plan, but whatever…)

Transformation might start earlier in the process than you think, and it ends up looking like a pipeline – content is thrown down the pipe and refined until it finally comes out the other end. From another perspective, templating might resemble a tree where content starts at the top, gets subdivided into multiple other formats, which are then further templated into end-user consumable formats.

We tend to take this for granted, but the surge of headless CMS systems has suddenly thrown this back into play, especially – and sometimes only – when we start pushing to multiple channels. Without any ability to transform at the repository, developers are having to write their own templating layers, which often means duplicate templating code and logic in multiple consuming channels.

Frustratingly, most headless vendors refuse to concede to transformation closer to the repository. Multi-stage refinement – early-stage transformation, then later-stage templating – is not well-supported in the current crop of headless CMS.

Herein, I’ll make the case of why this needs to change.

Content Transformation as a Drivetrain

The drivetrain of a car is complicated as does a lot of stuff. Let’s consider the engine as the raw content stored in the repository, and the point where the tires touch the pavement is the final step in presentation to our audience.

The relationship of the power that comes off the engine and the rotational force applied to the ground through the tires is complicated and takes a lot of work to sort out.

• You can’t hook the wheels directly up to the crankshaft of the engine. The engine has to constantly turn, and if the wheels were connected directly, they’d have to turn around about 500 times per minute at idle. To fix this, we have clutches and torque converters to disconnect the engine so it can run while the car isn’t moving.

• In many cases, the crankshaft of the engine is a single rotational device turning on a north-south axis, while there are (at least) two drive wheels turning east-west. This is why we have differentials – they’re a series of gears with let the power of the engine turn 90-degrees and split into two outputs.

• Engines can rev from 500 rpm to sometimes 10,000 rpm, but they make peak power in a more narrow range called the “powerband.” Thus, we have transmissions which match the speed of the car to the powerband, so the engine stays as efficient as possible.

This is even a simplified analysis – we didn’t even talk about flywheels, transfer cases, driveshafts, and wheel bearings. A lot of stuff has to happen to get a single horsepower at the crankshaft to do anything productive at the wheels. Remember, in its pure form, an internal combustion engine makes pretty useless power. It has to keep running, it only turns one way, and it gets irritated when you try to put load on it outside a particular RPM range. So the drivetrain of an engine is a mechanical pipeline that “fixes” a series of problems along the way, and what comes out at the end – tire on pavement – is useful power.

Is your content the same way? Is the raw content stored in your CMS of the same value as the content that ends up in the hands of the consumer?

What the drivetrain of the car is designed to do – much like transformation and templating – is progressively refine (there’s that phrase again) the power of the engine until it’s an appropriate form to be applied to the pavement through the tires.

And here’s the point to remember: there’s one drivetrain that powers all the wheels. Each wheel doesn’t have its own transmission. We’ll come back to this in a second.

A CMS needs the ability to transform content prior to delivery to a consuming application where it will be further templated. This means transforming content as close to the “engine” (the repository) as possible so that all “tires” (consuming channels) benefit from it. This is in opposition to delivering nothing but raw content and expecting the consuming channels to bear all of the templating workload.

The problem with leaving all templating to the consuming channels is two-fold:

1. Latency: some transformation is complex and involves considerable computational power and perhaps external resources. Doing this on every read can be painful.

2. Multiple channels/platforms/languages: if we’re pushing this content into multiple channels, there’s some basic transformation that we simply don’t want to repeat in every channel.

Put another way, having content in its raw form might be less than helpful (remember our useless raw horsepower). Here are some real-life examples:

Example: Domain Specific Languages (DSL)

A health system had 900 clinics, each of which had to maintain their hours online. This meant storing open and close times for seven days of the week, and occasionally there was a clinic that was open for separate time periods in the same day – for example, from 8 a.m. to noon, then again from 2 p.m. to 5 p.m.

This was a painful interface to model. At a minimum, you’re looking at 14 different properties (open and close times for seven days of the week), or more when you consider the odd exceptions. Additionally, it was tedious. Time selectors generally suck, and editors were faced with 14 of them. It wasn’t pretty.

So what we did was allow (and train) editors on a simple domain-specific language, like this:

M-F: 8-5
Sa: 8-noon

Or:

M: 8-noon, 2-5
T-Th: 8-5

Or:

*: 8-3

We wrote a parser to turn this into a strongly typed object, which could then be turned into JSON or XML (and subsequently cached). The logic to do this parsing was complex and took considerable debugging to work out. Additionally, it wasn’t computationally cheap. But it worked beautifully, and with about five minutes of training, it made perfect sense to the editors.

(For a more common example, consider Markdown, That’s a perfect example of a DSL designed to simplify the generation of HTML.)

Example: Markup Macros

In many CMSs, editors can insert “macros” in rich text. These are text shortcuts which “expand” to larger text constructs – usually HTML, but they could theoretically be anything.

For example, the site for my book has a glossary with wiki-like links between pages. This is what they look like in the editor (from the entry for “version control”):

{%{
A version label is a designation applied to any particular
version. For example, a version might be labeled
"{{publication|Published}}" or "{{Draft}}". By versioning
content, the {{CMS}} retains a history of all the changes 
to a piece of content over time allowing {{editor|+s}} to 
rollback to prior versions if necessary or audit changes 
to content for security or regulatory purposes.
}%}

See those words in double brackets? Those are wiki-ish links that map to other terms based on ID slug and may display different text – different words entirely, or modified versions of the source word (i.e. – “editor|+s” links to “editor” but displays “editors”).I’ve written a parser to “expand” all these “macros” – to find the correct term, modify the source word, and add the hyperlink.

(And it’s not just me: Drupal has an entire architecture of “text filters,” and WordPress has a very popular system for handling “shortcodes” – I’ve used a couple in this very blog post, in fact. A more advanced version might be Episerver’s re-usable block elements that can be dragged into rich text fields.)

Example: Volume and Consistency

A law firm has 48 separate web properties. They all run on different platforms (some WordPress, some Sitecore, some hand-coded on non-executing environments like S3). They also have a series of blog posts, which they want to be able to insert into different sites (perhaps through a macro system, as described above).

However, they want the exact same format for the blog posts. They want a title, date, site name, preview, and link. They have a common HTML construct which can be styled for the individual property. What they don’t want is for each potential environment to “roll their own” templating code and form HTML from raw content. They want these all to appear the same way, in addition to the problem of some environments not being able to execute at all.

What they need, then, is transformation at the repository (the “drivetrain”). So each blog post has all its raw, editorial content, but it also contains an HTML representation of the last time it was saved which can be rendered to the page, unaltered, in any language, on any platform. This is a problem if the only thing the consuming platforms can get is the raw content to templating separately. They don’t need that – they need a bundle of content and rendering logic, cached to a string.

(Could these posts be pre-rendered and cached as HTML in some central location? Sure, but then you lose a couple things – searchability and any vendor-provided CDN or caching functionality – which we’ll discuss more below.)

The Need for Repository-Centric Transformation

These three examples both lead me to the same conclusion: I only want to do this early-stage transformation once. Here’s why:

• The end result of these two situations doesn’t change post-publish. (Markdown renders to HTML, full stop.)

• It’s relatively painful to do, computationally.

• It would be a lot of work to re-implement and maintain this logic for more than one language or platform. (What if it depends on a library that doesn’t exist for a particular platform?)

• Beyond the sheer workload, I really only want one canonical instance of this logic. (Markdown has many flavors. I don’t want developers applying their own preferred flavor to my source content.)

Think back to our car analogy – this is way up at the transmission level, right after power leaves in the engine. Late-stage templating is way back at the wheels. And once we start pushing content to multiple channels, this becomes a real problem. Remember: Each wheel on the car doesn’t have its own transmission. There’s one transmission, which all the wheels benefit from, just like there is content transformation we don’t want to repeat in each consuming channel.

(Consider the content from my glossary site. If I wanted to use that here, on Gadgetopia, I would have to rewrite all that wiki link parsing and expansion logic in PHP (it’s in .NET right now). Believe me when I tell you that I don’t want to do this.)

“Back End for Front End”

All the issues above aren’t problems if we own the entire content publishing apparatus, from start to finish. However, with the new crop of headless CMS, we often don’t. These are often SaaS systems, with which we interact through web APIs alone.

Additionally, the vendors are frustratingly dogmatic in their desire to stay away from any templating, including “drivetrain” transformation. They refuse to touch it on principle, priding themselves on only managing raw content and leaving it up to the consuming services to template.

I was bouncing these issues off of a handful of vendors, and a couple mentioned that I should use a “backend for frontend,” or “BFF” as some are calling it. This would be my own infrastructure basically acting as middleware – it stands between the headless CMS and my website (or whatever). My website communicates only with my BFF, and it’s the only thing that talks to the CMS.

In this way, my BFF becomes my drivetrain. Consuming channels pull from the BFF and complete their late-stage templating individually.

This solves some problems:

• We can implement our own endpoints in the BFF.

• We can manage the transformation pipeline, doing some transformation in the BFF. (In the glossary example above, I could render and cache the wiki link functionality at the BFF level. A request for that rendered content would never even need to go further “back” to the CMS.)

But there are disadvantages too.

• The vendor often has elaborate caching systems in place to support massive loads, and you’re paying for this – this is often a huge selling point in the value proposition. In these cases, you lose all of that, and now have to re-implement caching in your BFF.

• In addition to caching, you lose all the benefits of the vendor’s API, most painfully the query logic. You effectively now have to re-implement their API, including the ability to search everything, and avoiding this might have been the reason to decided to go headless in the first place.

• For the vendor, it whittles away their value proposition, and may even reduce their product to a glorified database UI. This is a perilous place to be. Data storage and related admin interfaces are not that hard to replicate, and if your value proposition dwindles, so might the reasons for keeping you around.

I’m not saying a BFF isn’t valuable and appropriate in some places, but when a vendor immediately punts to that “solution,” they’re basically saying: “You’ve pointed out a valid problem, and the solution we’re recommending neutralizes a bunch of huge advantages to our product that we sold you on. So fix that yourself…but continue to pay us.”

Clearly, this isn’t a panacea.

Possible Solutions

Support for early-stage transformation is not prevalent in the current marketplace, but some of this can be chalked up to the fact that the market is immature. “Headless” is an idea that has just lit up in the last few years, and most of these platforms are still under heavy development as they adapt to what customers want.

Most platforms will support some event-driven processing via webhooks. When content is published, some of these systems will POST the entire contents of the published item to a specified URL, which can then take actions either on external systems or on the originating system via its API (the second case study in this post describes how Contentful does this and provides sample code for receiving a webhook call).

Using these frameworks, we could transform an object. When News Article X is saved, a webhook could receive that request and expand the macros in it…but then what? What does the webhook do with the resulting data?

Ideally, we would push our result back into the content object itself, perhaps in a “hidden” field which editors can’t manage from the interface. With this, a content object might have a “Body” field and a “Processed Body” or “HTML” field. Consuming channels could use either as they needed.

Pretty quickly, however, we’re going to run into a couple annoying problems: (1) webhook recursion and (2) version proliferation. If I was to save this data in a field and republish, I might trigger the same webhook again. Additionally, I now get two versions of every content object when I publish, which can confuse auditing and governance processes.

What I’m not seeing in the market right now are either of these two options:

1. “Silent” saving/publish capability, which means the ability to save/publish and explicitly avoid triggering events like webhook calls. In the absence of this, I’d like to at least some explicit nod toward managing webhook recursion – some method of identifying when something has been published by the very webhook it’s about to call again.

2. An ability to save into the current version. Otherwise, you’re going to end up with an extra version every time you publish.

Clearly, both these situations are exceptions, but in the circumstances we’re describing, they’re exactly what they need. (Consider too that they exist in many other, more traditional CMS, so they’re part of the mainstream “CMS development toolkit,” so to speak.)

Barring this option, there’s a solution, which might be a little hack-ish –

We could update a “companion” content object. Meaning, we might have a “News Article” type and a “Published News Article” type. The latter would contain some extra fields to store the transformed result. When I save a version of the former, a webhook or event actually copies its content – along with some post-transformation data – to its “twin.” Consuming channels would only be allowed to access the companion object – the “Published News Article,” in our example.

What we’re doing here is sort of phantom decoupling. We’re pushing pure content into a separate location (really a separate object) from which we’ll allow delivery. (There’s some precedent here in ECM. Many of those systems have a concept of “workspaces,” and when content is published, it’s often pushed from one workspace to another.)

Another feature I’m waiting for a vendor to embrace is publishing “filters” which can transform content from management to delivery layer. The secret of headless CMS is that most of them are actually decoupled. You manage the content in one environment, and when you publish, it’s being pushed to another environment which is optimized for mass delivery. This usually means a separate environment on the vendor’s infrastructure, or in many cases, a full-blown CDN. So your content object is actually “set free” into a larger environment when you publish (I’ve measured in this Contentful – it takes a couple seconds to proliferate around the world into Fastly and CloudFront).

With this architecture, I’d like to see a publish webhook which allows me to affect what gets pushed into the delivery CDN. So when I publish content from the management environment, a webhook is called, from which I can return modified data. That data is what gets published. To use a terrible metaphor: published content is “bounced” off a webhook, and the CDN catches the rebound.

This would allow me to do some content transformation (adding expanded content to an extra field, for example), and publish the transformed version while maintaining the purity of the original.

I had some discussions with vendors around these points, and made a little progress when making my case:

• Directus actually implemented a publish event hook for me (and sent me the commit to prove it). So you can now run a transform event when content is requested, but before it’s given back. My reading of their code (it’s open-source, remember) tells me that you could implement early stage templating through an event hook. That doesn’t necessarily cache, but it’s a step in the right direction.

• One of the developers working on Contentful engaged with me in a running discussion about the idea of “metadata” on a content object. This is, simply, additional data that’s tacked onto content. It’s not versioned, you can’t manage it from the interface, and it’s not “first order” content, but you could use it to transform content and “attach” the result (for instance, the result of a transform) to be delivered with the content. This idea is just a discussion at the moment, but I like where it’s going.

• The new version of Evoq(which will be “hybrid headless”) has a concept of “visualizers” which are methods to template content prior to delivery via an API. This means you can get the raw content, but also a repository-centric transformation of it.

At the core of these solutions is a desire to avoid the BFF we discussed in the previous section. If I can just do a transformation and get the transformed result somehow back onto the content itself then I get to keep two things which are critical to the entire headless value proposition: (1) the performance of their delivery environment (which I’m paying for whether I use it or not), and (2) their native search API.

Conclusion

Headless CMS wholeheartedly embraced the paradigm of “separate content from presentation.” I don’t fault them for this, but this can get destructively dogmatic in some ways. The leap from raw to presentation isn’t necessarily a one-step process. Rather, we should look at this as a pipeline or a drivetrain that refines content through a series of steps.

There is value in transforming content close to the repository, or progressively denormalizing content in such a way to increase manageability, performance, and consistency in the consuming channels. Headless vendors need to take some steps in this direction.