Open Success Measurement

After more than two decades of measuring success at large digital media organizations, I recently had an experience that caused me to rethink everything. I’ll show you how I built a system that allows editors to set up their own experiments and track conversions entirely within their CMS and without any developer support.

But first I’ll tell you how my old thinking went (and maybe this seems familiar to you):

Content producers are too busy to actively manage their own digital experiments and therefore their use of analytics tools should be limited to simplified dashboards. Further, analytics should be a compromise between developers — who build for maintainability and scale — and content producers — who require a more flexible platform for testing different content approaches.

— The nattering inside the head of every digital strategist

Two recent consulting engagements rocked my belief system and forced me to build a new model. On one project, I was advising a select group of newsrooms around the world on how to make success-measurement more accessible to content producers. At the same time, I was setting up success analytics on the web and across a suite of third-party services for a leading U.S. broadcaster. Together, these projects forced me to simultaneously work as an analyst at both the 10,000 foot level and at the practitioner level.

The problem statement

In my first project, I was working with a leading developer I had known previously from my time at NPR. Our charge was to find a way for multiple newsrooms to collaboratively build an optimized user conversion point for a newsletter sign up or a membership appeal. On the one hand, we wanted to iterate on a single, cross-newsroom component to determine that, say, a red button outperforms a blue button. On the other hand, we recognized that the design was probably less important than the content, and the individual newsrooms each had their own opinion of what content would work best for their brand.

In the second project, I needed a quick way of seeing how many people were engaging with various appeals across the site for donating or signing up for a newsletter or podcast. Although the broadcaster had no immediate need to test multiple designs, I wanted to engineer it in a way that would allow them to test in the future without development or extensive retooling of their data layer.

Set up analytics

This tutorial assumes you have already set up an analytics system that works with your CMS, such as Google Analytics and Google Tag Manager. If you’re using WordPress, this is super simple with the Site Kit wizard. You’ll also want to disable insertion of the GA code on your page, since this is handled by GTM. To make this work, you must:

1. Create a GA Tag with the tag type GA4 Configuration. Call it GA4
2. Enter your stream ID from the GA4 property in your analytics admin as the measurement ID.

From now on, select GA4 in the Configuration Tag dropdown menu for any future tags.

Configure variables for your CMS

Think of GTM variables as your tool-set. Google could enable all of them out of the box, but then you’d be overwhelmed. At the same time, you’re going to want the right tools for the job. Below is my toolset:

The variables in the top box require no configuration. Just select the Configure button at the top and click the ones you think you’ll need. You can always add more later. The GA variable is the only User-Defined variable you’ll need, and you should have defined that already. I also recommend adding the top one because it allows you to assign triggers or labels from an image alt tag — add this by clicking New, select Auto-Event Variable and set Element Attribute as the Variable type and type alt as the Attribute name. I’d also recommend the bottom one because it allows you to assign triggers or labels from the page name — add this by clicking New, select JavaScript Variable and type document.title in the Global Variable Name field.

Create a generic click listener

If you haven’t created a generic click listener in the past, you’ll want to add this one before going further. Although this is technically a trigger, we’ll call it a listener because it won’t be used by any tags. Its sole purpose will be to help us better understand how various components in our CMS make data available to our tag manager.

In GTM, click Triggers and the New button on the trigger panel. Now set up a generic click trigger that tracks all clicks, which is the default option. It should look something like this when you’re done:

Explore CMS-accessible Click Variables

Let’s put that generic click listener to work. Before you can start setting tags, you need to understand how GTM reads the specific user interaction you’re trying to track. If the user clicks a button, for example, is GTM taking the trigger from the button element, the link element, some image element, or the text element? To find out, click Preview. The assistant will load the site in a new window and start tracking every click (because of the generic click listener you set earlier). I clicked a podcast subscribe badge. Here’s what the Google Tag Assistant saw:

Note the variables and values that are available to define our tags and triggers. Can you spot the one that is easily available from the CMS? As a hint, take a look at the media edit screen for this particular image in WordPress:

We’ve discovered that the Alternative Text is a variable that is easily accessible from inside the CMS and is also picked up by GTM when a user clicks on that podcast badge.

GTM doesn’t care whether it’s easily editable within the CMS. In fact, some developers and analytics specialists may even want to tag variables that are not easy to edit. But that’s inherently an exclusionary philosophy. It’s effectively saying, if I give an editor this power, they will break my shit.

This misses the point of what we’re trying to do here. Experiments are supposed to be ephemeral, lightweight and breakable. These are sandcastles that help us learn. When the experiment is over, commend it to the sea!

Create a flexible trigger

The trick is to build flexible triggers, rather than the fixed triggers we usually set in GTM. For instance:

Note that the trigger on the left only applies to elements with a specific class on a specific page. This is useful for my purpose of seeing how many people on the books page click a purchase link, but it isn’t something an editor could reuse elsewhere on the site. The one on the right will trigger any time an editor includes the word subscribe in the alt text of an image.

You could make this even more generic (and at the same time protect against false positives) by using a reserve prefix like hbSubscribe. Just remember, your alt text is doing double-duty as a descriptive tag for screen-readers, so don’t go too far off the reservation.

Create a dynamic tag

This approach becomes even more valuable if you allow for dynamic content (where the label in your GA report changes depending on the content). Begin, as you normally would, by configuring the tag (click Tags select New, click anywhere in the Tag configuration box and then select Google Analytics, Universal Analytics). After selecting Google Analytics: Universal Analytics from the list of options of where to send the data, be sure to click the dropdown that says Select Settings Variable… and select your GA variable. If you fail to do this step, all other work will be lost when you click Save. You will not get an error message, just endless confusion. Because Google.

Next, select Event as the track type and fill in your tracking parameters. As always, how you label these parameters (or if you label them at all) has no impact on whether the code will work, but the labeling will determine how this event will show up in GA. What you put on Category and Action will both show up on the Events table, and either Label or Value will show up in the table that appears when a user clicks Action in GA. So good labeling might look something like this:

Category	Audio
Action	Podcast Play
Label	Title of Podcast
Value	{{ minutes played }}

Note that you’ll probably never fill in a Label AND a Value on the same tag. I do it here only to demonstrate that Labels are for qualitative and values are for quantitative information.

In the case of our podcast tracking experiment, my final tag will look something like this:

Note that the Action is specific enough to be understandable to anyone, and the Label is based on the full alt tag assigned by the editor. Be sure to hit Preview to test that the tag is firing properly and then hit Submit to push your changes live to the site. Within a few minutes, you should start seeing results:

Et voila! You now have an experimentation tag and trigger that is just as narrowly confined as you need it to be understandable to analysts, but which offers as much flexibility for open experimentation as your editors require.

Tell your editors how to build their experiments

Now you need to tell your editors that they can start uploading podcast subscribe badges with abandon, and to be sure to include the word “subscribe” (or whatever you set) in the alt text, and to make the rest of the alt text as descriptive as possible.

Conclusion

This philosophy doesn’t live and die by the alt tag. I also have open experiments with buttons, where the trigger is the url of our newsletter signup page and the Label is the button text. So if an editor changes the text of the inline button without telling me, I’ll be able to call her up and report that her new language is beating the pants off the old language.

And that’s the point, really. Instead of looking at conversion points as the protected domain of a rambling digital bureaucracy, your content producers will start to take back their instincts. But now, they’ll be doing it with constant feedback from the audience.