When running a Developer Relations program, it's typically not dollars or leads that directly define your success, but whether you're enabling developers to get their job done. Measuring that can be a tricky task, but at Orbit's Community Camp virtual event, four experienced DevRel leaders shared their advice on measuring success in Developer Relations:
Joe Nash, Staff Developer Educator at Twilio
Mary Thengvall, Director of Developer Relations at Camunda
Matthew Revell, Developer Relations Consultant and founder of DevRelCon
Jeremy Meiss, Director of DevRel and Community at CircleCI
This post summarises the key takeaways, detailing how you can pick the right metrics and put them into practice so you can deliver on your DevRel goals.
WHAT MAKES FOR A GOOD METRIC?
What makes for a good metric will depend on what success looks like to you. They should help you track progress towards a set of goals or milestones. For Joe Nash, getting to those stats starts with the questions you're asking or are being asked of you and your team. In particular:
What is the reason that someone is paying for this community to happen?
What is it that community members have come here to do?
The answers to those foundational questions will help in "making sure that you are serving those needs and that you are tracking metrics that help you know you're serving those needs," says Joe.
Of course, DevRel doesn't happen in a bubble, and it's not only the community's needs that matter. That's why Jeremy Meiss advises folks to "make sure that the metrics you've identified for success track up to something else that the business wants... you've got to be able to show how it helps the bottom line for the company. And answer that question before it's even asked." So another foundational question Mary Thengvall urges you to consider is "what are the goals of the company right now?" and find a way to work those back to the impact DevRel is having. "I'm going to be tracking the things that show the value that we're bringing to the company," says Mary, "and how that also impacts the community. Because we have to keep both of those things in mind when we're tracking metrics."
SUGGESTED SUCCESS METRICS
The right metrics for your DevRel program will follow from your goals and the needs of the business in terms of reporting, but there's no lack of examples, and the panel shared some that have worked well for them in the past, too.
For Matthew Revell, some stats he's tracked across almost all of the DevRel programs he's been involved in tell you, "how many people are coming in? How many people are staying? For how long? And then... how happy are they?" Everything else falls out of that, I think."
An important one for Mary is "DevRel qualified leads." That's useful for DevRel teams who are "making all of these fantastic introductions from the community back into the company." Putting "a term around that process and measuring the number of them can change the whole way the company views them," says Mary. While Matthew suggests "attributable revenue" is "a really good thing to show to people to carry on having them fund what you're doing." For Jeremy, a good metric is "returning users, and what percentage of the total number of active users are returning because for us, if we can raise that... it means we're getting more and more people to care about what we're doing and caring about our products."
Whereas Joe suggests tracking engagement. "If you look at engagement as an incentive system, it's far easier to see what engagement means and where engagement is. You're looking for people to perform the activities that you want and engaging with the rewards that they're able to get from performing those activities in a meaningful way." This is useful because if metrics suggest that one of your engagement loops is failing, you can dig in by asking yourself, "Is that a case of the incentive not being strong enough or the cost of that activity [being too high] for that member?"
Jeremy also suggests reaching out to others, both within and outside of your organization, to get ideas about metics. They could come "from your data team," says Jeremy, or from "talking with other community managers - hop on a call... and bounce some ideas off. See what's worked for them. Chances are, there's somebody out there that's tried it, or is currently trying something that you're thinking about". So use that knowledge "to inform whether or not that's a metric that's going to work for your community."
A common failure case to avoid early on is the temptation to try and measure everything. As Mary Thengvall cautions, "if I don't know why I'm tracking a particular metric... it's not going to get me the answers that I'm looking for." A good rule of thumb here is if a metric goes up or down, and in response, your behavior doesn't change, then you shouldn't measure it. When a metric changes, it should encourage or discourage you from undertaking an activity, cause you to re-think your strategy, or try different tactics to achieve your goal. If that's not happening, then you're not benefitting from measuring it.
GETTING STARTED WITH GOAL SETTING
When you're setting a metric for the first time, it can be difficult to set a goal for it. How do you know what a realistic number is? Jeremy says you need to start small: "I think it's important to get engaged, and then from there, you have a baseline to build from."
You can accelerate finding appropriate metrics by iterating with experiments. "I love experiments," says Mary. "It gives you the ability to be more flexible" and iterate quickly, without getting bogged down in building out the underlying infrastructure. The experimentation process should be one of discovery though, warns Mary - one that's "less about trying to find metrics that fit the story that you already have in your mind and more about asking the questions that you want to find answers to, and then seeing where that leads you." Being open to following the data also extends to so-called vanity metrics too. As Mary notes, "there are times when tracking vanity metrics makes sense... it gives you a way to spot patterns." For example, if you see a spike in a number on a particular day, digging into what caused that spike might lead you to understand the actions and activities causing your stats to change better.
TIME-FRAMES AND REVIEW PERIODS
Once you've decided on a set of metrics, it's useful to consider how often you should monitor and review them. Joe suggests that "there are definitely numbers that are useful to keep track of almost day to day: engagement, and the number of messages sent in community spaces," for example. But some metrics come from systems owned by other teams. "Keeping tabs on them often requires checking in... so those tend to fall off the radar really quickly. And that's where I think the danger lies," says Joe.
For Jeremy, the right review period is more clear-cut: "weekly... it's easier to plan and make adjustments when you're looking at them on a regular cadence like that than to do it every other week or monthly," so you can be reactive and make small adjustments to activities along the way. Mary agrees, "especially when you're doing experiments, tracking weekly, or more regularly than that if you're trying to iterate quickly, is extremely important."
TELLING A STORY WITH NUMBERS
One last tip the group had was about the importance of getting beyond the raw numbers. "Numbers are great," says Mary, but "being able to shape them into a narrative" is what really matters. As Matthew says, "numbers in their raw form are kind of uninteresting; you have to tell a story to put them in context." And that can take time and repetition, especially when communicating with folks beyond your own team. "The first time or six times that you present those metrics," says Jeremy, you need to be "making sure that you are providing context, explaining what it is, and why it matters." The best way to achieve that is through a story that pulls together the disparate stats you measure into a clear narrative that makes it easy for others to see the result that comes from them, "and not just numbers."
That story though will only be as good as the underlying metrics you're following. When you've found the right ones, they'll keep you on track and working towards your goals as well as aligned with both the needs of your community and the business.
Want to know more about how you can define and measure success in Developer Relations? Join us on October 6 at the Future Developer Summit Episode 3: Defining success & metrics. You can grab your free Community Pass or a Thought Leader Pass for a richer experience.
