Staying true to our mission - enabling learning and experience sharing among community members - we are hosting regular DevRelX Community Sessions, hour-long, community-led conversations. At every session, our community members present the topic they’re experts in and, together with a group of peers in Developer Relations, discuss challenges and share insights on various topics.
Would like to participate in future sessions? Join the DevRelX community.
This article summarises the third session that took place on June 22, titled "Hacking Your Community Metrics & Engagement". Alex Angel, Chief Community Officer at Commsor, presented tips and led a group discussion on:
What metrics actually matter when measuring "success" for your community?
What is "good" engagement and how do you report on it?
What do you do when your efforts aren't working?
Other session participants were:
Michael Hall, Engineering Manager, Community and Developer Relations at InfluxData
JoAnn Peach, Global Developer Marketing, Developer Program at Nvidia
Courtney Robertson, Web Design and Developer Advocate at GoDaddy
Gabs Ferreira, Content & Community at Alvin
Kenny Rogers, Developer Advocate at Stacks Open Internet Foundation
Richard Muir, Data Journalist at SlashData
1. Define metrics and OKRs
Alex: When we’re talking about community success, there's lots of chatter about engagement and everyone is super focused on engagement and how to measure it. I want to dispel some myths around it and hopefully make you all feel more confident in the work that you are doing or the strategies that your teams are working on. Lastly, I’d like us to dig in a little to what you do if things aren't working or how you make the call of whether it's going no go for what you've been pursuing or what the data is telling you.
Let’s start with the main thing that I want to get across here - there's no holy grail metric. Many conversations I am involved in when I'm consulting with organizations or chatting with people who are in our community (Community Club) always come down to a question “What’s the one thing I should be focused on?”. Well, it all depends on what are your business goals. Sometimes it feels limiting to frame all of your community-building efforts into that business context, but if you are building a community around the business, everything should tie back to the organizational goals and OKRs, whether it’s your community, DevRel or marketing efforts.
Think what are the things that you are doing and how they tie back to other teams within the organization. Take these relationships that you’re building, connections that you’re making and content that you’re creating in the community context and tie it back to specific metrics like revenue, retention, lead generation or other goals that are teams are focused on. This will help you show the true value that you’re creating for the organization.
It’s important to stick to this mindset when you’re doing your quarterly planning and setting the OKRs. A common trap people fall into it’s a flat focus on the numbers. It’s less about the raw numbers themselves like how many people are in the community or on our social media.
You can easily track the number of your event attendees, but what does that actually mean?
The context is what matters the most. You should think “we’re tracking the number of attendees because we have recognised that if someone attends an event they are more likely to buy our product or retain, or do X, Y and Z behaviours.” This is the type of context that is really important in helping you focus on metrics that matter to your company.
Every community and every organization is different, so you have to understand what are those top priorities for your company and align your goals and metrics with them.
For example, you want to know how someone being part of the community impacts their retention in our product, how it impacts product usage, member satisfaction and the likelihood to upsell if they're already a customer. If they aren't a customer and your community is dedicated to being at the top of the funnel, then what's the impact on the likelihood to close or how does it affect the time to close the sale? Each of those could be viewed from a revenue perspective and while they can be difficult to track at times, especially if don't have full access to that data, this type of analysis helps understand how community impact other parts of the business.
If it's an open source community, the value is in figuring out the impact on the product. Look at how many bug reports are filed by community members, how many of them are resolved by other members, how many ideas are sourced from the community, and how many features have contributions from community members - all of these are among many things that you can be thinking of when building a community.
Now I would like to pass it over to the group to hear where you’ve been, where you’re at and hoping to go.
Developer ecosystems and personas
Kenny: I think there are a few challenges when it comes to measuring developer-specific efforts. One of the most common metrics is monthly active developers but for example, in our case, we're focused on growing an ecosystem of interrelated tools, a decentralized protocol, and a smart contract language, so there are different parts and not one tool or product that I can measure the use of. But having to navigate a bunch of different data points can feel like you're just throwing numbers together. Consider the few most valuable metrics without getting distracted by the ones that maybe don't necessarily mean anything.
Avoid overmeasuring and know you won't be going to be able to attach a metric to every single thing that you're working on.
Stick to things that you can measure and use to that to get a general pulse of your community.
Alex: I think it's equally as important to have that qualitative data to support your quantitative results. I don't think you're overthinking it, it just comes down to what can you as one person or a small team reasonably do. If we can't really support this type of work, that's totally fine but it’s exactly that context piece right there that I was referring to. Understanding how your ecosystem is actually performing and how people are feeling about it is as important.
Jo: I see a similar challenge too. We have a broad swath of different developers, from game developers and researchers, and we have all sorts of different efforts that are focused on our different products. Trying to match community metrics with those separate developer communities and developer programs can get quite challenging. It comes down to trying to figure out what those commonalities are, while the game developer isn't necessarily going to care about what they're doing in the omniverse community, or data scientists are going to have a completely different mindset than other developers. If you have only managed developer programs that have been focused on one specific product, this will be a new ball game.
Working with different product teams that have community managers is key in defining what that baseline is and what platform should you use across the board for all of your different products.
Alex: I'm seeing more and more companies take a similar approach when it comes to strategy for their DevRel or traditional B2B or B2C community. Maybe it's fine to break it out and to have these different personas and understand how each of those operates within the ecosystem by associating relevant metrics with each of them.
Now all of these different audiences will have different touch points, so you need to have some overall objectives that essentially everything ladders up to, and then just getting the two and those directly tie back to the company objectives and vision. Then under that, each persona or function can have its own two to three things that are going to be those key results.
Jo: Completely agree with the personas. What I’m wondering about is what comes next. For example, your main objective is engagement, with your content, training engagement and different subprograms within the developer program. How do you define what that engagement should look like for each of those different personas?
Courtney: I organized a group of DevRel people that are the WordPress ecosystem and a few months back we got chatting together about wanting to meet and connect with our counterparts. Given that this is open source work we're peers even if we're at different companies even if, and we also might represent some work happening inside WordPress and as well as our organisations. What is interesting is that I don't seem to come across any consistent DevRel metrics for the WordPress projects. Each of us has our own employers and initiatives, for instance, I would be in the developer education area within the WordPress space. It also relates to the work I’m doing at GoDaddy as a developer advocate for building our first community. I'm trying to wrap my head around a really broad understanding of DevRel inside of the WordPress ecosystem and Open Source.
Members before metrics
Rich: I think when we’re talking about community success, leading it back to the bottom line of the business is one important way to look at it. But I also think that metrics that can indicate the health and strength of the community are also those around the interactions between the members.
These interactions may not have direct impact to the business, but community should only exist if people are finding value in it.
Sometimes this could mean looking at the interaction between community members and not necessarily interacting with the content that is posted by the community managers. Think about the independence of the community, how long would the community last and sustain if community managers just stopped doing anything? Is the community self-sustaining, are members active enough without these prompts and energy shots? And think about new member referrals - are your members bringing in their peers?
Alex: The sticker the community is the more relationships and connections each of the members are going to have with each other. Being able to understand how those relationships are forming is important because then you can figure out what types of activities you or the rest of your team can be doing to help facilitate that further and to make sure that the community environment you have this is going to sustain that. But then you can also tie that directly back to the impact that it has on the organization.
For example, if you know that Member X is incredibly active and is chatting with 50 different people in the community as compared to Member Y who's only talking to 10 and isn't quite as active, you can see how that behaviour and those connections within the community tie to the behaviour in the rest of the ecosystem. It's all about pattern recognition and people's behaviour and community building. It's wonderful in that it is a human connection and at the same time it's a bit of a social experiment for businesses.
2. Turning numbers into information
Gabs: I’m also curious what tools are folks using to measure all those different metrics across different platforms.
Alex: I’m biased as I’m part of the Commsor team and naturally we’re using Commsor for our community as well. And the reason I joined is that back then there were no tools to help you with that, and a lot of that work was happening over endless spreadsheets. Now there are companies like Commsor and other tools out there that help you get those top-level insights but then you still have to dig into the specific sand context that we talked about. That could mean doing your own research or survey and analysing what that data means. Curious what others think.
Micheal: Before I was working on my own as a community manager and was building a tool called Savannah. One thing that I've found while working on it and using it is that it's really easy to get a whole lot of numbers but not a lot of information. Trying to separate or trying to find the signal in all of the metrics’ noise is a difficult thing. You have to be really mindful about why you're measuring what you're measuring and what questions you're trying to answer. If you just come into it hoping that answers will pop out without questions, then you're just going to get lost in the metrics.
Rich: Think about the metrics in the way that you affect what you measure. Whatever metrics you take is what you move towards, so if you’re focusing purely on the benefit of the business, there could be a risk of becoming inauthentic in some ways. The business bottom line should not be the ultimate aim of the community.
Michael: Think about the story that you can tell based on what's going on in your community or the value that it’s bringing. Something I like to focus on and highlight is whenever I see somebody from our community, whether they're a customer or they're using our open source product, who is actively answering questions and helping other users and customers in our community. Even if there are no numbers associated with that, just being able to show that not only are they happy with our product but they're so happy with it that they want other people to be happy with our product. That says more than a graph showing an increased number of people in the community does.
Kudos to Alex and the rest of the group for a session filled with learning!
Blog with Part 2 of this session will follow soon, stay tuned.