Defining success and metrics in DevRel
What does “success in DevRel” look like? And how do you measure whether you are being successful or not?
Sounds like a tough question to answer and in fact is - but measuring the impact of your activities is key to success. Defining success and metrics in Developer Relations and Developer Marketing was the central theme of the Future Developer Summit Episode 3.
Don’t worry if you missed it, that’s why this article is here. Together, we’ll be going through the highlights, while Episode 4 is coming on December 8. You can explore all the highlights from Episode 2 and Episode 1 if reading this leaves you wanting more (I hope it does).
You can also watch all 2021 sessions: https://vimeo.com/showcase/futuredevelopersummit
Let’s get started
Moschoula Kramvousanou, Head of Client Relations at SlashData welcomed everyone to Episode 3 and gave us a short preview of what we will be exploring throughout this event. Then, she introduced the first session which was a developer trends update on what your internal developer team wants.
Retain or Retrain? Internal DevRel success
Richard Muir, Data Journalist at SlashData, presented the latest data on what developers expect and need from the company they work for and how you can address these needs for internal DevRel success. All insights shown were the answers that 19,000 developers shared in the Developer Nation survey, and were presented to an audience for the first time.
Here are some highlights:
Though compensation is important, very few developers are purely financially motivated to switch employers
The majority of developers consider learning and development when deciding to change jobs
Don’t ignore the ‘softer’ side – company culture, environment, and flexible working play a strong role
Eastern European developers are more focused on financials and their career, but Chinese developers care more about culture and environment.
Panel Discussion | Defining Success & Metrics
Moschoula then introduced the much-awaited panel members:
Lori Fraleigh, Principal Group Product Manager, Azure SDKs at Microsoft
Christie Fidura, Director, Global Developer Marketing at Salesforce
Jennifer Hooper, Sr. Director, Developer Marketing, Brand & Content at Armory
Amara Graham, Head of Developer Experience at Camunda
Here are the topics discussed.
On defining the right metrics
There’s a fear of measuring because you are afraid of the result. This devalues the power of what you are trying to do. Measure everything! Metrics are our friend. Start somewhere, measure and use that information. Even if it takes you from 0 to 1, that’s a 50% increase. The most important thing to get started is to get comfortable with the idea that you need metrics and then start measuring. Don’t focus so much on tools, start measuring now. We [at Salesforce] don’t have a transactional relationship with developers, so our focus is on community. We try to find a member of the community and understand them and their pain points and then tackle those issues. Try and find what success means to [your developers] and take that back to work on that.
Do developer personas feel engaged with what we are creating/producing on the documentation side? We see how they interact with it and then try to figure out if they feel enabled. I’m looking at how people are sipping through that content and then on what they look for. Are they looking for more? Also, we’re working on a way to help them without them needing to go through our whole content to make a decision for the problem they are trying to solve.
We start at understanding the adoption of the SDK. Why is someone using an older SDK? Is it an awareness problem or understanding the SDK features? We then take action and see how the metric changes.
We try to understand the context. What is success? We think about it from the user perspective. How are people looking into things? What is a meaningful metric? And from that, we get smarter and smarter.
Quality metrics: How important is engagement?
It is really important that you are doing the right thing for the right people. Your efforts might not matter if they are for the wrong people. In fact, it might even be harmful to you, the trust in your brand and your communication. If you make users think you are talking about X but you are talking about Y, you have lost their trust. For example, if you use a lot of popular keywords to make your content rank better and attract lots of clicks, and then your content does not 100% match these keywords, even though you might show that you are getting a lot of clicks, they are from the wrong people. People who visited your page looking for something else than what they got. At Heroku, we focused on who is using your product and understand the personas to avoid sending false signals and getting people engaged.
I would rather have one person who wants to hear from me than 1,000 who don’t. I like unsubscribes, it’s from people who don’t want to be here and this saves me time and effort I would spend trying to talk to them. We are hypnotized by social media and seeing big numbers. But the small, more impactful number is the one that matters and can take your communication to the next level.
Looking at docs and forum posts, we see big numbers of visits and it’s good to have these pageviews on your radar; unfortunately, most of the time, the best performing posts are the most generic ones. We had a page ranking #1 that was about a specific error that somehow became a very popular search. I advocated that we remove it from the index because it was not a valuable entry point to our technology or product. You’re sacrificing pageviews but this is not the quality engagement and enablement you are looking for. The pages that generate the most traffic should be helping the company or developers get what they need instead of funnelling traffic that is of low quality to us.
How do you remove friction on the product side?
We try to quantify the “getting started” experience: the first ‘Hello World’ or API call. It’s not about the metric itself - it’s about understanding where that friction is. We also measured “time to publish” an app in our app store. We measured how developers spend their time submitting an app and used tools to help them solve these problems so they can spend more time helping their users.
Hello, world! aka Q&A time
Community quantifiable metrics to address company goals
Go super-granular. Instead of trying to solve X amount of questions, focus on the most important ones. The ones you can off-load from support. Try getting answers to these questions through either a piece of content or directing them to a forum post and make this a one-touch contact. That doesn’t necessarily reduce tickets, but it does reduce the number of one-touch tickets. The other thing is focusing on a topic area to reduce the number of tickets and make it self-served, instead of going through support. Don’t focus on reducing all tickets across different topics at once. Focus on one at a time and do it often. Then you can produce a metric to show how tickets have been reduced across different themes/topics.
How can I say that X amount of features have come from community feedback?
Our SDK is on GitHub so we make sure to tag the requests from the community and then track them.
How do you measure and bring together quantitative and qualitative feedback? Which tools do you use?
I will not present a number without some kind of context. It can be very harmful to do that. My example: I will not often present the number of page views. I stay away from that because it doesn’t give you context. Did people get what they need? It doesn’t tell an interesting story. Better provide context, quotes, history along with the numbers.
A spreadsheet is enough for me. It’s all about what you got in it. Context is important. People are afraid to ask the community what they feel. Why be afraid to ask the people who will ultimately help you shape what you deliver? Getting their feedback only means that what you are producing is right for them. I will reach out to our community members and ask for their feedback “you can help me make things better for you”. These are your people, it is worth having their opinion, more than the people in your organisation.
Do you survey your developers? How often?
At Microsoft, we have surveys - in-product - that run continuously. We’re constantly tracking NPS and getting direct feedback on products and docs. It’s a constant experience.
Nothing beats direct customer feedback. But you need to be open to it. Don’t fall into the trap of disagreeing with their opinion. What someone is communicating with you, is their perception of what you offer. You need to make sure you are speaking in the right context and not just relying on your internal (company) audience. There is nothing more valuable than getting feedback from the end-user perspective. Even someone who is the biggest fan of your product has their own perceptions of the product.
What is your opinion on NPS? Do you do developer and partner NPS differently?
Figure out what you are trying to get that score on. Are you trying your NPS for services, content or community? I do struggle with NPS for content. You should be producing quality content. Developers are looking for content, what we might miss out on is the channels they are looking for. The question is “do you feel satisfied”? I’m not asking “how am I doing as a developer marketer” I’m asking “am I offering you all you need to succeed?”. Because that’s what’s important. I need to know what it is that you [the developer] need. That’s what the survey is for. I look at it as a holistic thing instead of NPS.
If you are using Azure because your company is the customer and you are not happy with it, you are not going to be a promoter. It’s not a developer vs partner thing. It’s about whether developers using our product are happy.
What tools are you using to determine whether users are interacting vs just looking?
I use Google Analytics, partly because I am focused on documentation. So we can see who is working in our cloud or platform product. It shows the vanity metrics. What you need to do if you’re looking at documentation, is be careful how you interpret those metrics. A glossary page can have a high bounce rate, higher than a tutorial, because many users quickly get what they need and then leave. Look at your user flows, try to see how people are stepping through your documentation. Where they enter and where they leave, what they are looking for. This way, you can help them access the information they are looking for, with fewer clicks. Also, look at search terms. What people are looking for and what results they are getting. Investigate further, are they looking for something your product can do or is it something that it can’t do?
All hail SEO!
What’s your final word of advice?
Jennifer: Decide why that metric is important. Understand why what you are measuring is important, the context.
Christie: All hail SEO! Find out what people are looking for and give it to them. Don’t be afraid to measure.
Amara: Nothing is set in stone. If you don’t like the metrics you are using today or what you are tracking is not right for you, make a note and move on. It’s OK to choose a different direction or say “success means something different to us now”.
Lori: Don’t be afraid to embrace the red. If the metric doesn’t tell the story you would like it to tell, embrace it and get the qualitative data to give you the context and pivot to the right way for success.
Episode 4 is coming on December 8. The episode’s theme is “The Evolution of Developers & DevRel” and we will be looking at the evolution and the future of developers and developer relations, featuring:
An industry panel A trends update on the evolution of emerging technologies
A community-chosen Masterclass on supporting developers in their journey from student to specialist and
A Fireside Chat with Stack Overflow
You can preview the full agenda here:
Episode 4: The Evolution of Developers & DevRel
December 8, 2021 | Time is shown in PT
07:30 Refocus energy and clear mind
Mindfulness meditation session
Unwind and tune in with a 20 min well-being session. Open to all, whether you have lots of experience in meditating or none at all.
08:00 The evolution of emerging technologies: What does the future hold?
Q3 Developer Trends with Michael Condon, Data Storyteller at SlashData
We will discuss the evolution and future of emerging technologies, based on rich historical data and current trends.
08:20 The evolution of Developers & DevRel
Industry Panel with:
Amir Shevat, Head of Product - Developer Platform at Twitter
Arabella David, Head of Developer Marketing at WhatsApp
Jennifer Sable Lopez, Sr. Director, Developer Relations at OutSystems
Annie Mathew, Director, Developer Relations at Microsoft
09:00 Networking Breakout / Comfort Break ☕
09:20 A developer’s journey from student to specialist, and how to support them!*
Master Class with Christina Voskoglou, Sr. Director of Research at SlashData
10:00 Stack Overflow Fireside Chat*
10:30 Thought Leader Networking*
* Accessible to Thought Leader Pass holders.