page contents

Set up your beta testing for success

Bringing early users into the beta version of your software can be stressfully fun.  User testing is always fun and working for years in the software industry they are always so insightful.  No matter how many times I have done it, I always get excited and a few butterflies.  Any release of software is taxing on your nerves.  It is the culmination of all your user stories, requirements and countless hours of development time.  The entire process needs to viewed from multiple perspectives.    

The Marketer Perspective

How will the impressions of this beta version affect the overall brand?  Remember there is no such thing as a neutral brand interaction.  You are either increasing or decreasing. How will the communications inbound and outbound be routed and measured?  If there are new automated messages that will be generated from the software it is important to make sure that your branding is consistent.  How does the new system exceed the original brand promise?  Of course, the new release is going to provide additional features and functionality but which ones matter most to which customers?  It is important to ensure that early users see the original brand value.  

The Customer Success Perspective 

Your customers are your most cherished assets.  If you are part of a larger organization that has the resources for a dedicated customer success team they must be involved in the plan and communications for all beta testing.  How will beta users be recruited?  Are you going to use customers only, non-customers, or a mix?  

The Development Perspective 

This is going to be the stress test for all the hours of programming.  Did we get the requirements right?  Even after a good QC review, users always have a tendency to find something.  How will the results be communicated back to development and what is the mechanism for tracking and prioritizing any feedback or bug discoveries?  

The Customer Perspective

A company that you use/buy from has invited you to join a beta group for their next release.  Why do you want to participate?  The new version will, of course, have additional power/features that the old version did not have, which ones mean the most to you?  When I say we need to look from the customer perspective, I mean that we need to bring the value to them out.  It is our job to remove as much friction from the process as possible (that is just good marketing right).  We need to "see" through the eyes of the customer (maybe through the eyes of each persona) what the value is to them.  

The Technology Perspective

There are two main categories of the technical perspective, the software itself and the research tools.  For the software, this is the system that your beta users will be using.  Because software is very specific I will not go into too much detail here.  Depending on how your system is deployed there are many metrics that you should know.  

Just like live software you will want to track.  You should have these stats for your current software so you should have a baseline to compare this beta group to.  

  • DAU - Daily Active Users
  • WAU - Weekly Active Users
  • MAU - Monthly Active Users
  • # New Accounts (this can be broken into new Orgs and Users if applicable)
  • Usage Metrics:  This can be the number of docs/projects/images/etc... that are created in total and how many the average user creates.  

The second category is the research tools that can provide a ton of insightful data.  I have been involved in several SaaS companies so Google Analytics is where all tracking starts.  For SPA (Single Page Applications) you can setup a google property for the sub-domain for the application and you can gather some good quantitative measures.  With Google you can easily setup multiple properties to track separately.  There are several session tracking tools on the market, some are free, some are paid.  There are enterprise systems from ClickTale and Loop11 that provide a ton of integrations and features.  One of the tools that I have used frequently is HotJar.  The team at HotJar have recently added the ability to have 'incoming' feedback from users which is a very nice addition.  HotJar allows you to record a sample of user sessions.  This lets you see through your user's eyes what your software is doing.  The recordings can then be shared with the development team to track bugs and/or user interface issues.  If you are following SCRUM, the information from the recordings provides the highly valuable user impact to the backlog issue and the development team can judge the number of points it will take to correct any issue.  

Finally, having the quantitative and qualitative data collection in place will provide you and your team with the best information to make tomorrow better than today.