Set up your beta testing for success

Bringing early users into the beta version of your software can be stressfully fun.  User testing is always fun and working for years in the software industry they are always so insightful.  No matter how many times I have done it, I always get excited and a few butterflies.  Any release of software is taxing on your nerves.  It is the culmination of all your user stories, requirements and countless hours of development time.  The entire process needs to viewed from multiple perspectives.    

The Marketer Perspective

How will the impressions of this beta version affect the overall brand?  Remember there is no such thing as a neutral brand interaction.  You are either increasing or decreasing. How will the communications inbound and outbound be routed and measured?  If there are new automated messages that will be generated from the software it is important to make sure that your branding is consistent.  How does the new system exceed the original brand promise?  Of course, the new release is going to provide additional features and functionality but which ones matter most to which customers?  It is important to ensure that early users see the original brand value.  

The Customer Success Perspective 

Your customers are your most cherished assets.  If you are part of a larger organization that has the resources for a dedicated customer success team they must be involved in the plan and communications for all beta testing.  How will beta users be recruited?  Are you going to use customers only, non-customers, or a mix?  

The Development Perspective 

This is going to be the stress test for all the hours of programming.  Did we get the requirements right?  Even after a good QC review, users always have a tendency to find something.  How will the results be communicated back to development and what is the mechanism for tracking and prioritizing any feedback or bug discoveries?  

The Customer Perspective

A company that you use/buy from has invited you to join a beta group for their next release.  Why do you want to participate?  The new version will, of course, have additional power/features that the old version did not have, which ones mean the most to you?  When I say we need to look from the customer perspective, I mean that we need to bring the value to them out.  It is our job to remove as much friction from the process as possible (that is just good marketing right).  We need to "see" through the eyes of the customer (maybe through the eyes of each persona) what the value is to them.  

The Technology Perspective

There are two main categories of the technical perspective, the software itself and the research tools.  For the software, this is the system that your beta users will be using.  Because software is very specific I will not go into too much detail here.  Depending on how your system is deployed there are many metrics that you should know.  

Just like live software you will want to track.  You should have these stats for your current software so you should have a baseline to compare this beta group to.  

  • DAU - Daily Active Users
  • WAU - Weekly Active Users
  • MAU - Monthly Active Users
  • # New Accounts (this can be broken into new Orgs and Users if applicable)
  • Usage Metrics:  This can be the number of docs/projects/images/etc... that are created in total and how many the average user creates.  

The second category is the research tools that can provide a ton of insightful data.  I have been involved in several SaaS companies so Google Analytics is where all tracking starts.  For SPA (Single Page Applications) you can setup a google property for the sub-domain for the application and you can gather some good quantitative measures.  With Google you can easily setup multiple properties to track separately.  There are several session tracking tools on the market, some are free, some are paid.  There are enterprise systems from ClickTale and Loop11 that provide a ton of integrations and features.  One of the tools that I have used frequently is HotJar.  The team at HotJar have recently added the ability to have 'incoming' feedback from users which is a very nice addition.  HotJar allows you to record a sample of user sessions.  This lets you see through your user's eyes what your software is doing.  The recordings can then be shared with the development team to track bugs and/or user interface issues.  If you are following SCRUM, the information from the recordings provides the highly valuable user impact to the backlog issue and the development team can judge the number of points it will take to correct any issue.  

Finally, having the quantitative and qualitative data collection in place will provide you and your team with the best information to make tomorrow better than today.  

Top 5 Powerful Analytics KPIs

Top 5 Powerful Analytics KPIs

    Key Performance Indicators are important, and particularly important when performing text analytics with survey data.  When conducting text analytics it is important to know which KPI metric you are interested in.  There are many outputs when conducting text analytics and it is imported to have the output in mind before you begin.  The output from text analytics can be a coded list of topics with the frequency counts.  If the data is from hotel customers you may want to know how often customers mention; linens, comfort, TV, and room service.  Another objective that uses the same hotel data could be to understand if mentions of room service have an impact on NPS ratings?  Yet another KPI could be revenue.  The objective might be to explore the relationship between mentions of room service and how much a customer spends at our hotel.  In this example you are using the revenue amount as your KPI.  In this post we will discuss the five powerful KPIs that we see used frequently when conducing text analytics.   


KPI #5 is Sentiment analysis  

    Sentiment looks at the words and compares them to an established dictionary for what words are positive, which words are negative and which are neutral.  When using Sentiment as your KPI the analysis is focused on uncovering which words, combination of words and which topics are increasing or decreasing sentiment and by what amount.  Sentiment is one of the most popular KPIs in text analytics because it can be conducted on almost all unstructured text.  This makes it useful on social media data where structured variables are limited and sentiment can be used to enrich other metrics such as percentage of mentions and reach.  


KPI #4 is Customer Satisfaction Score or CSAT

    CSAT is a broad term that describes many different types of customer service survey questions. The goal of any CSAT score is to measure a customer’s satisfaction level with your company.  The scale for measuring CSAT isn’t strictly defined.  Some have it on a 5 position scale ranging from very unsatisfied to very satisfied.  Others use a score that is derived from calculating the number of happy customers divided by the number of customers asked.  


KPI #3 is Customer Effort Score or CES Score

    Customer Effort Score is a metric to measure customer service satisfaction with one single question. The belief is that service organizations create loyal customers by reducing customer effort.



KPI #2 is Net Promoter Score  

    The NPS score was created by Satmetrix in 2003.  The NPS question is “How likely is it that you would recommend [company X] to a friend or colleague?” and the answer is offered as an 11 point scale.  When consumers answer zero through six they are considered detractors.  Customers that answer with a seven or eight are passive and those that answer nine or ten are the promoters.  In the HBR article The One Number You Need to Grow, “By concentrating solely on those most enthusiastic about their rental experience, the company could focus on a key driver of profitable growth: customers who not only return to rent again but also recommend Enterprise to their friends.”  According to (the website run by Satmetrix) “More than a decade after it transformed the business world, NPS® still stands alone as the only customer experience that predicts business growth.”

There are some that are critical of and question the predictability of NPS. “To this date there is not a sigle scientific (peer reviewed) study supporting that NPS predicts growth. The study that was used to launch NPS in HBR is also flawed (and HBR is not a peer reviewed magazine)." - Sven-Tore Bengtsson (Source: Link

    When using NPS as your KPI you can explore what topics are driving up or down your NPS score.  Using the output to focus the team on what areas of focus will have an impact on the score that matters to you.  

KPI #1 is Revenue

    The KPI that is most sought after is of course revenue.  It is the R in ROI and if your data can link customer comments to the amount of revenue that is generated by a customer than your text analytics will be able to provide insights into what is driving actual customer spending.  We have a case study example on (here is the link: ).  The case goes into detail on how Jiffy Lube was able to perform such an analysis in order to better understand which customer comment topics were driving revenue.  


When performing text analytics it is important to know which KPIs are available for you to include in your analysis.  


  1. Sentiment Analysis - 
  2. Mining the Web for Feelings, Not Facts - 

Do you clean up your digital garbage?


Taking out the trash is a chore.  In our homes, we take out the garbage to keep a tidy home.  What about our digital garbage.  As you can see from the snippet, we are creating 2.5 billion gigabytes of data every day.  While this sounds like an amazing amount of information, I am wondering how much of it is crap.  When was the last time you cleaned up an old digital account?  Do you close your online account and delete the data or do you just move on?  When you no longer use an online account and then don't close the account or communicate to the vendor they are stuck with your digital trash.   

I got an email today from Optimum informing me of a change to their email system.  

Starting May 13, 2016, if you haven’t accessed your Optimum Online email account for 90 days, your email account and its contents will be permanently deleted and cannot be recovered. This change only impacts your Optimum Online email account. Your access to using your Optimum ID will not be affected.
— Optimum Customer Service Email

When you get cable internet or heck internet service from anyone these days they all offer a bunch of FREE email accounts.  The account was setup automatically and there was no way to deny it.  Now Cablevision/Optimum is stuck with all these user accounts and undoubtedly lots and lots of junk mail and spam that is filling up the unused accounts.  Bravo to Optimum for taking a step to clean up some of the digital garbage.  Can you think of one or two accounts that you don't use anymore?  Why not take a few minutes to go and delete those old accounts? 

Is your content strategy inaccessible?

Do you create hurdles for readers to jump through?

Is your content strategy inaccessible?  I don't know what some brands are thinking.  They post all over the social-sphere (Twitter, Facebook, LinkedIn) and then they pull the trigger on sponsored posts which simply means that they are paying to develop the content, then they are paying someone to blast post it, then they are paying the networks to extend the reach.  All of this is to do what?  Get me to click right?  Go and read what they have to say.  BUT when I do click (which they pay for), I am not taken to the content, I am taken to a SIGNUP page.  If I want to read their post I have to become a member.  

Lopsided Relationship from the start?

I have nothing against becoming a member, signing up for your updates or something, but this is the start of a relationship and you want my Name, Address, City, State, Zip, County, Email, Company, Title, Number of Employees before I can see if I even see this one post?!  Before I even know if I care what you have written.  That is very lopsided, I give you all my details, become a member of your list, have to fill out more forms if I ever want to get off your list and what do I get?  I get to read your post.

Do your tactics help or harm your strategy?

It is extremely important, in all areas, but specifically in the content space to test out the user experience.  If you tweet something, go to twitter click your own link and see what happens.  Then ask yourself, what would I do/provide/signup for to read this?   

Tim Cook says he will not combine a MacBook and iPad, but they already have

iPad Pro with Logitech case on LEFT and MacBook Pro on the RIGHT

iPad Pro with Logitech case on LEFT and MacBook Pro on the RIGHT

The Verge has a report that Tim Cook has stated that Apple will not combine a MacBook and the iPad.  Here is a link to the Verge article.  

When traveling:  The iPad Pro for me is a portable replacement.  I am very happy to travel with one charging cable (lightning) that charges both my iPhone and iPad.

All about the Apps:  The iPad Pro functionality has mostly to do with the power of the apps that you need and the apps that are available.  For me I need at the ready; Word, Excel, Powerpoint, Keynote, Canva, Email, Slack, Twitter, Facebook, Linkedin, Chrome, iMessages and Skype. The iPad has great versions of all these apps.  

All about the FOCUS:  For me, when I am working on my MacBook, I notice that I have tons of apps, windows, chats all open at one time.  I have an email 1/2 done, a word doc 1/2 done and then a notification comes in and I am using gestures to swtich between things.  I have only been using the iPad Pro for two days, but I find it very comfortable using iOS, by having all the apps in full screen I am able to focus on the task at hand. 



When your product does not evolve

With all of the changes going on in the software world sometimes, your product can't evolve.  That is what happened to our friends over at screenr.  I was notified today at as of November 11th screenr would be shutting down.  As you can read in the email below, their technology relied on Java RE and in their own words is antiquated. Reliance on a third party technology for the basis of your solution is fundamentally risky.  Don't get me wrong there are tons of advantages to using a popular technology base.  In many cases you can get your solution built faster and you are able to reach a wider audience.  

Evolve or Die

In order to keep your product current it is important to always have a plan B and plan C ready for shifts in technology and consumer behaviors.  I will miss screenr and wish them well in the future.  Sometimes things just have to go away.  Unlike the walking dead technologies that are still holding onto FLASH. 

Having Employees write reviews can be costly, to the tune of $1.25 million in Canada

Reviews are great for your site or your app and everyone wants to get good reviews.  Some brands are tempted to have employees write reviews as a way to get good reviews fast.  We have two examples of the problem this month.  The first news to break was from Amazon's effort to curb the practice by suing people who offer to write fake reviews.  The second is from Canada that has fined Bell Canada for having their employees write fake reviews.  

Amazon sues 1,000 people for offering to write fake reviews.  This article on the Huffington Post provides a great overview of what Amazon's position is. "Amazon's legal counteroffensive, however, appears to be one of the most aggressive attempts yet by a major U.S. e-commerce company to fight back. Its lawsuit alleges that individuals would write five-star reviews about products they never even tried, and plotted with product makers to subvert Amazon safeguards that are meant to bolster confidence in the website's reviews."

Here is the LINK.   

Bell hit with $1.25M fine for planting 4-star reviews for phone apps

Bell hit with $1.25M fine for planting 4-star reviews for phone apps has the report about Bell Canada and the $1.25 million dollar fine for having employees write fake reviews for their mobile application. "The Bell and Virgin mobile apps were launched in November 2014 and immediately garnered four-star reviews on Apple's iTunes App Store and Google Play Store. At the time, CBC News reported that possibly half a dozen or more of the rave write-ups were penned by Bell Canada employees – many in senior positions. None of the workers disclosed that they worked for Bell."  (LINK TO ARTICLE)  

Honest reviews are powerful for your brand.  It is the result from creating a great experience for your customer.  Focus on delivering awesome and good reviews will come.  

"When information is cheap, attention becomes expensive"

Great video from @jasonsilva #Marketing #ux

"When information is cheap, attention becomes expensive" - James Gleick "Cinema is cartography for the mind" (my favorite attention technology) (and we should be discerning in what we allow to colonize our attention)

Posted by Jason Silva on Sunday, September 20, 2015

Creepy Email Marketing

creepy1When you are communicating to your customers I would advise to avoid the ‘creepy triggers’.   All marketers love the LTV metric and I need not go into all the ways it is better to keep a customer than have to gain a new one.  

I use many online SaaS (software as a service) tools to do my job.  Some I love, some I don’t.  They all reach out with emails about how to best use their tools/services.  Some focus on case examples of other customers hoping it will inspire me to use their tool more.  Some focus on the company’s philanthropic efforts so I feel that by being a customer I am also making the world a better place.  The past two examples are what I call 2015 brand normal.  There isn't anything in their tactic that is bad or pushing the envelope or trying to be edgy.  They are using current norms to communicate their value to customers.   The creepy one came in this week. I get an email from a creative service I use.  In the email the person says “I noticed you recently created a design in XXXX but never got a chance to share/download it. We are wondering why?”.  I understand that when I am using an online service the company that is hosting it can tell what I have been doing.  But we don't talk about it.  It is that big brother, matrix style observation we both know is going on.  It is like an unspoken do not ask, do not tell.  The service is one I use to create many concepts and drafts before I complete one and download it - so the one in question was nothing out my normal pattern of usage so why the email?  What purpose did it serve their brand?  I am already a customer and have been using the tool weekly for over a year.  As Scott Stratten said in one of his earlier talks, “There is no such thing as a neutral brand interaction. You either improve or hurt it.”  This was definitely a hurt moment.  Will it change my recommendation of the tool to friends?  No, but friends that work in sensitive industries (banking, pharma) - yes it will.