Carmen R. Miller
Thoughtful Product Design
← Back
Summary
THE COMPANY
The Adventure Challenge (TAC) is a scratch-off adventure company dedicated to bringing people closer through playfulness and authenticity, utilizing mystery "challenges" to complete together.
THE GOAL Build a strong user testing process in this scrappy young startup.
**CONTENT NOTE: some of the testing examples in this case study are from products that address sexual and emotional intimacy and so contains mature references.
Product Needs
This product is unique on the market—scratch-off adventure products to help you connect more deeply with the people in your life through shared experiences—therefore the challenges in user testing were equally unique:
- Product Education for Market TestersCreate a clear enough idea of the concept to equip the survey tester to give effective feedback, without prior brand knowledge.
- Consistent Reporting Across Products Ensure our methods were consistent enough to "compare apples to apples" and create solid testing precedent across full suite of very different products (for couples, kids, friends, and families).
- Actioning the DataHow do we apply feedback on a product that is meant to push people out of their comfort zones and therefore creates polarity within survey results? how do we stay responsive to the user's voice without losing the heart of the product or creating something "vanilla" instead of daring and adventurous?
What We Built
1.Bookended Testing Process
Conceptual > Physical > Conceptual
Steps 1 and 2 below already existed in rough form, but when taking on this role, I advocated for adding Step 3 to measure our success in development.
We needed a way to ensure that the customer feedback we were gathering during testing was being interpreted and implemented successfully.
The Improved Development Flow:
- Conceptual TestDetermines if the raw concept has potential to move forward. Feedback is analyzed by me and implemented at the stakeholders' discretion.
- Physical ‘Usability’ TestGives practical feedback from our internal testers on how the challenge works in the real world. Feedback is used to refine the challenge further and those that are approved go into the final copy process.
- Conceptual Test *NEW*The final score on the fully developed challenge with final copy. Success was achieved when our final conceptual score was significantly higher than our first conceptual score! It means we listened to our users and implemented feedback effectively.
SEE EXAMPLE of beginning and final scores from our "Quickies" product (a set of scratch-off cards with short challenges that facilitate sexual and emotional intimacy for couples).
NOTE ABOUT SCORES: Physical scores are almost always higher than Conceptual scores because the testing pool is made up of existing customers who already love the TAC brand. Conceptual scores are pulled from broad market surveys of people who may or may not be familiar with The Adventure Challenge brand.
2.
Improved Survey Language
Crafting questions to create more actionable data and better insights
This was low hanging fruit! The existing language wasn't wrong per se, but needed consistency across tests, and vigilance to keep language neutral and avoid bias or leading questions.
3.Dynamic Reporting Sheet
One source of truth for all testing to be shared and compared
My own exhaustive "working sheet" of testing results was consolidated into a reporting "master sheet" that could be easily referenced across the company for helpful insights to inform Product, Marketing, Retail, Customer Service, etc..
Columns were sortable to be able to compare scores between individual challenges and from product to product, and useful quotes and testimonials were captured for the Marketing and Creative teams.
4.
Expanded Customer Testing Base(‘The Adventure Lab’)Real world feedback for us, and perks for them
I managed communications, perks, and recruitment of this quality customer testing base. Our ‘Adventure Lab’ was full of enthusiastic product fans who tried out our challenge concepts in the real world, and I sought to improve their experience as much as possible through honing email copy, creating warm interactions, and thoughtful perks.
We started a recruitment campaign from our existing U.S. customer email list, and quickly increased membership from 3,900 to 7,100 participants within a few months.
5.Customer Advisory Board
Our most-trusted "hero" customers
Under the direction of our CMO, I reached out and interviewed customers to select a well-crafted and diverse VIP Customer Advisory Board to give real time, "customer voice" feedback on Marketing and Product decisions in exchange for compensation and perks.
Considered in selection:
- Customer story
- Experience level with product
- Diversity of age, gender, ethnicity, and relationship type
- Ability to articulate their customer story and feedback
- Openness to share
- Availability and willingness to participate
Planned Next Steps
With more time in this role, I would have continued to grow TAC product testing in the following ways:
- Complete library of testing scores and data with key insightsWith a large backlog of products previously released without customer testing, I was continuously working backwards to gather feedback and gain insights about past products to inform upcoming products. With more time I would have completed the full catalog of products.
- More dynamic experience for Adventure Lab membersBetter designed emails with cleverly written copy, more creative perks and merch, exclusive social media groups, etc..
- International testing in the product development stageThis was still a bit of an afterthought when I came onto the testing team, but with distribution in Europe, UK, Canada, Australia, New Zealand, and plans for expansion into Mexico and additional countries, a more nuanced cultural context was needed for product creation going forward. This would have expanded the scope and capacity of my role and most likely would have required building a full testing team.
Key Takeaways from this Role
- Everyone wins when the user’s voice is heardIn product design and development, in order to create something that truly meets a need, the voice of the user/customer must be central. Make sure someone is their advocate.
- Data ≠ MeaningThe meaning of raw data is not always self-evident and can be easily affected by personal bias within a team. Both embracing empathy and seeking objectivity are important when interpreting feedback and creating action from information. This is how we create a well-rounded and well-crafted product.
- Stay curious and openLearning to ask the right questions is a superpower—with customers, with stakeholders, and with a team. Collaboration, humility, and ownership go a long way in making our work better and more meaningful.
Carmen R. Miller
Thoughtful Product Design
← Back
Summary
THE COMPANY
The Adventure Challenge (TAC) is a scratch-off adventure company dedicated to bringing people closer through playfulness and authenticity, utilizing mystery "challenges" to complete together.
THE GOAL Build a strong user testing process in this scrappy young startup.
**CONTENT NOTE: some of the testing examples in this case study are from products that address sexual and emotional intimacy and so contains mature references.
Product Needs
This product is unique on the market—scratch-off adventure products to help you connect more deeply with the people in your life through shared experiences—therefore the challenges in user testing were equally unique:
- Product Education for Market TestersCreate a clear enough idea of the concept to equip the survey tester to give effective feedback, without prior brand knowledge.
- Consistent Reporting Across Products Ensure our methods were consistent enough to "compare apples to apples" and create solid testing precedent across full suite of very different products (for couples, kids, friends, and families).
- Actioning the DataHow do we apply feedback on a product that is meant to push people out of their comfort zones and therefore creates polarity within survey results? how do we stay responsive to the user's voice without losing the heart of the product or creating something "vanilla" instead of daring and adventurous?
What We Built
1.Bookended Testing Process
Conceptual > Physical > Conceptual
Steps 1 and 2 below already existed in rough form, but when taking on this role, I advocated for adding Step 3 to measure our success in development.
We needed a way to ensure that the customer feedback we were gathering during testing was being interpreted and implemented successfully.
The Improved Development Flow:
- Conceptual TestDetermines if the raw concept has potential to move forward. Feedback is analyzed by me and implemented at the stakeholders' discretion.
- Physical ‘Usability’ TestGives practical feedback from our internal testers on how the challenge works in the real world. Feedback is used to refine the challenge further and those that are approved go into the final copy process.
- Conceptual Test *NEW*The final score on the fully developed challenge with final copy. Success was achieved when our final conceptual score was significantly higher than our first conceptual score! It means we listened to our users and implemented feedback effectively.
SEE EXAMPLE of beginning and final scores from our "Quickies" product (a set of scratch-off cards with short challenges that facilitate sexual and emotional intimacy for couples).
NOTE ABOUT SCORES: Physical scores are almost always higher than Conceptual scores because the testing pool is made up of existing customers who already love the TAC brand. Conceptual scores are pulled from broad market surveys of people who may or may not be familiar with The Adventure Challenge brand.
2.
Improved Survey Language
Crafting questions to create more actionable data and better insights
This was low hanging fruit! The existing language wasn't wrong per se, but needed consistency across tests, and vigilance to keep language neutral and avoid bias or leading questions.
3.Dynamic Reporting Sheet
One source of truth for all testing to be shared and compared
My own exhaustive "working sheet" of testing results was consolidated into a reporting "master sheet" that could be easily referenced across the company for helpful insights to inform Product, Marketing, Retail, Customer Service, etc..
Columns were sortable to be able to compare scores between individual challenges and from product to product, and useful quotes and testimonials were captured for the Marketing and Creative teams.
4.
Expanded Customer Testing Base(‘The Adventure Lab’)Real world feedback for us, and perks for them
I managed communications, perks, and recruitment of this quality customer testing base. Our ‘Adventure Lab’ was full of enthusiastic product fans who tried out our challenge concepts in the real world, and I sought to improve their experience as much as possible through honing email copy, creating warm interactions, and thoughtful perks.
We started a recruitment campaign from our existing U.S. customer email list, and quickly increased membership from 3,900 to 7,100 participants within a few months.
5.Customer Advisory Board
Our most-trusted "hero" customers
Under the direction of our CMO, I reached out and interviewed customers to select a well-crafted and diverse VIP Customer Advisory Board to give real time, "customer voice" feedback on Marketing and Product decisions in exchange for compensation and perks.
Considered in selection:
- Customer story
- Experience level with product
- Diversity of age, gender, ethnicity, and relationship type
- Ability to articulate their customer story and feedback
- Openness to share
- Availability and willingness to participate
Planned Next Steps
With more time in this role, I would have continued to grow TAC product testing in the following ways:
- Complete library of testing scores and data with key insightsWith a large backlog of products previously released without customer testing, I was continuously working backwards to gather feedback and gain insights about past products to inform upcoming products. With more time I would have completed the full catalog of products.
- More dynamic experience for Adventure Lab membersBetter designed emails with cleverly written copy, more creative perks and merch, exclusive social media groups, etc..
- International testing in the product development stageThis was still a bit of an afterthought when I came onto the testing team, but with distribution in Europe, UK, Canada, Australia, New Zealand, and plans for expansion into Mexico and additional countries, a more nuanced cultural context was needed for product creation going forward. This would have expanded the scope and capacity of my role and most likely would have required building a full testing team.
Key Takeaways from this Role
- Everyone wins when the user’s voice is heardIn product design and development, in order to create something that truly meets a need, the voice of the user/customer must be central. Make sure someone is their advocate.
- Data ≠ MeaningThe meaning of raw data is not always self-evident and can be easily affected by personal bias within a team. Both embracing empathy and seeking objectivity are important when interpreting feedback and creating action from information. This is how we create a well-rounded and well-crafted product.
- Stay curious and openLearning to ask the right questions is a superpower—with customers, with stakeholders, and with a team. Collaboration, humility, and ownership go a long way in making our work better and more meaningful.
Carmen R. Miller
Thoughtful Product Design
← Back
Summary
THE COMPANY
The Adventure Challenge (TAC) is a scratch-off adventure company dedicated to bringing people closer through playfulness and authenticity, utilizing mystery "challenges" to complete together.
THE GOAL Build a strong user testing process in this scrappy young startup.
**CONTENT NOTE: some of the testing examples in this case study are from products that address sexual and emotional intimacy and so contains mature references.
Product Needs
This product is unique on the market—scratch-off adventure products to help you connect more deeply with the people in your life through shared experiences—therefore the challenges in user testing were equally unique:
- Product Education for Market TestersCreate a clear enough idea of the concept to equip the survey tester to give effective feedback, without prior brand knowledge.
- Consistent Reporting Across Products Ensure our methods were consistent enough to "compare apples to apples" and create solid testing precedent across full suite of very different products (for couples, kids, friends, and families).
- Actioning the DataHow do we apply feedback on a product that is meant to push people out of their comfort zones and therefore creates polarity within survey results? how do we stay responsive to the user's voice without losing the heart of the product or creating something "vanilla" instead of daring and adventurous?
What We Built
1.Bookended Testing Process
Conceptual > Physical > Conceptual
Steps 1 and 2 below already existed in rough form, but when taking on this role, I advocated for adding Step 3 to measure our success in development.
We needed a way to ensure that the customer feedback we were gathering during testing was being interpreted and implemented successfully.
The Improved Development Flow:
- Conceptual TestDetermines if the raw concept has potential to move forward. Feedback is analyzed by me and implemented at the stakeholders' discretion.
- Physical ‘Usability’ TestGives practical feedback from our internal testers on how the challenge works in the real world. Feedback is used to refine the challenge further and those that are approved go into the final copy process.
- Conceptual Test *NEW*The final score on the fully developed challenge with final copy. Success was achieved when our final conceptual score was significantly higher than our first conceptual score! It means we listened to our users and implemented feedback effectively.
SEE EXAMPLE of beginning and final scores from our "Quickies" product (a set of scratch-off cards with short challenges that facilitate sexual and emotional intimacy for couples).
NOTE ABOUT SCORES: Physical scores are almost always higher than Conceptual scores because the testing pool is made up of existing customers who already love the TAC brand. Conceptual scores are pulled from broad market surveys of people who may or may not be familiar with The Adventure Challenge brand.
2.
Improved Survey Language
Crafting questions to create more actionable data and better insights
This was low hanging fruit! The existing language wasn't wrong per se, but needed consistency across tests, and vigilance to keep language neutral and avoid bias or leading questions.
3.Dynamic Reporting Sheet
One source of truth for all testing to be shared and compared
My own exhaustive "working sheet" of testing results was consolidated into a reporting "master sheet" that could be easily referenced across the company for helpful insights to inform Product, Marketing, Retail, Customer Service, etc..
Columns were sortable to be able to compare scores between individual challenges and from product to product, and useful quotes and testimonials were captured for the Marketing and Creative teams.
4.
Expanded Customer Testing Base(‘The Adventure Lab’)Real world feedback for us, and perks for them
I managed communications, perks, and recruitment of this quality customer testing base. Our ‘Adventure Lab’ was full of enthusiastic product fans who tried out our challenge concepts in the real world, and I sought to improve their experience as much as possible through honing email copy, creating warm interactions, and thoughtful perks.
We started a recruitment campaign from our existing U.S. customer email list, and quickly increased membership from 3,900 to 7,100 participants within a few months.
5.Customer Advisory Board
Our most-trusted "hero" customers
Under the direction of our CMO, I reached out and interviewed customers to select a well-crafted and diverse VIP Customer Advisory Board to give real time, "customer voice" feedback on Marketing and Product decisions in exchange for compensation and perks.
Considered in selection:
- Customer story
- Experience level with product
- Diversity of age, gender, ethnicity, and relationship type
- Ability to articulate their customer story and feedback
- Openness to share
- Availability and willingness to participate
Planned Next Steps
With more time in this role, I would have continued to grow TAC product testing in the following ways:
- Complete library of testing scores and data with key insightsWith a large backlog of products previously released without customer testing, I was continuously working backwards to gather feedback and gain insights about past products to inform upcoming products. With more time I would have completed the full catalog of products.
- More dynamic experience for Adventure Lab membersBetter designed emails with cleverly written copy, more creative perks and merch, exclusive social media groups, etc..
- International testing in the product development stageThis was still a bit of an afterthought when I came onto the testing team, but with distribution in Europe, UK, Canada, Australia, New Zealand, and plans for expansion into Mexico and additional countries, a more nuanced cultural context was needed for product creation going forward. This would have expanded the scope and capacity of my role and most likely would have required building a full testing team.
Key Takeaways from this Role
- Everyone wins when the user’s voice is heardIn product design and development, in order to create something that truly meets a need, the voice of the user/customer must be central. Make sure someone is their advocate.
- Data ≠ MeaningThe meaning of raw data is not always self-evident and can be easily affected by personal bias within a team. Both embracing empathy and seeking objectivity are important when interpreting feedback and creating action from information. This is how we create a well-rounded and well-crafted product.
- Stay curious and openLearning to ask the right questions is a superpower—with customers, with stakeholders, and with a team. Collaboration, humility, and ownership go a long way in making our work better and more meaningful.