Recommended Cleaner
A/B Testing
tl;dr
Problem
A previous A/B test showed that recommending certain cleaners didn’t help build host trust. Without clear reasons behind the recommendation, strong cleaners were often ignored.
Goal
Generate trust for hosts to boost conversion and empower cleaners by using data to highlight their real value in a way that feels clear and credible
Impact
How it all started
The original plan
It began as a huge idea to create a Preferred subscription program where cleaners could get benefits like special badges or first placement on searches when hosts are looking for them.
We believed this would reward high-quality cleaners, give hosts more confidence when choosing cleaner bids, and ultimately boost host conversion and commitment from cleaners.
Facing Challenges
When mapping out the program, we realized it would be quite complex and impact many areas at the same time:
Changing our focus
In one of our early presentations, our CEO flagged the scope and effort, and said:
That could change the whole project. We were about to spend months building a program without knowing if the key interaction, a host choosing a cleaner with a Preferred badge, would even move the needle.
Instead of building the full program, we decided to validate the assumption if Preferred badge would increase the hosts conversion
1st A/B Test
We ran an A/B test applying the Preferred badge to only 1 cleaner per search, putting the at the top of the list
Goals set
The results
We saw that cleaners with the badge were being picked more often, but conversion dropped by almost 1% when looking at completed projects.
Control
7.32%
Accepted bids
21.84%
Conversion
Variant
8.67%
Accepted bids
20.88%
Conversion
The Preferred badge was not being tied to actual cleaner performance for completing projects.
Learnings
We relied a lot on the algorithm to feature cleaners, and ended up hurting host trust
The best cleaner might not always be the first to bid, meaning that other cleaner with lower performance can be featured instead
We needed to put the focus on what the hosts values and matter
Hosts were not seeing the value of the badge, if it was real or perceived
Changing the strategy
Even though conversion dropped, we saw that the Preferred badge was helping hosts to make decisions and, ultimately, accept more cleaners bid.
This gave us an idea that if we could shift the way we “recommend” cleaners, we can keep the hosts' trust. We would only need to know how to show the real value of recommended cleaners
Gathering info for v2
I partnered with Data, Research, and Sales teams to learn how hosts actually make decisions when accepting bids, and define how to highlight and recommend a cleaner
Research
UX Research helped us gathered some info by running a multi-select survey to know hosts' key factors when accepting cleaners bid
Data
With the Data team, we explored what existing data could help us show cleaner value, without needing to track new events. These is what we found that could be valuable:
Now the challenge was how to bring this information into the layout in a way that felt appealing and could hosts perceive the cleaner’s value.
Benchmark
We compared how other apps were highlighting some specific elements of the UI to build trust and help user make decisions:
All of these apps support their recommendations with data and context.
2nd A/B Test
Re-imaginating the “Recommendation”
We tried a new approach of visualize cleaner’s value by introducing new data-based cards:
I led the effort to shape how these insights would appear in the product. That included:
Writing copy that was quick to understand but still credible
Creating visual treatments that blended into the UI without looking promotional
Results & Impact
The 2nd A/B test performed better in every key metric:
Control
7.32%
Accepted bids
21.52%
Conversion
Variant
11.67%
Accepted bids
26.88%
Conversion
Hosts trusted what they could understand. They felt empowered and now cleaners could be picked because they would have proven value.
Final Reflections
More Than Just Visuals
My role as a designer was about translating backend data into something users could trust. I worked across other departments to ensure the information we show felt understandable for host, and empowering for cleaners
Don't be afraid of change
We put a lot of effort preparing the Preferred Cleaner program, but sometimes others bring a fresh perspective. The sooner we find constraints, the better decisions we can make for the project!
Other Projects
Feature Onboarding
A new onboarding experience to reduce early churn, increasing by almost 9% the cleaner searches created
Quality Center
A dashboard to help hosts improve their Airbnb cleanliness rating, driving 3× more monthly integrations