We test and score robot vacuums using a battery of 5 data-driven tests. Our tests are a mix of objective testing and subjective assessment. This testing process applies to only robot vacuums.
See here for how we test cordless stick vacuums and see here for how we test all other vacuums.
Our 5 testing and scoring categories include:
In addition, we also test the following non-scoring categories.
- Noise
- Size & Dimensions
Cleaning Performance Test
Scoring Weight – 60%
Our cleaning performance score is based on a series of 5 or 6 objective cleaning tests. Some robots come with the ability to mop in addition to vacuum. We include this factor in the cleaning performance category and adjust the factor weights accordingly.
For robots that can only vacuum the 5 tests include:
For robots that vacuum and mop the 6 tests include:
Debris Test
Scoring Weight – 55% (vacuum only) & 45% (vacuum & mop)
Our debris test is an objective cleaning test. We complete a total of 12 debris tests using 3 different floor types (hardwood, low pile carpet, and high pile carpet) and 4 different debris types (cereal, rice, kitty litter, and sugar).
We measure 0.5 oz. of each debris and spread it out evenly on our testing lane. We let the robot vacuum complete 2 full cleaning cycles on the highest power.
We measure the amount of debris left on the testing lane to determine the cleaning performance for each floor type.
The debris test score is determined by how much debris by weight the robot vacuum was able to remove on each floor type.
Long Hair Test
Score Weight – 10% (vacuum only) & 10% (vacuum & mop)
Our long hair test is an objective cleaning test. We place 1.0 grams of long synthetic hair onto our high pile carpet testing lane. We let the robot vacuum complete 2 cleaning cycles on the highest power.
We measure the amount of long hair remaining on the hardwood floor and measure the amount of hair that got tangled in the brushroll and/or spinning side brushes. We compare these amounts to the starting amounts to determine performance.
This test examines how well the robot vacuum can remove long hair without the hair tangling around the brushroll and/or spinning side brush.
Pet Hair Test
Score Weight – 25% (vacuum only) & 15% (vacuum & mop)
Our pet hair test is an objective cleaning test. We place 1.0 grams of dog hair and embed it into the high pile carpet test lane. We let the robot vacuum complete 2 cleaning cycles on the highest power.
We measure the amount of remaining pet hair in the carpet fibers and measure any hair that tangles in the brushroll or spinning side brush. We compare these amounts to the starting amounts to determine performance.
This test examines how well the robot vacuum can remove the pet hair.
Edge Test
Score Weight – 5% (vacuum only) & 5% (vacuum & mop)
Our edge test is an objective cleaning test. We place 0.3 oz of sugar, 0.1 oz of kitty litter, and 0.1 oz of rice directly along the edge of the wall, extending out 4.0″ from the wall. We run the vacuum towards the wall on the highest power mode, cleaning as close to the wall as possible with a single forward and backward pass.
We measure the distance from the wall the robot vacuum was able to clean and measure, by weight, how much of the debris remained after cleaning.
This test examines how well the robot vacuum can remove debris when in close proximity to the wall / edge / room perimeter.
Crevice Test
Score Weight – 5% (vacuum only) & 5% (vacuum & mop)
Our crevice test is an objective cleaning test. We fill 3 crevices on tile flooring with 0.3 oz of sugar. We let the robot vacuum complete 1 cleaning cycle on the highest power and/or ideal cleaning mode.
We measure the amount of debris remaining in the crevices by weight and compare it to the starting amounts to determine performance.
This test examines how well the robot vacuum can remove debris from crevices / gaps / cracks.
Mop Test
Score Weight – 20%
Our wet cleaning test is a subjective cleaning test. We fully clean our ~1,000 sq. ft. office and studio space. After cleaning we assess the mopping pads and/or empty the dirty water tank and subjectively observe how much dirt and debris was removed.
This test examines how well the robot vacuums are able to remove dirt and debris on a broader scale while mopping.
Navigation Test
Score Weight – 20%
Our navigation test is based on a series of 3 objective tests. The 3 tests are weighted based on their relative importance.
These 3 factors include:
Cleaning Speed
Score Weight – 10%
Our cleaning speed test is an objective test. We let the robot vacuum perform multiple cleaning cycles in our ~1,000 sq. ft. office and studio space. We record the time it takes for the robot to complete these cycles and calculate the average time it takes to clean the floor.
This test examines how fast a robot can clean a ~1,000 sq. ft. space. Robots that can clean faster will score higher.
Obstacle Avoidance
Score Weight – 40%
Our obstacle avoidance test is an objective test. We set up 4 obstacles (mug, shoe, cord, fake pet waste) in an area for the robot to clean. We let the robot clean the floor and observe the process. We record any time the robot fails to avoid an object.
The obstacle avoidance performance of the robot vacuum can fall into 1 of 3 tiers for each object:
- Fully Avoided
- Partially Avoided
- Not Avoided
This test examines how well the robot is able to recognize and avoid obstacles while cleaning.
Cleaning Efficiency
Score Weight – 50%
Our cleaning efficiency test is an objective test. We let the robot perform multiple cleaning cycles in our ~1,000 sq. ft. office and studio space.
During these tests we keep track of 3 factors with equal scoring weights:
- Can it fully clean the space?
- Can it return to the docking station?
- Can it not get stuck?
This test examines how efficiently a robot can navigate a space.
Build Quality Test
Score Weight – 5%
Our build quality test is a subjective test. We examine and assess the major material and design components of the robot vacuum, rating the build quality of each element.
Specifically, we are looking for build and design issues that are more likely to break and/or be difficult to use. Our assessment follows our standard 7 level tier system (exceptional, excellent, very good, good, fair, poor, and very poor).
Usability Test
Score Weight – 10%
Our usability score is based on 3 factors. The 3 tests are weighted based on their relative importance.
The 3 testing and scoring factors include:
In addition, we also record the specific parts and accessories included within the box. This is a non-scoring factor.
Setup
Score Weight – 20%
Our setup test is a subjective assessment. We set up the device, recording the length of time required to set up the device as well as the difficulty of the setup process. Our assessment follows our standard 7 level tier system (exceptional, excellent, very good, good, fair, poor, and very poor).
App
Score Weight – 75%
Our app test is a subjective assessment. We connect the robot to the app and take note of the features available. We examine the experience of using the app in relation to how the app improves the cleaning experience. Our assessment follows our standard 7 level tier system (exceptional, excellent, very good, good, fair, poor, and very poor).
Controls & Display
Score Weight – 5%
Our controls & display test is a subjective assessment. We examine and use the controls and display of the robot vacuum, including any on the dock, taking notes of how they are to use and how they impact the cleaning experience. Our assessment follows our standard 7 level tier system (exceptional, excellent, very good, good, fair, poor, and very poor).
In Box
Score Weight – 0%
Our in box test is a report of what items can be found in the box when unpacking the robot vacuum. Due to the this test being an itemized report, this is not a scoring factor.
Maintenance Test
Score Weight – 5%
Our maintenance test is based on 2 factors, with 1 being an objective test and the other a subjective assessment.
The objective factor is:
- Recurring Costs
The subjective factor is:
- Maintaining Tasks
Recurring Costs
Score Weight – 50%
Our recurring costs test is an objective test. We calculate the estimated annual costs of maintaining the robot vacuum and its various parts.
Maintaining Tasks
Score Weight – 50%
Our maintaining tasks test is a subjective test. We examine the tasks required to maintain a robot vacuum, taking note of their relative difficulty and time requirements. Our assessment follows our standard 7 level tier system (exceptional, excellent, very good, good, fair, poor, and very poor).
Noise Test
Score Weight – 0%
Our noise test is an objective test. We use a sound meter to measure the level of noise generated by the robot vacuum. We keep the meter about 3 feet away from the vacuum and run the vacuum on each power mode. If applicable, we measure the noise generated while self-emptying in addition to the ambient noise levels of the room use as a comparison.
This test examines the noise levels created by the robot vacuum. Vacuums are loud devices and while noise can be important to some, it ultimately doesn’t impact the vacuum’s ability to clean. In our view, noise levels aren’t a contributing factor to a robot vacuum’s overall quality and cleaning performance.
As a result, this is not a scoring factor.
Size & Dimension Tests
Score Weight – 0%
Our size and dimension test is a report of size, weight, and dust bin capacity, and water tank capacity of the robot and docking station.
This is not a scoring factor.