Study participants wore the fitness trackers while walking or running on a treadmill and while riding an exercise bike. [Image from Stanford University]
Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn and Samsung Gear S2 were tested in a group of 60 people. The results showed that measuring heart rate had an error rate of less than 5% on 6 of the devices. Some measurements were more accurate than others while skin color and body mass index affected measurements.
The study also found that none of the devices were accurate when measuring energy expenditure. The most accurate device was incorrect by 27% on average while the least accurate was off by 93%.
“People are basing life decisions on the data provided by these devices,” said Euan Ashley, professor of cardiovascular medicine, genetics and biomedical data science at Stanford, said in a press release.
The news out of Stanford comes months after Johns Hopkins researchers found problems with data from a popular blood pressure app.
About 1 in 6 consumers currently use wearables like smartwatches and fitness trackers, according to a Nielsen article. But consumer health devices aren’t held to the same standards as medical-grade wearables – they don’t have to go through the FDA. Most devices have warnings that state they are not medical devices and are not designed to diagnose illnesses and that the accuracy is not meant to be an exact match to medical devices.
While manufacturers do try to modify their algorithms to be as accurate as possible, it’s still hard for consumers to know what information is accurate or how the devices were tested for accuracy. The researchers wanted to evaluate how accurately consumer-grade fitness devices could measure both heart rate and energy expenditure.
“For a lay user, in a non-medical setting, we want to keep that error under 10%,” said Anna Shcherbina, a graduate student on the study.
Thirty-one women and 29 men volunteered to help test the accuracy of the devices. Each volunteer wore the 7 devices while walking or running on treadmills or while using stationary bikes. Their heart rates were measured using a medical-grade electrocardiograph and the metabolic rate was measured with an instrument that measured the oxygen and carbon dioxide in their breath. The researchers then compared the results from the actual medical devices to the fitness trackers.
“The heart rate measurements performed far better than we expected,” said Ashley. “But the energy expenditure measures were way off the mark. The magnitude of just how bad they were surprised me.”
Since each device uses its own custom algorithm for calculating energy expenditure, the researchers aren’t sure why the measurements are so wrong.
“All we can do is see how the devices perform against the gold-standard clinical measures,” Shcherbina said. “My take on this is that it’s very hard to train an algorithm that would be accurate across a wide variety of people because energy expenditure is variable based on someone’s fitness level, height and weight, etc.”
On the contrary, heart rate is measured directly and doesn’t require the calculations that energy expenditure does.
The researchers hope to do a phase 2 for this study and test the accuracy of the results during outdoor exercise instead of on a treadmill.
“In phase 2, we actually want a fully portable study. So volunteers’ ECG will be portable and their energy calculation will also be done with a portable machine,” Shcherbina said.
The study was supported by Stanford’s departments of medicine, genetics and biomedical data science and was published online in the Journal of Personalized Medicine on May 24.
[Want to stay more on top of MDO content? Subscribe to our weekly e-newsletter.]
