Loup Ventures conducted its annual test to see how popular voice assistants perform, and Apple’s Siri came in second behind Google Assistant. Both saw improvements across the board, and for the first time they included Amazon Alexa, too.
The test includes five query categories:
- Local – Where is the nearest coffee shop?
- Commerce – Can you order me more paper towels?
- Navigation – How do I get to uptown on the bus?
- Information – Who do the Twins play tonight?
- Command – Remind me to call Steve at 2pm today.
They conducted the tests using iOS 11.4, Google Assistant on Pixel XL, Alexa via the iOS app, and Cortana via the iOS app. The smart home devices they used for the tests included the Wemo Mini Smart Plug and Dimmer Switch, TP-Link Kasa plug, and Phillips Hue Lights.
Google Assistant and Siri were the the top performers in every test, although Alexa did beat Siri in the information category. Google was the top performer in information queries, which isn’t any surprise.
Looking at performance compared to last year, Loup Ventures said,
Over a 15-month period, Google Assistant improved by 11 percentage points, Siri 13 points, and Cortana only 3 points in terms of questions answered correctly. We do not have historical data on the Alexa app. We’re impressed with the speed at which the technology is advancing. Many of the issues we had just last year have been erased by improvements to natural language processing and inter-device connectivity.
Based on the complaints people have about Siri it’s interesting that Apple’s voice assistant outperformed Amazon Alexa—a platform that’s generally perceived as better than Siri. That said, Alexa beats Siri on smart speakers because of limitations built into HomePod. Maybe next year that’ll change with the improvements we’re expecting in HomePod OS 12 and Shortcuts in iOS 12.