Is the Amazon Echo 'always listening' and sending data to the cloud?

Question

Several news sources such as Intellihub and CEPro seem to suggest that Amazon's Echo home assistant constantly listens to conversations and sends them via the Internet to Amazon's servers. CEPro states that:

By saying a key phrase Amazon calls a “wake word” the Echo comes to life and begins listening for commands. By default, the wake word is Alexa.

If you reread that last sentence it may not make sense, especially if you are in the security field. According to Amazon, the Echo only listens for commands once it hears its wake word. How does it know when you have said the wake word if it wasn’t already listening?

Intellihub's article is similar in its sentiment:

The “Amazon Echo” device, a constantly-listening Bluetooth speaker that connects to music streaming services like Pandora and Spotify at the sound of a person’s voice, can be easily hacked and used by government agencies like the FBI to listen in on conversations.

(Note that I'm not particularly focused on exploring the hacking aspect of this question, since that would probably be too much for one question. My main focus is the always-on aspect and whether this sends data all the time.)

Neither article seems particularly keen to disclose a source for its claims, which suggests to me that they are unproven at best, or clickbait at worst.

Is the Echo always recording and sending data to the cloud, or are the above claims unsubstantiated? How does the Amazon Echo process data if it's not always sending data to servers in the cloud?

Answer 1

Yes, it's always listening. No, it's not always sending to the cloud.

Obviously the device has to be always listening to detect the wake word. However, that is done by a technology called.

1. How do Amazon Echo and Echo Dot recognize the wake word?

Amazon Echo and Echo Dot use on-device keyword spotting to detect the wake word. When these devices detect the wake word, they stream audio to the Cloud, including a fraction of a second of audio before the wake word.

(Echo FAQ)

Thus, according to Amazon, the wake word detection is on the device. This can be very easily tested by prohibiting the device from reaching the Internet in your router. The Echo/Echo Dot will still recognize the wake word, but the light ring will go red and the device tells you it has no Internet connection. So, we can very simply verify, that the wake word recognition is indeed done locally.

Only after detecting the wake word the device contacts the Alexa cloud service.

According to Amazon, the device only streams to the cloud when the light ring is blue and it doesn't listen at all when you muted it and the light ring is beaming solid red. Of course, cautious people can verify that with network tools like Wireshark to make sure it really only transmits then.

Answer 2

Is the Echo always recording and sending data to the cloud, or are the above claims unsubstantiated?

No on the sending. But yes on the recording.

I'm currently developing an Alexa hardware client for a company. The device is always listening. But you'd have to put in a wake word engine on it so that it would "wake up" from the passive listening mode and switch to active "listen for command" mode.

The firmware would only send the statements after the wake word (switch to active mode) into the cloud to process them as commands.

• As far as the device is concerned, you wouldn't want to send every statement it gets into the cloud for processing because that would consume too much bandwidth and power.
• As far as Amazon is concerned, they also wouldn't want each and every Alexa client to send everything it hears because that would take a tremendous amount of bandwidth. Furthermore, that would result to too much unrelated data that would mess with the system's continuous learning. Imagine trying to learn what your teacher is saying (valid commands) when everyone in class is speaking at the same time (every other statement that's not a command).

How the Amazon Echo does process data if it's not always sending data to servers in the cloud?

In passive mode, the device has an internal wake word engine that listens all the time for the wake word. While I was testing out Alexa in Raspberry Pi, I had to put in either the Sensory or KITT.AI engine for this purpose. In fact, when I tried out the prototype Alexa client code for my Linux machine, it had to be "push-to-talk" because there was no wake word engine.

Answer 3

By saying a key phrase Amazon calls a “wake word” the Echo comes to life and begins listening for commands. By default, the wake word is Alexa.

If you reread that last sentence it may not make sense, especially if you are in the security field. According to Amazon, the Echo only listens for commands once it hears its wake word. How does it know when you have said the wake word if it wasn’t already listening?

Echo listens actively for the keyword and takes the words spoken after keyword for NLU processing. Here is my understanding how echo achieves this neat feat.

Echo is built on Texas Instruments DM3725 Digital Media Processor.

This TI SoC has two key pieces inside, first is ARM Cortex-A8 MPU, and the second one is TMS320DM64x+ DSP. The ARM core should be running Linux and the DSP is running firmware.

When idling, the ARM core is taken to lowest possible power state and Linux is completely suspended. At this time the DSP and 64KB On-Chip RAM are active. The DSP firmware processes noise coming in from the mics and attempts to identify if a keyword (e.g., Alexa) is spoken. As soon as it identifies there's a keyword, DSP sends an interrupt to wake up the ARM core which in turn resumes Linux. But, remember, while Linux is waking up the human who said Alexa would have continued speaking (as in, “Alexa, what time is it?“). The DSP buffers the "what time is it?" part on the on chip RAM. And when Linux is resumed Linux fetches the buffered speech and uses Natural Language Processing (partly local, partly cloud) capability to understand what Human said.

As you see the design is totally created to be least power hungry one and to avoid need of including cloud for keyword detection and initial buffering. As a matter of fact keeping the ARM core at lowest powers state ensures that the silicon heats the least when idling thus in a way bringing long life to your device.

I am leaving out discussion of attempts to hack echo as the question was following:

the wake word recognition is indeed done locally.

Answer 4

Yes.

See, for instance, How private is the new Amazon Echo? (there are plenty more similar to be found with minimal effort)

Like Siri, Amazon Echo works in the “cloud,” running on Amazon Web Services. Therefore, the processing required to “understand” your command isn’t handled on the device itself

However,

After all, it’s always listening, so shouldn’t we be concerned about Big Brother?

No, says Amazon. The tech giant says it does not listen to or record private conversations in the home. If someone is concerned, they can use the bundled remote to press the mute button, which shuts off the “always listening” device, so it’ll be inoperable until you activate the microphone again.