Smart devices stop eavesdropping with Kirigami

In an age where smart speakers and devices keep listening to our environment, a new technology promises to keep the benefits of audio sensing while preventing your private conversation from leaving your home. Researchers at Carnegie Mellon University have developed Kirigami, an innovative intrinsic filter that detects and removes human speech to recognize activities before processing audio data.

This breakthrough solves the growing privacy concerns: While sound data can power valuable applications such as health monitoring and environmental sensing, it can also capture sensitive personal information from our conversations.

“The data contained in the sound can help valuable applications such as activity recognition, health monitoring and even environmental sensing. However, this data can also be used to infringe on people’s privacy,” explains Sudershan Boovaraghavan, who received his PhD. From Carnegie Mellon’s Department of Software and Social Systems.

Unlike previous privacy solutions that only change audio data, Kirigami takes a more basic approach by filtering the source language. This distinction has become crucial to the rise of complex AI speech recognition systems like Openai’s Whisper, which can rebuild conversations from what was once considered safe.

“Given that these models have a lot of data, some previous techniques will leave enough residual information, small fragments, which may help restore some of the voice content,” said Yuvraj Agarwal, associate professor at Carnegie Mellon. “Kirigami can prevent these models from accessing these fragments.”

This technique is a binary classifier that determines whether speech exists in audio recording. What is particularly valuable about Kirigami is its lightweight design that can run even a small, affordable microcontroller. By processing “edge” data – directly on the device rather than on the cloud – sensitive voice never leaves the microphone.

Kirigami also offers customizable privacy settings. Users can set an aggressive threshold that takes precedence to remove all voices, but may also eliminate some non-voice sounds, or a less aggressive setting that can retain more ambient audio while accepting voice content slightly higher risk.

“Kirigami cuts most of the presentation but doesn’t have other environments for event recognition that you care about,” said Haozhe Zhou, a doctoral student who co-led the project. “You can still combine it with previous technology to give you extra privacy.”

This balance between privacy and utility is crucial for many beneficial applications of audio sensing. For example, Mayank Goel, an associate professor at Carnegie Mellon, uses audio sensing to “remind “daily tasks for people with dementia”, monitors children with attention disorders and evaluates signs of depression in students.”

“These are just examples that were done in our lab,” Goel notes. “You will find similar situations around the world where you need noninvasive data from that person’s daily life.”

The research shows a new avenue for smart device manufacturers and developers. Instead of forcing consumers to choose between beneficial technology and personal privacy, Kirigami suggests that we have both. As smart home technologies become increasingly ubiquitous, such solutions are crucial to maintaining user trust and protecting sensitive information.

The research team believes that developers can easily tweak Kirigami to meet various privacy requirements for different applications. Their work is all on ACM’s interactive, mobile, wearable and ubiquitous technology and ACM Mobicom ’24.

As our houses are filled with more listening devices, technologies like Kirigami offer hope that we can benefit from the intelligence around us without sacrificing our most personal conversations.