Hear it with your eyes

Hear it with your eyesHear it with your eyesHear it with your eyes

Opening doors to people with extraordinary talents

Hearing AI
Some members of Microsoft's Hearing AI team. L-R: Swetha Machanavajhala, Elizabeth Halper, Anirudh Koul, and Michael Johnston. Photo by Tanvi Surti

For more than a week, the carbon monoxide alarm in Swetha Machanavajhala’s home was loudly warning that she was in a toxic environment. But until a neighbor alerted her, the software engineer had no idea of the danger she was in. Machanavajhala is hard of hearing, and years later her experience inspired her to seek a way for the 360 million people worldwide who have some form of deafness to “see” sound. The result is the Hearing AI app that gives visual cues to environmental noise levels and sounds, and conveys not just what others say, but how the words were expressed.

As a Microsoft employee, Machanavajhala is provided with the tools she needs to do her best work. She is also encouraged to create new products and refine existing ones to make them accessible to everyone. That’s not so surprising; after all, Microsoft’s mission is to empower every person and every organization on the planet to achieve more.

From prototype to product
Machanavajhala works with the Azure network monitoring team, which has a large presence in Silicon Valley. Since 2015, though, she’s had the chance to expand the skills and experience she brings to her “day job” by working on a personal project that she’s passionate about.

“I wanted to do something that benefits people who are Deaf or hard of hearing,” she explains. “With that in mind, I set out to take a step forward to develop something and put it in the customers’ hands. I led a team of 16 people in the 2015 Microsoft worldwide hackathon and we won the second place under the Tech for Good category.” That success in a competition involving 15,000 Microsoft employees motivated her to go forward.

The journey continued as she worked on the project with a team of students at University College, London, then with college interns in Vancouver, Canada, who were working in the Microsoft Garage program to develop experimental products. The Hearing AI app is the result and is currently available to users in the United States and Canada via the Microsoft Garage website. A team of regular Microsoft employees now work in their spare time to refine the app and add features. It will be available via the app store in the future.

The Hearing AI app gives Deaf or hard of hearing users important information about their auditory environments.

No barriers to productivity
Machanavajhala was raised in Chennai, India, where she learned to communicate by lip-reading and speaking. But when she entered the workforce, large group meetings and telephone calls presented a challenge. As a Microsoft employee, she’s now able to work effectively with her colleagues in the office, in Vancouver, London, or anywhere else, because the company provides a captioning service that’s available on-site or remotely. What others say is then translated into text for Machanavajhala to read, so she can then respond appropriately. Sign language interpreters are available at Microsoft offices to support employees with deafness who prefer that mode of communication.

Microsoft’s leader on disability matters is Jenny Lay-Flurrie, the company’s chief accessibility officer. Describing Lay-Flurrie as her role model, Machanavajhala relates that “She’s Deaf, and she’s the main inspiration for all of us. She has done a lot of work on the accessibility area of Microsoft and she has brought Microsoft to a level where people recognize around the world so we are following her steps.” In fact, Lay-Flurrie has been recognized by the White House as a champion of change for disability employment.

When Lay-Flurrie joined Microsoft, she didn’t disclose her disability to anyone during the hiring process for a technical role, relying on her excellent lip reading and speech to get by. Though she performed well in her job, the energy and hours required to compensate for her inability to hear became exhausting, so she explained the situation to her management and received support. Lay-Flurrie eventually transitioned into a disability-focused role for the company, advocating for customers and employees.


“We’ve got a ridiculous amount of talent that’s left out there in the workplace, but not in the workforce.”

— Jenny Lay-Flurrie, Chief Accessibility Officer Microsoft

Freeing potential in the workplace and beyond
“I live with my disability; I don’t suffer it. I actually think that it’s giving me a lot of strength,” asserted Lay-Flurrie, while speaking at the Code for America Summit in Oakland. “But if you look at the unemployment statistics for disability they’re somewhat criminal, in my view. They’re double the rate of those without disabilities and it means that we’ve got a ridiculous amount of talent that’s left out there in the workplace, but not in the workforce.”

During Lay-Flurrie’s tenure, the move toward a fully inclusive workplace has accelerated. One of the major programs to shift the internal paradigm about disability hiring led to the successful employment of autistic individuals. Lay-Flurrie was shocked at the response to their announcement of the program, which resulted in receiving nearly a thousand resumes. “These are resumes that can often lead with, ‘I work three hours a week at Safeway,’ and then at the bottom of them you’ve got a master’s or PhD or an engineering degree!” Autistic applicants now experience a totally redesigned hiring process that uses two- to four-week “academies” rather than standard interviews to evaluate candidates. When applicants become employees, they receive individualized coaching and support both inside and outside the office.

The Hearing AI app was developed as part of Microsoft's annual worldwide hackathon for employees, which receives many submissions in the "Tech for Good” category.”

“There are 1.2 billion people in the world with disabilities,” says Lay-Flurrie. “These are our employees, our customers, our friends, our family. We need to empower people with disabilities to come to work as they are. Everything of who they are. To go for it!”

The drive to make Microsoft’s products more accessible benefits greatly from the presence of employees who can fully understand the needs of customers. Machanavajhala is one of many who are a source of ideas for improvements to existing products as well as proposing new ones. “I want to be able to create a benefit for everyone, and help remove all the stigmas about people who are Deaf or hard of hearing to show we can do anything that others can do,” she says.

You can learn much more at the Microsoft Accessibility website.