Finding Buddy is built on peer-reviewed research into how blind and visually impaired people can find everyday objects using only sound.
253 million people live with a visual impairment, including 36 million who are fully blind. Canes, guide dogs and navigation apps help people move through the world, but none of them help someone find a specific object: a dropped key, the TV remote, or a backpack.
Say “Finding Buddy, where’s my backpack.” Speech recognition runs on the device, using VOSK.
A YOLOv8 model, trained and optimized on 100+ everyday objects, finds the object in the camera feed at about 10 frames per second.
Spoken directions update continuously until you reach the object and pick it up.
Directions in 3D, given as a clock position plus distance: “2 o’clock, 1.5 meters.” Most assistive tools only guide on a flat 2D plane, but objects can be above or below you too.
For small objects like keys, a fork or a toothbrush, Finding Buddy guides your hand instead of your body. Large objects like a chair or a car get body-level directions.
of small objects found with hand guidance (18 of 20), in 21 seconds on average.
of large objects found with body guidance (19 of 20), in 24 seconds on average.
of room scans identified every object in the room (12 of 12).
Tested with four blindfolded sighted participants, first on an iPhone and laptop, then on a portable Raspberry Pi 4 version. The full method and results are in the paper.
Developed with researchers at King Khalid University and King Fahd University of Petroleum & Minerals, where it is used as a teaching example in undergraduate courses. We are collaborating with King Faisal Specialist Hospital & Research Centre and Dr. Sulaiman Al Habib Medical Group, the two largest medical groups in Saudi Arabia.
The full Python implementation is on GitHub under the MIT License. Run it on a laptop, using your phone as the camera.