Softnote: Resistive Stretch Sensing to Assist Instrument Learning for People With Visual Impairments

Collaborators: Catherine Yu, Jiaqi Liu

My contribution: sensor fabrication

CONCEPT

Advances in e-textiles and wearable technologies have enabled on-skin sensing and computing. In this project we explore gesture detection in connection to resistive stretch sensing. Specifically, our project is motivated by the piano learning experience for people with visual impairments as it is hard for them to keep track of fingerings when practicing. Beyond piano learning, our approach also fits into the larger picture of hand gesture detection as it detects finger tapping and movements.

Materials: Arduino Lilypad, conductive yarn for sensor fabrication, conductive fabric and threads for knitting and connecting components, resistors, and non-conductive yarn and fabric.

CHOICES OF YARNS

Conductive yarns vs. resistive yarns

The behavior of yarns and our access to them both influence our choices of yarns. First, we select conductive yarns instead of resistive yarns -- among the types of yarn we have access to, the latter one provides a wider range of readings, but they showed irregular changes as it is stretched, i.e. even if the yarn is at rest, the resistance changes during our experiments. On the contrary, the conductive yarn presented more consistent readings and it is sensitive enough for our purpose, even with a small range of change.

Selecting normal yarns

Besides various conductive yarns, we also experimented with different normal yarns considering stretchability and thickness for each of them. The stretchability sets the range of resistance changes, and whether the changes are enough to provide available signals. We also considered the thickness of yarns, as our toy knitting machine can only knit relatively thin yarn.

CHOICES OF FABRICATION TECHNIQUES

Knotting vs. knitting

From the end users' standpoint, knotting might have a lower barrier than knitting since the user doesn't need to learn the knitting machine. We tried different types of knots and tested different tightnesses (see the picture below). However, the overall resistance of knotting turned out to be much lower than what we wanted and burned our Lilypad :(

We then switched to knitting using the toy knitting machine shown below, and the knitted sensor showed satisfying performance. 

SIGNAL PROCESSING

CALIBRATION

We observed the the initial resistance values aren't always consistent, therefore we designed a calibration workflow before each use to increase the performance robustness across uses and users. 

L: Singals of holding three fingers; R: Signals of tapping three fingers

FUTURE WORK

Communication: To make the sensor more user friendly and practical in multiple scenarios, a bluetooth module could be added to make it wireless.
Output: The output signals can also be integrated with audio feedback, which could benefit people with visual impairment, or we could provide API interfaces for gesture triggering.
Sensor: For the sensor itself, maybe we could experiment with more yarns and find one thinner. Adjusting the knitting mechanism might help with removing the folding in our current sensors.
Signal Processing: More research could be done for handling finger dependences with software.