STRUM: Strum Tracking via Reflected Ultrasound on Mobiles

Acoustic Sensing | On-device ML | Mobile Computing

1 minute read

Summary

I designed STRUM (Strum Tracking via Reflected Ultrasound on Mobiles) as a project for the Fall 2026 Mobile Computing and Its Applications course at Seoul National University. The proposed interface combines touchscreen chord selection with bare-hand, mid-air strumming using a single smartphone.

Status: Project proposal and system design; implementation is planned. The repository currently contains the proposal and illustration assets.

Overall workflow of STRUM: mid-air strumming, acoustic hand tracking, gesture recognition, and guitar sound output.

Overview

  • Target device: Galaxy S24+ (Android)
  • Interaction: Select chords with one hand and strum in mid-air with the other.
  • Goal: Provide a portable, two-handed guitar-playing interface without additional equipment.

Proposed Architecture

The planned pipeline has four stages:

  1. Data collection: Transmit multi-frequency acoustic signals and record reflections from the hand.
  2. Hand tracking: Use LLAP-based signal processing to estimate 2D hand trajectories.
  3. ML-based inference: Detect strums and estimate direction and relative intensity from sliding-window trajectories.
  4. Guitar sound output: Combine the recognized strumming information with the chord selected on the touchscreen.

STRUM system architecture, including acoustic sensing, hand tracking, ML inference, and the guitar audio engine.

Implementation and Evaluation Plan

The proposal targets an Android prototype with Kotlin and C++ signal processing, self-collected motion data, and lightweight on-device models. Key challenges include phone motion, reflections from the body and surroundings, and responsive sound output.

The planned evaluation compares strum/non-strum detection accuracy and model inference latency before and after optimization, including quantization and pruning. These are planned experiments, with results to be added after implementation.

Project Proposal

View the full proposal (PDF)  ·  Download PDF

First page of the STRUM project proposal.

Second page of the STRUM project proposal.

Third page of the STRUM project proposal.

Repository