SonicBooth
It started with a noisy dinner.
SonicML lets the AI direct dialogue, timing, lighting, sound and physical effects rather than simply generating text.
<experience state="ready" />
Description
SonicBooth started with a simple idea: What if artificial intelligence could do more than talk to you? What if it could see you, recognize you, interact with its surroundings, and become part of a physical environment?
I wanted to explore what happens when AI moves beyond a screen and becomes something people can interact with naturally. Rather than building another chatbot, I set out to create an environment where software, electronics, and artificial intelligence work together to produce a shared, interactive experience.
SonicBooth combines local large language models, computer vision, eye tracking, voice interaction, custom electronics, sensors, lighting, audio, airflow, and displays. I've designed and assembled electronic circuits, integrated microcontrollers and embedded computers, and developed the software that coordinates everything. The challenge isn't just getting individual components to work, but making them behave as parts of a single system.
At the center are four persistent AI personalities that can interact with visitors and with one another. Each has its own personality, humor, and relationships. Their conversations and storylines can develop over time rather than starting over with every interaction. I've even experimented with LoRA training to give the personalities unusual behaviors, including teaching Grandma, one of the characters, to get distracted and forget what she was talking about.
I also created SonicML, a custom XML-based language that allows AI to control the physical environment. Instead of generating only dialogue, the AI can describe sequences of speech, lighting, sound, timing, and physical effects. These instructions are interpreted and executed by the booth's hardware controllers. I also developed a Python and LLM-based pipeline that generates C++ code for ESP32 microcontrollers, making it easier to experiment with new hardware behaviors.
SonicBooth brings together many of the things I enjoy most about engineering: designing something from scratch, experimenting with unfamiliar technologies, building custom hardware and software, and figuring out how to make systems work together in ways they weren't originally designed to.
My long-term goal is to make SonicBooth an inclusive experience where people can interact with technology without needing to understand how it works. It's an ongoing experiment, and much of the fun comes from discovering what's possible along the way.
Why?
SonicBooth started with an annoying dinner.
A large party was seated near us at a restaurant, and pretty soon we could barely hear ourselves think. I started wondering whether you could create a booth that isolated the people inside from the noise around them.
The first ideas were mostly engineering: sound-dampening materials, active noise cancellation, directional audio, and even an air-blown barrier that could help separate the space without completely enclosing it.
Then something changed the direction of the project.
A family friend suffered a severe concussion. Loud noises and bright lights became debilitating for her. Going somewhere as ordinary as a restaurant could be difficult enough that she often preferred not to go out at all.
Later, my own daughter suffered a concussion after a fall. She developed many of the same sensitivities to light and sound.
At that point SonicBooth stopped being primarily about making a restaurant quieter.
The environment shouldn't decide who gets to enjoy it.
Instead of designing an experience for the average person and then adding accessibility features, I wanted the environment itself to adapt to the person using it.
Someone sensitive to sound should be able to make it quieter. Someone sensitive to light should be able to change the brightness and color. Someone who has difficulty using conventional controls should have another way to interact.
Displays should adapt. Controls should be simple. The technology should accommodate the person rather than requiring the person to accommodate the technology.
Once the booth could control sound, light, displays and physical effects, another question followed: why couldn't all of that also become part of the entertainment?
That led to AI personalities, persistent storylines and experiences that can use the physical environment itself as part of the interaction.
What started as "this restaurant is too loud" became: can you build an environment that more people can enjoy because the environment adapts to them?
Technologies Involved
What I Learned
Building adaptability into the system from the beginning creates very different possibilities than building for an assumed average user and adding accommodations later.
Once an AI can affect lighting, sound, displays, airflow and physical devices, the model becomes part of an environment rather than simply a chatbot.
Giving an LLM access to physical devices requires a constrained layer between intent and execution. That problem eventually led to SonicML.
SonicBooth required a lot more hands-on electronics work than I'd done before. I used ChatGPT to review photos of my solder joints and critique the results as I learned.
My early attempts were getting roughly 3/10. With practice and feedback, I eventually worked my way up to 9/10, at least according to ChatGPT.
SonicBooth became considerably more interesting when the goal changed from blocking restaurant noise to adapting an entire environment around the person using it.
