Making the Lego Ideas Grand Piano fully playable using a Raspberry Pi, and using it as a game controller.
This is a modification of the original set so that each individual key is recognized. This makes it fully playable, unlike the original. The feel of the keys is unchanged and it supports full 25-key rollover with realistic sounds for each key, thanks to samples that my friend Mihir provided.
Code and diagrams are included for anyone who wants to try modifying their own set. The modification was designed to be easily put together using cheap and easily available materials. If you run into issues, you can open a pull request or contact me through GitHub and I may be able to help (no guarantees).
This is documentation for what is the second major version of this project. It was designed to be far more reliable and easy to reproduce than the first. I wouldn't have recommend for most people who own the piano to try the first version of the project, but with some time, I think anyone could reproduce this one quite well.
The cool part is that the software supports full 25 key rollover, so you can press multiple keys at the same time and the sound will still be realistic.
Some keys are a little uneven since I was experimenting with switch types, but if you want to make your own, using the same switch creation procedure described further down this page should result in pretty even keys, as long as you're careful and test evenness as you go.
The code for the user interface is served by the Pi itself. It's a web UI that should be accessible over the local network using any device with a modern web browser. It's fully open source and not dependent on an app store (or even the internet!), so it doesn't carry the same risks pertaining to being discountinued, etc.
You can also use the piano as a very strange game controller.
I tested it with Batman: Arkham Knight, since the combat doesn't require many potentially uncomfortable button combos. I played a single player game because the piano didn't play well with anticheat in some multiplayer titles. The latency is not bad, and using UDP made it very easy to program. Theoretically, one could improve this by reprogramming the protocol in bluetooth, but if you're worried about latency, (I hope) you wouldn't use this in the first place.
You can play wirelessly, but it was plugged in when I recorded the video since I had been playing for a bit and needed to charge the piano.
Note that I haven't updated the code for the new switch design, but you can dig through the git history or get in touch with me if for some cursed reason you want to reproduce this.
You can find the code here.
Setup Process (do this before starting hardware work)
The setup process has been greatly simplified since the first
version. A lot of items here are not in as much detail as they
could be, since online tutorials for these steps online will be
more comprehensive, and if something doesn't work right away,
Google is your best bet anyway. For each step, you'll very likely
have to just Google how to do it, but let me know if something
looks off.
Room for improvement
The software is far from perfect, and could be improved in many
ways, most of which would offer much better responsiveness. The
reason these haven't been done is because I haven't yet been able
to justify the time commitment, but let me know if you'd be
interested.
The key idea behind the design is using copper (or aluminium) tape around the base and each key and connecting each key's tape to a GPIO output on the Raspbrry Pi to determine whether or not a key is pressed by checking whether or not the circuit was closed. This design results in keys which feel unchanged from those the original set, and hardware that can be built without needing to get a bunch of switches or printing a circuit board. In these pictures, I used a mix of aluminium and copper tape, but practically, it doesn't matter. If your tape is thin, it might tear easily - an easy fix for this is just to double up.
We start by building just the keyboard portion of the Lego set (this starts at bag 11 of the instructions) - we'll revisit the rest of the set once this is working reliably. If your piano is already built, take it apart and lift out the keyboard and power functions assemblies so that you can work on the areas in the following pictures - it's annoying, but not as tricky as it looks, based on my experience. Once that's done, we'll wrap the entire base underneath the keys in tape (or the individual plates which are then linked) - this will be our ground once we connect it to the Raspberry Pi's ground. To do this, connect it using the tape across the bottom of the base so you can connect to the Raspberry Pi ground wire on the other side (remove the Dupont connector using the tweezers, and wrap the tape around the end of the wire). We also add a wire on 25 separate GPIO pins starting from 2, and an additional wire for ground (consult a pin map of your model for this step). We haven't yet built the switches, but this is a good time to test out the software by manually closing the circuits and making sure the keys on the UI light up/produce sound.
Take apart the bottom of a key (the three pieces under the long brick)
Slightly pry open a paperclip, so that it can lay within the 3x1 brick. Seal it in with the 3x1 plate between the sides of the paperclip
Cut long strips of tape that are just a little less than the width of the thin part of each key. Lay them along the length of each key so that they lay over and secure the exposed part of the paperclip (eyeballing the length usually works, and you can fix this with more tape anyway).
Use the cone that the key rests on to poke a hole in the tape where it would normally go. Cut a thinner strip of tape along the side to reseal this, and check that the connection still works.
For the black keys, the tape should go over the edge slightly, but it can stop a bit before the edge for the white keys. The only important thing is that they connect to the base when pressed down, so check the connections.
Remove the Dupont connectors from the ends of the wires using tweezers, and bend them around the ends of each paperclip (be careful - use the tweezers for this). Secure the connection with the tape. Repeat the process with each key in order. Try not to mess up the key order, but if you do, the software can allow you to remap the keys (as long as you have connected the wires to GPIO).
Don't forget to connect the ground wire to the base
Once again, it's a good idea to check the connections before proceeding.
Now we get back to building the piano. Ignore the transparent tape on top of the keys in these pictures - these pictures were from the old switch design, but the steps from this point on are the same. We're going to build the rest of the Piano around the keys. Starting from the beginning, we omit the portion where we add the bracket for holding the power functions battery box included with the set (since we don't need it), but otherwise proceed as usual until step 76.
The next pictures show the differences in the next few steps. We use the parts from these sets to seal in the wires between the keys and the Raspberry Pi, and try to orient it so that the ports are easily accessible from the right side panel, (and/or add a battery bank here if we want it to be entirely wireless).
Make sure everything works before you seal it in. We then build the rest of the piano as usual. The only other tricky part is step 288, but isn't too bad if the keys are handled a little carefully.
Once we're done, the Raspberry Pi should be accessible from the side panel, so we can plug it in, add a power bank, etc.