Warm-up (FAQ)
WHY?
To install the Mac mini group to a remote data center and be able to control each of them. By control, I mean being able to reinstall the operating system, if necessary, completely. And why is that? For example, QA connects and tries to reproduce the user’s problem and break the network connection. Or the Mac mini is part of a Ci/CD system, where the system goes into an inaccessible state in the worst case (a failed agent update, for instance).
Wake on Lan?
It will not solve the problems described above.
Physically push a button, seriously? Is there no other way?
To put it simply, no. To put the Mac mini in recovery mode, you must hold down the power button
Why Servo?
- Solenoid: high current, requires more space.
- Solder the wires directly: A lot of work, loss of warranty, complexity of replacement. Doesn’t make sense. (commenting three months later when KVMac4 mk4 is in development. I was wrong; replacing power button makes sense)
- Software Path. You need to reverse engineer the USB-C port (you can find examples online), which is unreliable as any OS update can break this solution. Well, as far as I have searched so far, there is no option to put Mac mini into recovery mode.
MDM server?
- worth the money
- will not solve all the problems described in the beginning
Instead of an introduction
I am pleased to present you with a stand for 16 Apple Mac minis. Below is the third iteration, v3, Mark III or Mk3, of my stand. I think the previous two versions are worthy of mentioning but not repeating.
I would compare them like this (because I designed, built, and used them).
v1
simplicity – 8/10
reliability – 4/10
usability – 5/10
v2
simplicity – 2/10
reliability – 7/10
usability – 4/10
v3
simplicity – 7/10
reliability – 9/10
usability – 9/10
As you can guess – the main goal is reliability and ease of operation.
Assembly instructions, all the necessary links, and printable models will be in the second article. In the meantime, I encourage you to subscribe to my Instagram, Twitter (X) and join the Discord community. It means a lot to me. Thank you!
Story
At first, I wanted to skip it, but when selecting the photos, I realized how much work has been put into this project over the last year. It would be blasphemy to omit a bit of the story.
Well, I think it will add value to the project to show what it cost me in labor costs.
The project started in April 2022 with v1, a proof of concept, so to speak.
No one has ever done this before me (at least not publicly). So, I progressed little by little.
I’ll put photos and detailed descriptions of v1 and v2 under spoilers:
Version 1
After a bunch of experimentation that I shared in Instagram stories and a few posts, I came up with this solution:
The kinematics are as simple as possible. The servo uses the included lever and directly presses the button on the Mac mini.
At first, I wanted to use PiKVM in a standard enclosure, connecting servo HAT and PoE in this way:
But I ended up putting it together like this:
It was an interesting experience next.
It was very complicated to get a picture from a 16-port KVM switch. Lots of tests with different EDIDs. Experiments with different hardware options (Dr. HDMI and others). And in the end there are still problems. It was also quite difficult to learn how to control it via serial port (but I did it). In short, I do not recommend repeating my experience.
This Aten CS19216, which cost about 1.6k euros, supported 4k and USB3.0.
I had an idea to connect a USB disk to it and put OS images on it, but in the end I gave up this idea.
Without further ado, the final look of version 1.
And demos:
Overall, everything was working. Watch with sound
P.S. Once again, I’ll give a link to my Instagram, as I share the development there in the stories, not just the final result. And I am happy with the feedback from you.
Version 2
Work on the next version started four months later.
I realized that I had to rethink the concept completely.
Looking ahead, it turned out well, but it was too complicated. No, TOO COMPLICATED!
Version 1 had reliability issues. Probably 16 of the cheapest plastic servos couldn’t work perfectly. It’s surprising (sarcasm)
So I decided to experiment. I settled on this option, a compromise between reliability and popularity (reliable and unpopular ones will be difficult to replace and probably more expensive).
Small digression: I also tried a solenoid but concluded that there was no point. It requires a large current. I must still make a specific guide part to push the solenoid where I want it to go (button). The high current would require a relay HAT or extension board (with 16 relays) and an additional power supply.
I was still holding on to the installation with the ports up. However, I changed the switch to TESmart HKS1601A1U-EUBK (similar to the one I use in version 3). I also added a socket block directly on the shelf to reduce the number of wires.
You can evaluate the mechanical solution for yourself.
And in action:
In general, taking out a Mac is easy enough. The button presses perfectly and does not require fine-tuning of the servo because of the springs. If assembled carefully, I am sure it is very reliable. But there is an overengineering taste, isn’t it?
Case-stand for Raspberry Pi, PiKVM, PoE HAT, and servo HAT I decided to design myself:
In the end, however, the PoE HAT was abandoned because the switches in the data center did not support PoE at the time of installation.
And several flaws became immediately apparent:
- Complex cable management took up a lot of space.
- There was no front panel. In many big data centers, this is an important condition for installing equipment so as not to disturb the airflow from cold to hot corridors.
- The chosen shelf can hardly withstand such weight. Even in the photo, you can see that it is slightly bent.
Conclusions have been drawn. The main one was that I needed to rethink the kinematics and mechanical side of the project.
On the plus side, I chose a servo and a KVM switch and learned to control them from the Raspberry Pi and through the PiKVM interface.
P.S. Once again, I’ll give a link to my Instagram, as I share the development there in the stories, not just the final result. And I am happy with the feedback from you.
And below is what we’re here for, version 3.
Design
I hope my motivation is clear. Now, about implementation. I started three times from virtually nothing. The first two versions clung to the stand developed earlier. This was a mistake, and in the end, I completely redesigned the approach.
After moving the Mac mini around on the shelf, I found that it fits 16 pieces perfectly in this configuration.
The picture is from the thinking stage. I put under Mac minis whatever I could get my hands on to choose the installation angle.
To be able to install or take the Mac mini off the shelf easily. And most importantly, to eliminate any positioning nuances in its seating position. I ended up coming up with this seating design.
In v2, I switched to larger servos and took the approach that one servo serves 2 Mac minis simultaneously. This way, the stand can be divided into eight blocks, which are easy to repeat. And they are connected with the usual bolts. I could have done with just 3D printing, but I decided that, in this case, adding bolts would benefit the overall rigidity of the design.
It also allowed us to solve the cable management problem quite simply and beautifully.
Luck plays a relatively high role in such projects. And I was lucky to find an almost perfect shelf for my purposes. I literally had no choice. Because the overall weight is very serious, and the shelf I used for v1 and v2 is a bit saggy, and that’s scary.
So, I used the StarTech UNISLDSHF192 2U shelf (supports a total weight load of 50lb (22.7kg)). I also bought the UNISLDSHF19M (supports a total weight load of 50lb (22.7kg)), and it seems to be the same. Maybe the difference is in the certification (or the seller sent the wrong thing). The shelf was actually the very first purchase for the v3 due to the high maximum shelf load. I wanted to make the stand reliable and solid.
The shelf is mounted upside down, potentially losing 1U underneath, but if you have the same stand underneath, this problem is gone.
For the data center, it is important to use a front panel. The fans are supposed to be there. However, according to my measurements (with IR camera and monitoring dashboards), the overpressure in the cold aisle allows the stand to vent perfectly.
If they don’t like it and require me to add fans, I’ll update the article. (btw, I know I have data center engineers in my subscribers. Hello to you!)
Kinematics
As I wrote above in the story about v2, I settled on one servo that serves 2 Mac minis.
I’ve lost count of how many experimental variations there were, but this is the first photo of the final kinematics version I’ve found.
It’s evolved a little bit since then.
And again, I’ll give a link to my Instagram, as I share the development there in the stories, not just the final result. And I am happy with the feedback from you.
How KVM is implemented
At the heart of KVM is the excellent project PiKVM
More specifically, one of its features that is not obvious at first glance. The ability to connect PiKVM to a common KVM (link to docs) and essentially make a web interface to control a group of devices connected to this KVM. All you need is to control the KVM itself to switch ports on it, which can be done quite easily using a serial port.
How to control servo drives
Using the Servo HAT for the Raspberry Pi proved to be the optimal option
To control Servo HAT, I used self-written Python scripts and PiKVM interface customization options to call these scripts.
Where to put the power cords
I won’t deny it, and maybe this part needs some improvement.
Cables can be placed on the sides of the shelf using vertical PDUs if the server cabinet allows it. This particular one is quite narrow, so we just put it in the back. This had no noticeable effect on temperatures.
Reminds me of a super-computer cluster from some sci-fi movie
Well, that’s the whole story. Originally, I wanted to write the whole thing in one piece. But I’m afraid you wouldn’t have gotten to the assembly part. So, I put it in a separate article.
In the meantime, I encourage you to subscribe to my Instagram, Twitter (X) and join the Discord community. It means a lot to me. Thank you!
Read about how to assemble the stand in Part 2
I’m a systems engineer in JetBrains company. Uptime Lab founder. I’m glad to see you on my website! I hope you find my content useful. Please subscribe to my Instagram and Twitter. I post the newest updates there.
Looks great Ivan! I’m glad you included the previous revisions, interesting to see the evolution and refocusing on your core design goals of usability and reliability.
With everything you’ve learned from your 1u, 4 Mac project, do you think you’ll have a mk4 of the KVMac16 to include the power button changes or are your preferring the density/enclosed nature of KVMac4?
Looking forward to part 2!
Thank you! I apologize for the late reply.
I think the continuation of this idea can be a full-fledged custom bench. But with the idea of keeping the Mac mini in its original case.
The concept of disassembling and replacing the power button looks more promising. It’s more efficient, more reliable, and allows for higher-density installation.
Hope to share more news in the foreseeable future
just curious are mac minis noisy for homelab? for example 6 of them
I wouldn’t use them as a homelab…