As promised in the last month’s community update Housekeeping section, this is the first part of the cluster’s build log! This mini-series in which I will cover everything about upcoming changes our hosting cluster, and it begins with building a temporary cluster.
Let’s start from beginning. When we built the current hosting cluster a year ago, it was made in quite a hurry (some of the services needed to be migrated to the cluster on a moment’s notice). Also, it was the first time I (gamiee) and Matthew (fire219) were tasked with building a cluster. Because of this, some of the cluster’s features are missing and various things don’t work quite as well as we have hoped. For example, the cluster has power management boards (PCB on right side of every row), which are used primary for power distribution to the SBCs, but also it also has a I2C GPIO Expander for rebooting and turning off the SBCs. The I2C Expansion isn’t in use yet, because the primary node wasn’t able to detect it, so this is something that needs to be checked out when the cluster is turned off. And this is exactly the main reason why we are building a temporary cluster. We need to do some changes which require us to have the whole cluster turned off and on a workbench (not in housing room where isn’t much space), and this means that it will be inaccessible for several days. Obviously a long downtime wouldn’t be good when 100% of PINE64 community services are running on it. So we are going to build a temporary small cluster, which will have the minimal number of required nodes to have all services running, so we can make all upgrades with precision and at our own pace.
There are two things each node needs: a power supply and access to the network. Access to the network can be solved by using a classic Ethernet switch, but powering multiple RockPros, which are actually pretty power hungry, is a bit problematic if we don’t want to have multiple power adapters. One of the solutions is to design a very simple power distribution board.
When designing a power distribution board, we first need to calculate how much power the the whole cluster will require. According PINE64 Wiki – RockPro64 Page – Part Board Information, Schematics and Certifications, one node will need +12V and 3 – 5 Amps, supplied from 5.5mm/2.1 Barrel jack, or via JST XH connector, which is primary used for powering an external hard drive. I chose to use barrel jacks; 3 – 5 Amps is quite a lot, and that’s because it contains the current required for external PCIe device such as 3.5″ HDD. Various benchmarks show that the typical power consumption of each board is around 7 Watts. Our temporary cluster will have 7 nodes, so we need a power supply that can supply 12V and approx. 5 Amps. Power supplies which can fulfill these requirements include those used for laptops, classical PC power supplies (ATX) and enclosed power supplies.
I chose to use a classic ATX power supply since they are cheap and really easy to source. When sourcing a PC power supplies one needs to be certain that it can actually supply the required current. Some power supplies have two +12V rails, 12V2 which powers the CPU and 12V1 which powers everything else. Since we will use the main ATX power connector, this one uses 12V1 rail, so we need to determine if this rail can supply the voltage we need (most of those vendors don’t specify the currents for both rails, it mostly show sum of both rails for 12V). But if we will use a power supply that has a single 12V rail for everything (like I do), then finding the current for 12V is easy. Having sorted out the power supply, we can start designing the board.
As I mentioned before, we are going to use main ATX Power connector, which is a 20-pin or 24-pin Molex. We will use 24-pin one, since it contains additional +12 V pin. We can find this connector as Molex 39-28-1243. The pinout is here:
From this connector we will just need +12V, Ground and Power On pin. The Power On pin needs to be connected to ground by the “motherboard” to turn on the power supply. I added a 2-pin jumper header for this, so I can turn it on / off using the jumper. The last thing we need is power output for the nodes. There are several ways of doing this: we can use USB female connectors (with USB to Barrel Jack cable), barrel jack female connector (with barrel jack-to-barrel jack cable) and screw-on terminals or just holes, where you can screw / solder the wires going out from barrel jack cable. I chosen to use just the holes. Doing so I had to be careful about the size and distance between them. Since I picked a very accurate size and distance, it proved a bit problematic to solder cables into it (I will use screw terminals next time). Basically, that’s all! It’s nothing fancy, but it will do the job for the time being while the hosting cluster is in maintenance.
I was thinking about various layouts that can be used for this temporary cluster. I can place every node on the side, but that will be kind of big. I can also place the nodes vertically, but that would be harder to mount, or I could just stack the boards and use distance field, as Electr0Lyte did with his cluster:
This inspired me and I decided to use the classic layout, which most of those small clusters use. I designed the power board so that it will be mounted vertically together with a switch. I 3D printed the holders for the switch and power board, mounted it into the plexiglass and here we go we made our own temporary cluster!
This temporary cluster will be launched on September 2, 2021 from 8:00 AM to 11:00 AM UTC, If everything goes well. After the launch, the large hosting cluster will be moved to my place so I can do all the upgrades – something that couldn’t be made on-site.
This is all for this first update log, please let me know if you liked the extensive description of the process, or if it should be shorter. Also I want to say thanks to Lukasz, Matthew, PizzaMartijn, Maya.b and Electr0Lyte for all their help. See you in next update log!
15 responses to “Cluster build log: Moving to temporary cluster”
This is the coolest thing I have read about in a long time. I hope to get my Pine64 Phone soon!
Pardon me for being thick, but where is the storage on a system like this?
The main node (which manages all traffic) have eMMC, but the other nodes have network file system. The file system server => iSCSI is on RockPro64 which is on top, second row, having PCIe to M.2 adapter, with SSD.
Nice report; is there a technical-schematic drawing / blueprint available for this server where we can see all wires and description what they are used for?
Regardes
Hi Sebastian, there is temporary cluster’s power board schematics and board released. For the “primary cluster”, there is nothing available yet, but in incoming update logs, everything from software stack to hardware wiring or schematics will be described and published 🙂
Thanks for the work. I was wondering about the software stack that run on the cluster. Probably it is mentioned on old blog posts, but a short description of it would be nice to have.
Hi Eugen, Thanks! 🙂 The software stack is not much described even in older blog posts, but I plan to describe it together with how hardware is made in upcoming update logs, so stay tuned! 🙂
im not stay tuned because you delete my reply
What’s with the image on the main blog page. I don’t see it elsewhere in this post.
https://www.pine64.org/wp-content/uploads/2021/09/cluster-maintenance-p1-1230×692.jpg
I have been looking forward to this series, so huge thanks! 🙂 As per your question: I would love to here as much detail as possible, and I am also interested in the software stack, so don’t hesitate to share as much detail as possible. 🙂 I know time is always short and hard to find for writing, but I want to you to know, that we appreciate the effort of writing this (hopefully these) blog posts.
Keep up the good work!
Gergely
Your setup is a lot cleaner than mine ! I use a 8-pin Molex to barrel splitter.
My stack :
– a HA kubernetes cluster
– a distributed MinIO cluster
– an OpenWRT distribution as router
why admin delete my english post?
This is very cool. I’ve been considering a clusterboard, but I think I might be better off just hand-building a cluster to play with. Maybe I’ll do the regular 7-node board just for space purposes since I know what to expect, and then migrate some of the clustering configs and similar over to a larger one later.
I think I have one of every line of SBC PINE64 makes, I could surely cluster some together ubt don’t have nything really set up for that yet, not even a good space.
Hey guys is this a custom made chassis ? 🙂
Great work! I just wanna give Feedback of please more not less. If possible I’d be also interested on how you build the pure mechanical of the server, like did you order one custom case like this or did you rip out some fan plates from old machines and pin them in there?
Cheers