ARM SBCs have been around for a few years now. They were made popular and accessible mostly by the RaspberryPi released in 2012.
I’ve been enjoying building countless projects around these boards : web servers, home-automation, camera recording, IM bots,…
A new trend has recently appeared in the SBC world : the PCIe slot! The RaspberryPi 4 Compute Module exposes it via the IO board, and the Pine64 RockPro64 is equipped with a PCIe 4x open-ended slot directly on the board.
I haven’t had the opportunity to get hold of one of these boards, but I’m lucky enough to have a Quartz64 model A on hands.
The Quartz64 is based on the Rockchip RK3566 running @1.8Ghz. My board is equipped with 8GB of RAM. It also has Gigabit ethernet, 1 SATA connector, USB3, and of course, the PCIe port.
I experimented a bit with the PCIe interface by testing a couple of PCIe boards I had on hands at that time : the SATA controller board from Pine64 and an Intel 4x GBe board : they worked out of the box! Awesome, isn’t it?
I had an idea in the back of my mind : create a DIY low-power NAS based on a cheap SBC that can handle more than 2 drives (up to 6, preferably) without using USB. And it looks like this project is now completely possible! So… Let’s do it!
The first prototype was very rough : everything was laying on my desk, with all the wires dangling around. It was a bit messy, but it allowed me to check that the drives were working correctly.
When I ensured that everything was working as expected, I bought a few more items : the SATA controller with 6 ports and an enclosure for 6 2.5″ SATA drives.
This question haunted me for some time.
For now, the SBC is powered by a 12V power brick via a barrel connector. And the disks are powered by the board using those specific power cables by Pine64. If I understand correctly, this cable is connected to the 12V power supply from the board and embeds voltage s to provide 12V and 5V/3.3V to the disks.
It works fine with 2 drives but… I want 6 of them! And I’m not sure those voltage regulators would support more than 2 drives.
I could probably find a PSU that outputs 12V, 5V and 3.3V, build the wires and adaptors by finding the correct connectors, etc. Buuut… That looked too difficult for me, and I doubt the result would have been elegant and safe.
The solution I implemented is the PicoPSU. It’s a very small yet powerful DC-DC converter that is designed to power embedded PCs. It comes in the forms of small PBC mounted on a standard ATX connector that fits directly in the ATX connector of any standard motherboard.
It looks good but the Quartz64 does not have that ATX power connector! Fortunately, I remember a blog post on CNX-Soft about an ATX breakout board. It has the ATX power connector on 1 side, a screw terminal for +12V, -12V, 5V and 3.3V on the other side, and a switch and a few fuses in between.
The following picture shows the PicoPSU on the top and the ATX breakout board in the middle. The PicoPSU receives 12V from the black and white connector on the left. The Quartz64 is powered by the red and white wires on the bottom, and the SSDs by the classic red/yellow/black wires on the right.
This setup with 2 SSDs ran for a few months. But it’s still messy and fragile. I wanted to clean that up!
So I wanted to find a box to secure the whole build and hide all those wires… without breaking the bank!
The only compatible case I could find was the NAS enclosure from Pine64 but it supports only 2 drives.
I quickly figured I would need to tinker something based on a case that was not specifically designed to be compatible with the Quartz64! But hey! While I can do a lot of things with my 10 fingers on a keyboard, I’m not that good with my 2 hands and a screwdriver!
So I did not want to buy a shiny new case that I would certainly break while trying to drill a hole or something!
Luckily, I found an old (Pentium4-era) desktop computer that was going to be thrown to the landfill. I thought I could salvage the case and probably recycle the insides of that prehistoric computer!
So I emptied the case, cut the I/O shield with my Dremel and cut a thin aluminum plate so that it would fit in the case.
And I then drilled holes to fit the ATX breakout board, the PicoPSU and the Quartz64 with the SATA board.
While it’s not finished yet, I’m already pretty happy with the result so far! I’m planning on improving this build with an I/O shield for USB and Ethernet, connect the power button, the LEDs,… but that’s a project for another day!
Here are more details about the hardware I used in this build:
I installed the latest dev image from ManjaroARM on the eMMC flash module. Note that the support for the Quartz64 and the RK3566 is still in early development stage. So I expect a few issues and bugs, and I also expect for the support to improve very quickly in the next weeks and months!
As the PCIe controller is not supported yet in the default kernel (linux-rc, currently version 5.18-rc7), I switch to the kernel linux-quartz64 (version 5.17.0). Full support of PCIe should be added in kernel 5.19. Stay tuned 🙂
The proof that the support for the board is still pretty new is that at first, the filesystem on the SSD would corrupt as soon as I copied big files (> 4GB) on the disk.
This was apparently caused by an issue with the memory allocation for the SATA driver. With the help of the folks in the chat room dedicated to the Quartz64, we were able to patch the DTS/DTB files to work around this issue.
Those patches are not applied yet in newer manjaro images so I’ll provide these patches here. Use them at your own risks, of course! Neither I or the original author of those patches will be responsible for any issue that could occur when using those files!
Linux 5.19 should bring support for the PCIe controller on the Quartz64 and applying those patches won’t be needed anymore!
ZFS is very trendy right now, and I wanted to do some experiments with this very popular filesystem. Unfortunately, ZFS is not included in the Linux kernel. So you have to build the kernel module manually. ArchLinux and Manjaro provide the package in the AUR so the installation was not that difficult.
It worked quite well until ManjaroARM updated the kernel to a newer version that is not yet supported by the ZFS driver.
As I said earlier, the support for the Quartz64 is under heavy development, and Manjaro integrates bleeding edge versions of the kernel to provide the most up-to-date experience.
The Linux kernel is so bleeding edge that the ZFS module was not updated yet when I installed the new kernel on my setup, and it broke my shiny new ZFS filesystem.
I didn’t want my system to break randomly each time I ran pacman -Syu so… I decided ZFS was not the best fit for me right now.
At the beginning, I could find a good deal for only 2 SSDs. So after ZFS, I decided to test BTRFS. RAID 5 and 6 are not yet production-ready with BTRFS, but RAID 0 and 1 are completely supported. I’ve finally settled to format them in mirror mode (RAID 1).
This array will host the services I’ll run on this service : docker containers, my Matrix home-server and probably many others in the future!
I finally found a good deal for the 4 remaining SSDs, and I decided to use mdraid RAID 5 for these 4 drives.
I’ll use this array as data storage.
A screenshot is better than a thousand words!
As soon as I showcased my project, a lot of people asked about the performances of this setup. So here are a few numbers!
You’ll quickly notice that those are not the performances you were hoping for.
But in my opinion, knowing that everything worked nearly out of the box, those numbers are already quite great. And we’ll probably see those performances increase as the support for the board and the CPU improves in the near future!
dd# dd if=/dev/sdc of=/dev/null bs=4M status=progress iflag=direct
15497953280 bytes (15 GB, 14 GiB) copied, 44 s, 352 MB/s^C
3728+0 records in
3727+0 records out
15632171008 bytes (16 GB, 15 GiB) copied, 44.4116 s, 352 MB/s# dd if=bigfile.raw of=/dev/null bs=4M status=progress iflag=direct
31914983424 bytes (32 GB, 30 GiB) copied, 154 s, 207 MB/s
7609+1 records in
7609+1 records out
31914983424 bytes (32 GB, 30 GiB) copied, 154.001 s, 207 MB/s# dd if=/dev/zero of=./temp_zero bs=4M status=progress oflag=direct
6236930048 bytes (6.2 GB, 5.8 GiB) copied, 61 s, 102 MB/s^C
1497+0 records in
1497+0 records out
6278873088 bytes (6.3 GB, 5.8 GiB) copied, 61.5351 s, 102 MB/smdraid RAID 5 array:# dd if=bigfile.raw of=/dev/null bs=4M status=progress iflag=direct
31822184448 bytes (32 GB, 30 GiB) copied, 85 s, 374 MB/s
7609+1 records in
7609+1 records out
31914983424 bytes (32 GB, 30 GiB) copied, 85.2706 s, 374 MB/smdraid RAID 5 array:# dd if=/dev/zero of=./temp_zero bs=4M status=progress oflag=direct
4123000832 bytes (4.1 GB, 3.8 GiB) copied, 40 s, 103 MB/s^C
983+0 records in
983+0 records out
4123000832 bytes (4.1 GB, 3.8 GiB) copied, 40.0076 s, 103 MB/smdraid RAID 5 array : 60Mio/smdraid RAID 5 array : 90Mio/s8W (idle) – 12W (load)
On the hardware side, I would like to improve the wiring and cabling by adding an I/O shield for the ethernet and USB ports, add a power button, a reset button, a power led, and maybe paint the case to make it prettier.
I would also like to see how we could improve the performances of the system. There are probably configuration issues or bottlenecks that could be improved.
And finally, I would like to put this new server and NAS to good use! It’s already running my Matrix home-server. I would also like to store music and movies for my home theater and probably many other things in the future!
16 responses to “My DIY low-power 6 SSD NAS based on the Quartz64 ARM board”
Thanks for your article, I am also very fan of rockpro64 and quartz64 to run cool stuff.
I managed to boot manjaro arm release https://github.com/manjaro-arm/quartz64-a-images/releases
I am currently stuck with pcie support, as when I patch the kernel to linux-quartz64 (pacman -S linux-quartz64),
the board boots but usb does not work any longer, so I am not able to login.
Can you please detail the steps for pcie support?
Thx for your help
In my case, PCIe worked ‘out of the box’ with the linux-quartz64 kernel. I had to apply the patches for the DTS mentionned in the article to workaround memory corruption issues when transferring big files, though.
So, according to my notes, here’s what I did to install and patch the kernel:
# Install the kernel linux-quartz64
pacman -S linux-quartz64
# Convert the DTB to DTS
dtc -I dtb -O dts -o /root/rk3566-quartz64-a.dts /boot/dtbs/rockchip/rk3566-quartz64-a.dtb
# Apply the patches to the file /root/manjaro.dts...
# Backup the old DTS file just in case...
cp /boot/dtbs/rockchip/rk3566-quartz64-a.dtb /boot/dtbs/rockchip/rk3566-quartz64-a.dtb.backup
# Convert the new DTS to DTB
dtc -I dts -O dtb -o /boot/dtbs/rockchip/rk3566-quartz64-a.dtb /root/rk3566-quartz64-a.dts
You can also keep the original DTB file and create a new one with the patches. In this case, you can add a new entry in /boot/extlinux/extlinux.conf so that you can choose the DTB file to be loaded at boot. Ex:
default 2
menu title Quartz64 Boot Menu
prompt 0
timeout 50
LABEL Manjaro ARM
KERNEL /Image
FDT /dtbs/rockchip/rk3566-quartz64-a.dtb
APPEND initrd=/initramfs-linux.img console=ttyS2,1500000 root=PARTUUID=235dfca5-e43f-48d3-8b03-5a06174833bb rw rootwait audit=0 splash plymouth.ignore-serial-consoles
LABEL Manjaro ARM patched
KERNEL /Image
FDT /dtbs/rockchip/rk3566-quartz64-a-jf.dtb
APPEND initrd=/initramfs-linux.img console=ttyS2,1500000 root=PARTUUID=235dfca5-e43f-48d3-8b03-5a06174833bb rw rootwait audit=0 splash plymouth.ignore-serial-consoles
However, I haven’t tested USB in this configuration yet, I did the whole setup using the serial console and SSH. USB3 and SATA ports are mutually exclusive, you cannot use both at the same time. Have you already tried to connect your keyboard/mouse to the USB2 ports?
Thanks for taking time to answer me.
I am a bit confused with your answer as I don’t find dtb files in the minimal release I have installed.
As I want a server release I opted for Manjaro-ARM-minimal-quartz64-a-20220530.img.xz
This release comes with no dtb at all. As you can see out of the box it comes with empty /boot and /root
The only place where I can find drivers and dtb for quartz64 is https://gitlab.com/pgwipeout/quartz64_ci/-/pipelines
When I download the latest artifacts.zip, I comes with rk3566-quartz64-a.dtb, lib/modules/5.17.0, unfortunately image provided by pgwipeout rk3566-quartz64-a.dtb.img.xz does not boot for me as of 0530 release.
$ sudo ls -al /root /boot
/boot:
total 8
drwxr-xr-x 2 root root 4096 May 30 08:54 .
drwxr-xr-x 17 root root 4096 May 23 03:10 ..
/root:
total 12
drwxr-x— 3 root root 4096 May 23 03:11 .
drwxr-xr-x 17 root root 4096 May 23 03:10 ..
drwxr-xr-x 2 root root 4096 May 23 03:09 .inxi
Thanks so much for your support. I would like to try the same release as you as I am not very confident with my linux kernel patching skills.
Would you be so kind to tell me which one you used to get all the drivers and dtb on your boot image?
Regards,
I’m sorry to have taken so long to answer you!
Mhm it’s strange that you cannot find any DTB file on your setup… It shouldn’t be able to boot without them!
It’s normal that /root is empty, but /boot should definitely contains a few files (the kernel, DTB, U-Boot,…). Note that /boot is located on another partition. It should be automatically mounted at boot, but it’s worth checking if it’s mounted (see /dev/mmcblk1p1 that is mounted in /boot):
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
dev 3801676 0 3801676 0% /dev
run 3938476 908 3937568 1% /run
/dev/mmcblk1p2 14869164 3716328 10514040 27% /
tmpfs 3938476 0 3938476 0% /dev/shm
tmpfs 3938476 0 3938476 0% /tmp
/dev/mmcblk1p1 218512 41980 176532 20% /boot
tmpfs 787692 0 787692 0% /run/user/1000
$ ls /boot
dtbs extlinux idbloader.img Image initramfs-linux.img u-boot.itb
According to my download history, I installed Manjaro-ARM-minimal-quartz64-a-20220523.img.xz from here : https://github.com/manjaro-arm/quartz64-a-images/releases/download/20220523/Manjaro-ARM-minimal-quartz64-a-20220523.img.xz. Can you give it a shot and see if it works for you?
I like the idea of reusing the ATX case and the ExpressCage looks good.
I think what we really need for our ARM devices is a method for the eeprom to emulate pc BIOS and UEFI. This technology is esoteric and rarely tread, in my experience, but I’m doing what I can to better understand the PC boot process. Such a change would be a huge step in the right direction, because, as far as I am aware, this limitation is the only significant barrier to booting any operating system that supports ARM.
You may want to take a look at https://tow-boot.org
However, UEFI isn’t the only thing needed for running random OSes on random ARM hardware. x86 is much more standardized than your average embedded platform.
E.g. you do have ACPI and the vast majority of hardware is auto-discoverable. In contrast, ARM must usually rely on device trees
Still, the biggest roadblock are drivers. Intel and AMD processors keep backward compatibility and even then there’s only so many variants out there.
On the other hand ARM SoCs are quite diverse. Their core features can differ between vendors. Even something so crucial like an interrupt controller (the thing that tells the CPUs to switch what they are doing when certain events occur) can be implemented differently and so it may need a new driver.
That is a very good point. Perhaps if there was a way to embed that device tree in the spi flash rather than expecting every OS to provide it. 16 mbytes is a lot of space.
Hello,
nice project.
Why didn’t you use the ATX power supply from the scavenged HP case ?
This certainly should have worked too…
Thanks
The original power supply from the case would have perfectly worked, but it’s way too big (in size and power) and it has a fan. I wanted my build to be 100% silent so I switched to this PicoPSU power supply. It’ll also allow to easily move the build in a smaller case in the future 😉
Hello,
I am interested in your project but I am less hardware-maker savy than you all folks.
What do you think – will this project be one day convertible by beginners?
Even if not so deep in console programming – might it be one day more or less working out of the box?
I appreciate serious answers.
Enjoy your day
Olga
I’m not so good at DIY either… I just took my time and used salvaged parts (telling myself that it won’t matter if I break them). Otherwise, Pine64 also provide some enclosures for their board (looks at SBC accessories on their shop). For example, they designed a desktop/NAS enclosure for the RockPro64 (not sure if it’s compatible with the Quartz64, though).
Regarding the software setup, this will definitely improve with the time. Linux distribution and software support for the Quartz64 are still pretty young, but things are improving very quickly. Who knows… maybe one day OpenMediaVault will build an image for the Quartz64?
This sounds like a really interesting project and I’d love to do something similar as a home server, but I have few concerns.
I grew up with DOS and then Windows and i run a Windows 11 based home server.
The hardware side of things I am very comfortable with. I have very little experience of Linux and what I do have, I got by following step by step guides for installs of OpenVPN and Plex onto Raspberry Pi’s.
My Windows server gets through a lot of electricity and produces a lot of heat and noise. It has 4 x 6Tb hard drives in it, pooled using the Windows drive pooling service which lets you create a number of repositories with different levels of redundancy alongside each other. I am interested in switching to an Arm based core with SSD Sata drives in a Raid configuration or something similar to the Windows drive pooling service, but as a complete Linux novice I don’t know if this is a reasonable objective or step too far too quickly.
Is it possible to get more detailed guidance on the software side of how you set up your server? Where is the best place for me to learn what I would need to know?
Thank you
For this project, I installed and configure everything manually : install the OS, configure the firewall, set up the disks in 2 RAID arrays (btrfs and mdraid), samba,… I haven’t taken enough notes during the installation process to be able to write a step-by-step guide about it, unfortunately.
You might want to have a look at Openmediavault which provides a nice web interface to configure the whole system, storage, network shares,… But I don’t know if they already provide an image for the Quartz64.
Going through some similar problems here but without the same tooling at your disposal; in journeying through this at home I’ve been musing about a need for a new kind of file system that borrows some of the features one gets by using OverlayFS for the read-only metadata tax deductions, as well as the append-only concurrent write batches from other popular fuse fs’s at the moment. Mixing and matching good and bad SSDs and HDDs respectively for a unified way to break open the HDD IO bottlenecks on my disks has me constructing something similar, so your notes here will be vital for me going forward. Thanks a billion!
nice project, I just bought some cheap old HP Microserver (4 x 3,5″ bay, one empty 5,25″ slot).
I will just use this as is (almost with Pico PSU) running Xpenology, maybe thats a case for rehousing your Quartz64