< Back

TrueNAS Scale

TrueNAS Migration

Over the last few years, I have maintained a growing data collection and a long list of self-hosted services. I've quickly become a regular reader of r/datahoarder and r/selfhosted. Recently, I began approaching the limits of my hardware with 19 of my 22TB storage in use. It was clear that an upgrade was due, but I needed to figure out how I wanted to see it through.

The state of affairs

Before migration, I was running Unraid with a Basic license. My server had the following specs:

  • AMD Ryzen 7 3700X
  • TeamGroup 64 GB DDR4
  • Nvidia GTX 1050 Ti
  • 2x 3TB Seagate Barracuda HDD
  • 1x 16TB Seagate Exos HDD
  • 1x 512gb Intel M.2 NVME SSD
  • 1x 240gb Sandisk SATA SSD

The HDDs were part of the Unraid array with no parity configured, whilst the SSDs were used as caches for highly volatile and application data, respectively. This meant that none of my data had any redundancy and, unfortunately, no backups.

As part of this migration, I wanted to increase storage capacity, introduce redundancy for data that I didn't want to lose, and introduce a backup system.

Additionally, I was eager to move to a repeatable application deployment format. The applications deployed in Unraid's docker feature were deployed from the "Community Apps" store, which would be difficult to reproduce if later my requirements changed.

Evaluating options

As I saw it, there were two options: stick with Unraid, add drives to the pool, and introduce a parity drive, or migrate to TrueNAS Scale.

The decision ultimately boiled down to one key component: multiple storage classes. Choosing TrueNAS allowed me to introduce a highly durable, highly redundant storage class for application data I was unwilling to lose such as family photos, specific application data, hackathon project submissions, etc. This storage class would be separate from the regularly read/written drives used for bulk file storage.

The move to TrueNAS also came with a move to Kubernetes. I was later informed that TrueNAS will be dropping native Kubernetes support in 2024, so my deployment strategy reflects that.

The plan

  1. Acquire 2x 16TB HDDs.
  2. Delete data until it all fits on the 3TB drives.
  3. Install the new HDDs
  4. Install TrueNAS
  5. Create a VDEV and ZPOOL for low assurance data (3x 16TB RAIDZ1).
  6. Mount the 3TB HDDs and copy data to the new ZPOOL.
  7. Create a VDEV and ZPOOL for high-assurance data (2x 3TB MIRROR).
  8. Deploy Kubernetes and associated services

The hard(ware) reality

Installing the HDDs proved to be a difficult situation. The IPC G4380 case my machine is mounted in has many drive bays, but they leave little room between the drive and the CPU cooler for cables. Additionally, 3 drive bays conflict with the installed GPU (used for transcoding media and local LLMs).

My poor planning was highlighted when it was identified that the motherboard had no remaining SATA ports for the additional drives. Fortunately, John Cave, a friend who also maintains a homelab, had an 8-port SATA host bus adapter (HBA), which I purchased from him.

A quick stop at the local computer shop solved the drive mounting issue using 5.25" to 3.5" adapters.

Installing TrueNAS on the M.2 SSD also proved to be a mistake, with the 240gb SSD not offering sufficient storage for my application data. Another trip to the local computer shop yielded a 1TB PNY SATA SSD to replace the 240Gb Sandisk drive.

Software

Seeing as TrueNAS is removing Kubernetes, I followed a guide from the TrueNAS forums to install Kubernetes (K3S) in a Systemd namespaced container (nspawn). The default K3S ingress controller, Traefik, was replaced with nginx-ingress for improved flexibility, and the nVidia container toolkit was painstakingly installed to allow for GPU passthrough to LLM and Transcoding pods.

This was my first time working directly with Kubernetes, particularly with a fresh install, and it was an exciting experience. It took some time to understand all of the entities and relationships that Kubernetes introduces to support its highly modular design.

Following this, applications were deployed to the "cluster" using Podinate, a tool developed by the aforementioned John Cave.

Today, I can deploy a Docker container to my server with domain-based ingress, SSL, multiple storage classes, and GPU passthrough in just a couple of minutes.

Future Improvements

Looking forward, I am interested in introducing strict ACL/ACPs to allow others to deploy to my cluster during hackathons without worrying about my other deployments being affected. I am also interested in increasing the number of applications I am hosting, reducing my reliance on AWS for personal projects, and deploying tools like Coolify in a VM.

Lastly, I am yet to introduce a backup solution. It would be foolish to let this fall by the wayside as I did with , but only time will tell.