Home Setup a Glusterfs volume for k3s persistent storage
Post
Cancel

Setup a Glusterfs volume for k3s persistent storage

Posted Aug 14, 2022
By Jason Witty
8 min read

Overview

This document will outline how to setup a gluster fs volume using Odroid HC1 (home cloud 1) SBC as bricks in a distributed / replicated array and use this storage in your Kubernetes or k3s cluster. The goal of this project is create a highly performant and highly available persistant storage system using single board computers to host each brick in order to create complete redundancy in your volume. (In contrast to a classic RAID setup will provide redundancy in the case of a drive failure but not in the case of a failed host.)

Glusterfs Cluster

Please note that this guide can be used to setup storage using any set of SBCs that follow the same principals of an Odroid HC2. These devices were a perfect fit for this application because they were originally cheap (around 60$ USD) and had enough IO to max out a 1GB ethernet port and work great over POE. Since I originally purchased these SBCs Hard Kernel has since discountinued them due to the chip shortage, this has caused them to get sold by scalpers on AliExpress and Ebay for over 200$, these devices are not worth over $200 so please don’t pay these prices. From private sellers you can often find them at near msrp if you are patient or you can choose a different SBC. The HC4 seems like and obvious choice but I personally dont like the form factor of this device and think it will be difficult to mount. My personal research led me to the next best option being a mini based board a or b from waveshare (20-30$) which will accept a 2242 M.2 SSD (20-100$ on newegg depending on size) and a raspberry pi compute module (~30$). At the time of writing this compute modules are also in short supply but I expect that this will be resolved over time.

What you will need to get started

  • Single Board Computers (SBCs) - You will need multiples of the amount of replicas you want to keep in your volume. You can read more about your different options on the glusterfs architecture page. I went with 3 replicas in a Distributed Replicated Glusterfs Volume to achieve both optimal performance and data redundancy at the cost of giving up some storage. This meant I initially needed SBCs and scaling will then be done in multiples of 3. If you were to configure your volume with 2 replicas, you could start with 4 and scale up in multiples of 2.
  • SSDs, M2 or Hard disks - One for each SBC.
  • Networking - Switch, cables, etc
  • Micro SD cards - Choose endurance Micro SD cards to ensure you dont need to replace them often as these devices will be always on.
  • Micro SD card reader - to write OS images from your PC
  • (optional) PPOE adapters - These devices run wonderful and stable over POE as they use less than 5 watts to power at all times when using an SSD. PoE Splitter

Image OS to SD card

Download the Jammy image (ubuntu-22.04-5.4-minimal-odroid-xu4-20220721.img.xz) from odroid.in and image with etcher on your desktop.

Image with etcher

Setup Bricks

Odroid devices are completely headless, meaning you will need to manage it remotely. Place the sd card into the device and power it on, then check in your DHCP reservations to see the IP address that it was assigned and ssh to it using the username root and password odroid. Once you are logged in, reset the password, set a static IP for your device on your DHCP server (screenshot from unifi below) and reboot.

Setting static IP in unifi console

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

$ ssh root@192.168.1.29
The authenticity of host '192.168.1.29 (192.168.1.29)' can't be established.
ECDSA key fingerprint is SHA256:eZPKKXBY3E0nzOJzof/nFFvK7Q/51wudC0E/SvHtaTw.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.1.29' (ECDSA) to the list of known hosts.
root@192.168.1.29's password: 
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.4.196-406 armv7l)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
Last login: Sun Aug 21 21:15:06 2022 from 192.168.1.88

root@odroid:~# passwd
New password: 
Retype new password: 
passwd: password updated successfully
root@odroid:~# reboot

Now that the ip address and password have been set and we have rebooted, we want to pull down the setup script and transfer them to the brick to setup. Since this device does not even have git or curl or anything installed at this point in time we will pull down the script from the repo on your desktop and then transfer to the bricks using scp.

The hostfile txt file is currently configured for my personal network please modify this file accordingly to match your own network specs. Please note that it is very important for your hostfiles to be configured on all bricks as well as clients for glusterfs to operate properly. Even if you configure your mount paths to use IP addresses you will find that your clients will still attempt to connect via hostname. If you are having problems with glusterfs at the server or client please double check that your hosts are all set correctly. Local DNS is also recomended in case something is missing but the system is rather chatty so having hostfile will eliminate a good amount of dns requests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

$ git clone https://github.com/jasonwitty/homelab-glusterfs
$ cd homelab-glusterfs

$ scp * root@192.168.1.28:~/
root@192.168.1.28's password: 
gluster-hosts.txt                                                                  100%  188   153.8KB/s   
setup-brick.sh                                                                     100% 3491     3.3MB/s                                                 

$ ssh root@192.168.1.28
root@192.168.1.28's password: 
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.4.196-406 armv7l)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
Last login: Sun Aug 21 21:40:37 2022 from 192.168.1.88
root@odroid:~# ls
gluster-hosts.txt  setup-brick.sh

root@odroid:~# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0 476.9G  0 disk 
mmcblk1     179:0    0  58.2G  0 disk 
├─mmcblk1p1 179:1    0   128M  0 part /media/boot
└─mmcblk1p2 179:2    0  58.1G  0 part /

Now we can execute the setup script, which completes the following tasks

  • Setup Nala package manager.
  • Fetch fastest mirrors and update system.
  • Install common tools needed for setup and management.
  • Setup partitions and prepare SSD. Please note that this will erase your primary disk. Recomend running lsblk to verify disk is named sda, for example if you are using an nvme it will be named nvme0n1.
  • Create mount point and modify /etc/fstab to automatically mount disk.
  • Setup Gluster repositories and install glusterFS.
  • Restart system.

Review Script here: https://github.com/jasonwitty/homelab-glusterfs/blob/main/setup-brick.sh

Usage: sudo bash setup-brick.sh gluster-hosts.txt odroid7 /dev/sda arguments: hostfiles, hostname, drive name

Run script on each brick, updating the hostname and optionally customizing the hostfile on each machine. This will take some time to complete, once completed it will automatically resart the system.

** You will find in this script I am installing a specific version of gluster and not the latest (10.1 at the time of writing this). This is because I have been running it for some time and need new bricks to match previous versions. I have done extensive testing on 8.6 with k3s and proxmox and know that is it stable. Feel free to try the latest version but I have not tested it and have no need to upgrade at the current time.**

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ root@odroid:~# sudo bash setup-brick.sh gluster-hosts.txt odroid1 /dev/sda

UPDATE:::: Setting hostname to odroid8
deb https://deb.volian.org/volian/ scar main
deb-src https://deb.volian.org/volian/ scar main
UPDATE:::: install nala
Get:1 https://deb.volian.org/volian scar InRelease [15.5 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports jammy InRelease [270 kB]                                    
Get:3 http://deb.odroid.in/5422-s jammy InRelease [1717 B]                    
Get:4 http://ppa.launchpad.net/hardkernel/ppa/ubuntu jammy InRelease [18.0 kB]          
Get:5 https://deb.volian.org/volian scar/main Sources [6260 B]      
Get:6 https://deb.volian.org/volian scar/main armhf Packages 
...
Brick Setup complete !
See https://www.wittyoneoff.com/gluster-fs for next steps.
...

After reboot log back into the each brick and run the following commands to make sure everything was setup correctly.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

            .-/+oossssoo+/-.               root@odroid8 
        `:+ssssssssssssssssss+:`           ------------ 
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 22.04.1 LTS armv7l 
    .ossssssssssssssssssdMMMNysssso.       Host: Hardkernel Odroid XU4 
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 5.4.196-406 
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 1 min 
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 663 (dpkg) 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.1.16 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Terminal: /dev/pts/0 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: Hardkernel ODROID-XU4 (8) @ 1.400GHz 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Memory: 106MiB / 1991MiB 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+
.ssssssssdMMMNhsssssssssshNMMMdssssssss.                           
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/                            
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.

root@odroid8:~# hostname
odroid8

root@odroid8:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
tmpfs             203920    3448    200472   2% /run
/dev/mmcblk1p2  59953892 2745040  57192468   5% /
tmpfs            1019580       0   1019580   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
/dev/mmcblk1p1    130798   14186    116612  11% /media/boot
/dev/sda1      491134172      28 466112432   1% /gfs/brick1
tmpfs             203916       0    203916   0% /run/user/0

root@odroid8:~# cat /etc/hosts
127.0.0.1	odroid
127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters

#hostname added by glusterfs setup script
127.0.0.1     odroid8

#added by glusterfs setup script
192.168.1.11 odroid1
192.168.1.12 odroid2
192.168.1.13 odroid3
192.168.1.14 odroid4
192.168.1.15 odroid5
192.168.1.16 odroid6
192.168.1.27 odroid7
#192.168.1.28 odroid8
192.168.1.29 odroid9

root@odroid8:~# glusterfs --version
glusterfs 10.1
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
Homelab, Storage
k3s glusterfs
This post is licensed under CC BY 4.0 by the author.
Trending Tags
glusterfs k3s
Contents

-

-

Trending Tags
glusterfs k3s