Mdadm non standard encrypted raid

Following on from the failed AOE RAID experience, I'm going for mdadm RAID 10.

I am choosing mdmadm and RAID10 because it supports any size array of disks, not just matched pairs. For more information about the layout see the The Wikipedia page, there's no point me repeating that information here.

I'm using 3 off 1TB WD drives, non shingle. I would use all 5 of the ones I own but finding a case to house so many disk drives is proving challenging these days unless you buy a rack case and they are super expensive (at least for hobby projects like this). I do have a case with enough slots but the motherboard is too big!
THis is on POP_OS!

uname -a Linux pop_os 5.4.0-7634-generic #38~1595345317~20.04~a8480ad-Ubuntu SMP Wed Jul 22 15:13:45 UTC x86_64 x86_64 x86_64 GNU/Linux

The machine has 2 CPUs and 40 cores, and says 48Gb RAM though, though I thought it was supposed to be 64Gb so I need to work n that.

My disks still had remnants of the last experiment so I needed to run this first

# mdadm --stop /dev/md0

I'd already done this, so I'm not 100% sure if you need it (repeat for each disk)

mdadm --zero-superblock /dev/sda

This is to create the array, with default chunk of 512Kb

# mdadm --create /dev/md/trip --level=10 --layout=f3 --raid-devices=3 /dev/sda /dev/sdc /dev/sdd # cryptsetup luksFormat /dev/md127 ... enter new passphrase ... # cryptsetup luksOpen /dev/md127 etrip ... enter new passphrase ... # mkfs.ext4 /dev/mapper/etrip

This is not my boot disk, I already have a SSD installed for that with a passphrase at boot. I don't want to have to type the above passphrase again, so I need to make it use a keyfile.

The instructions for this I found here so I won't repeat all the stuff, just the commands

Add it to /etc/fstab in the normal way

/dev/mapper/etrip /mnt/trip ext4 defaults 0 0

The keyfile can live anywhere on your drive. The keystore for the drive has 8 slots (so different people can mount it with different passphrases). One way is to just pump out 32 random bytes and I'm going to keep it in /root/

# cryptsetup luksDump /dev/md127

So hopefully there's one left.

#dd if=/dev/random bs=32 count=1 of=/root/trip_key

Add it to the keystore for the drive

# cryptsetup luksAddKey /dev/md127 /root/trip_key

And then add it to /etc/crypttab

etrip /dev/md127 /root/trip_key

And that should be that, if it mounts on reboot, then it worked. It did for me.

Now that's all done, lets do a test

# fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=16 --size=4g --iodepth=8 --runtime=60 --time_based --end_fsync=0 --filename=/mnt/trip/test WRITE: bw=106MiB/s (111MB/s), 23B/s-23.6MiB/s (23B/s-24.7MB/s), io=17.7GiB (19.0GB), run=60376-170878msec

... hmm ...
Not as good as I hoped. I got 325MB/s with 5 drives and no-encryption.

The md process is still going so I think I'll wait, maybe it's doing something

Woo, a few hours later, I'll take that

WRITE: bw=288MiB/s (302MB/s), 17.5MiB/s-18.8MiB/s (18.4MB/s-19.8MB/s), io=38.8GiB (41.7GB), run=137970-137972msec