Mdadm non standard encrypted raid

Following on from the failed AOE RAID experience, I'm going for mdadm RAID 10.

I am choosing mdmadm and RAID10 because it supports any size array of disks, not just matched pairs. For more information about the layout see the The Wikipedia page, there's no point me repeating that information here.

I'm using 3 off 1TB WD drives, non shingle. I would use all 5 of the ones I own but finding a case to house so many disk drives is proving challenging these days unless you buy a rack case and they are super expensive (at least for hobby projects like this). I do have a case with enough slots but the motherboard is too big!
THis is on POP_OS!

uname -a Linux pop_os 5.4.0-7634-generic #38~1595345317~20.04~a8480ad-Ubuntu SMP Wed Jul 22 15:13:45 UTC x86_64 x86_64 x86_64 GNU/Linux

The machine has 2 CPUs and 40 cores, and says 48Gb RAM though, though I thought it was supposed to be 64Gb so I need to work n that.

My disks still had remnants of the last experiment so I needed to run this first

# mdadm --stop /dev/md0

I'd already done this, so I'm not 100% sure if you need it (repeat for each disk)

mdadm --zero-superblock /dev/sda

This is to create the array, with default chunk of 512Kb

# mdadm --create /dev/md/trip --level=10 --layout=f3 --raid-devices=3 /dev/sda /dev/sdc /dev/sdd # cryptsetup luksFormat /dev/md127 ... enter new passphrase ... # cryptsetup luksOpen /dev/md127 etrip ... enter new passphrase ... # mkfs.ext4 /dev/mapper/etrip

This is not my boot disk, I already have a SSD installed for that with a passphrase at boot. I don't want to have to type the above passphrase again, so I need to make it use a keyfile.

The instructions for this I found here so I won't repeat all the stuff, just the commands

Add it to /etc/fstab in the normal way

/dev/mapper/etrip /mnt/trip ext4 defaults 0 0

The keyfile can live anywhere on your drive. The keystore for the drive has 8 slots (so different people can mount it with different passphrases). One way is to just pump out 32 random bytes and I'm going to keep it in /root/

# cryptsetup luksDump /dev/md127

So hopefully there's one left.

#dd if=/dev/random bs=32 count=1 of=/root/trip_key

Add it to the keystore for the drive

# cryptsetup luksAddKey /dev/md127 /root/trip_key

And then add it to /etc/crypttab

etrip /dev/md127 /root/trip_key

And that should be that, if it mounts on reboot, then it worked. It did for me.

Now that's all done, lets do a test

# fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=16 --size=4g --iodepth=8 --runtime=60 --time_based --end_fsync=0 --filename=/mnt/trip/test WRITE: bw=106MiB/s (111MB/s), 23B/s-23.6MiB/s (23B/s-24.7MB/s), io=17.7GiB (19.0GB), run=60376-170878msec

... hmm ...
Not as good as I hoped. I got 325MB/s with 5 drives and no-encryption.

The md process is still going so I think I'll wait, maybe it's doing something

Woo, a few hours later, I'll take that

WRITE: bw=288MiB/s (302MB/s), 17.5MiB/s-18.8MiB/s (18.4MB/s-19.8MB/s), io=38.8GiB (41.7GB), run=137970-137972msec

Building an Encrypted ATA Over ethernet (AOE) RAID Network for Serving Big Data? Don't bother

This was a waste of time, AOE on Linux this way is far too slow. - and it wasn't anything to do with the encryption. Or LUKS I did the same with mdadm

This year's project started out as an idea for building a performant SQL database. The size of the data I plan to process is larger than the system memory I have, so it will be I/O limited. I have a mix of SSD and mechanical drives available.

I have some ideas to explore so this blog will follow the success and failure of those ideas as I explore the space.

Hardware

I have 5x Western Digital WD RE2-GP WD1000FYPS 1 Tb hard drives. I got those for £18 each from ebay. I made sure these were not Shingle Drives.

I had a spare mtoherboard which was declared unrepairable and the company I worked for paid £2000 for a replacement. I was tasked with disposing of it, so I took it home - free case and a PSU. But when I booted it up, I noticed it booted and then overheated - I spent £3 on a new CPU fan and it worked once again. It only had 2Gb of RAM so I replaced it with 16GB for £47.80

Finding a case with enough hard drive bays was a challenge. I bought what I thought was one for £30 but it had a misleading description. I've ended up swapping out my existing terminal.

Operating System

The plan is to serve AOE targets and combine them into a RAID. That way the storage capacity is expandable by adding more vblade servers and is not limited by the capacity of a single machine. It also offeres the possibilty of using different OSes to do the serving. I wanted to use OpenBSD but discovered they had removed AoE from their offering. Thanks Theo. Plan9 will serve AoE and I will add that via 9front at some point but am not keen to mess about with that today. I shall use the old warhorse Debian for the moment.

A beauty of simply serving vblades is the host is simple. It just needs decent I/O (which might not be in Linux favour). And you can swap your OS out and use another and just serve up the raw disk.

For some reason, USB storage devices seem to have complexified. Just dd the iso to the USB device wants to not work all the time. So one technqiue I like now is to use QEMU to boot the install disk and write straight to the drive.

With the target drive attached to a Linux box which shows up as /dev/sdf stsart qemu up like so:

qemu-system-x86_64 --enable-kvm -m 1024 -cdrom debian-10.4.0-amd64-netinst.iso -drive file=/dev/sdf,driver=raw

(make sure the -m 1024 is there, Debian installer will crash with a crypitc message)
and then run the Debian installer as normal.

Next up add the vblade package apt install vblade

#!/bin/sh if=enp3s0 /usr/sbin/vbladed -d 1 1 $if /dev/sdb /usr/sbin/vbladed -d 1 2 $if /dev/sdc /usr/sbin/vbladed -d 1 3 $if /dev/sdd /usr/sbin/vbladed -d 1 4 $if /dev/sde /usr/sbin/vbladed -d 1 5 $if /dev/sdf

So that's the first setup, the next stage is to create the software raid on another machine

RAIDin time

Over to another machine and we can test the vblades

apt install aoetools

Then seek out the vblades

root@hex:/ # aoe-discover; aoe-stat e1.1 1000.204GB enp3s0 1024 up e1.2 1000.204GB enp3s0 1024 up e1.3 1000.204GB enp3s0 1024 up e1.4 1000.204GB enp3s0 1024 up e1.5 1000.204GB enp3s0 1024 up

All looks good, now let's join them together, for that we'll need lvm - at least I hope this is the right way, I've never done it before, I'm copying this process from the gentoo wiki

apt install lvm2

Tag the partitions with volumes - the disks already had a partition table, hopefully it works - I've only left the message on the first one.

# lvm pvcreate /dev/etherd/e1.1p1 WARNING: ntfs signature detected on e1.1p1 at offset 3. Wipe it? [y/n]: y Wiping ntfs signature on e1.1p1. Physical volume "e1.1p1" successfully created. lvm pvcreate /dev/etherd/e1.2p1 lvm pvcreate /dev/etherd/e1.3p1 lvm pvcreate /dev/etherd/e1.4p1 lvm pvcreate /dev/etherd/e1.5p1

Then join them together

# cd /dev/etherd # vgcreate raid0vg0 e1.1p1 e1.2p1 e1.3p1 e1.4p1 e1.5p1

The instructions there use RAID1, so for this first experiment I'll do that. I don't plan to do that but for now I'll go with it for learning sake

root@hex:/dev/etherd# lvcreate --mirrors 1 --type raid1 -l 100%FREE --nosync -n raid0lv0 raid0vg0 WARNING: New raid1 won't be synchronised. Don't read what you didn't write! Logical volume "raid0lv0" created.

The Wiki creates an EXT4 filesystem on this

# mkfs.ext4 /dev/raid0vg0/raid0lv0 mke2fs 1.43.4 (31-Jan-2017) Creating filesystem with 488377344 4k blocks and 122101760 inodes Filesystem UUID: 8c8a45f5-27fa-4964-befa-143d32365878 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000, 214990848 Allocating group tables: done Writing inode tables: done Creating journal (262144 blocks): done Writing superblocks and filesystem accounting information: done

this took about 30s and the two of the 5 vblades were at about 15% CPU during that time. It is now mountable.

root@hex:/dev# mkdir /mnt/raid root@hex:/dev# mount /dev/raid0vg0/raid0lv0 /mnt/raid root@hex:/dev# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/raid0vg0-raid0lv0 1.8T 77M 1.7T 1% /mnt/raid

runnign fio on that got some terrible speeds

WRITE: io=1432.9MB, aggrb=3206KB/s, minb=3206KB/s, maxb=3206KB/s, mint=457632msec, maxt=457632msec

3Mb/s! I don't know if it was disk constrained or what. Only 1.1 and 1.2 had any activity in iotop, so perhaps the RAID level is a factor - it's not striping across all disks. I'm also going to turn on Jumbo frames

# ip link set enp3s0 mtu 9000

and see if that makes a difference. I'm also going to run iftop as well to see what that says

WRITE: io=1446.9MB, aggrb=6335KB/s, minb=6335KB/s, maxb=6335KB/s, mint=233857msec, maxt=233857msec

At least it doubled it. Interestingly it didn't show up on the server side. Ah, RTFM, it was a single process and waiting for a fsync. Try it multi threaded and don't wait for fsync like a real server

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=16 --size=4g --iodepth=8 --runtime=60 --time_based --end_fsync=0 --filename=/dev/mapper/raid0vg0-raid0lv0

and the result is a saturated 1Gb network link, just like I originally expected

WRITE: io=66441MB, aggrb=1105.7MB/s, minb=58076KB/s, maxb=76711KB/s, mint=60001msec, maxt=60092msec

So what good is a 1Gb disk? SATA on it's own would be fatser than that .... the answer is why I started - I have a couple of 10Gbe NICs and a dual 10Gbe motherboard (I haven't got that going yet, I need a VGA cable!

On top of that, we have encryption to do and we can do that on 1gb

# apt install cryptsetup

Need to reboot after this. And then get the LVM back into the system

# vgchange -a y raid0vg0 # lvchange -a y raid0vg0 # (this might not be needed)

And then we can encrypt the volume

# cryptsetup luksFormat -c aes-xts-plain64:sha256 -s 256 /dev/raid0vg0/raid0lv0

And then map the partition, it appears in /dev/mapper

cryptsetup luksOpen /dev/raid0vg0/raid0lv0 raid0lv0encripted

And then we can make a filesystem on it

# mkfs.ext4 /dev/mapper/raid0lv0encripted

One I got my ducks in a row with MTUs and testing the correct volume I got

WRITE: io=85259MB, aggrb=1420.5MB/s, minb=76220KB/s, maxb=103820KB/s, mint=60001msec, maxt=60023msec

Which isn't good - and 100% CPU on the client end doing the decryption - it certainly got the fans working!

CSP in Julia - Sieve of Eratosthenes

The Sieve of Eratosthenes

Finally I got my head round Julia's CSP technique. The nomenclature is a bit different to what I'm used to. This is using producer/consumer which is single threaded. To do multi threaded uses different primitives, which is a shame. Perhaps one can adapt this setup using multiple RemoteRefs. We'll see.

This algorithm is translated from the Limbo code in Stanley Marbell's book Inferno Programming With Limbo which is still my favourite programming language although I haven't written any code in it for something like 10 years!

function sieve(prime, pipeline) function unsieved() while (u = consume(pipeline)) > 0 if mod(u, prime) != 0 produce(u) end end produce(0) # send end of candidates end println(prime) n = 0 for n in consume(pipeline) if mod(n, prime) == 0 break end end if n > 0 # not end of candiates sieve(n, Task(()->unsieved())) end end const LIMIT = 1024 sieve(2, Task(()->(for i in [3:LIMIT; 0] produce(i); end)))

Another server - more courier fun

It's been a while since I did a proper courier installation. The one on CentOS was a botch. I have Debian Jessie in the VM now, phew

Usual stuff #apt-get install postfix courier-imap courier-imap-ssl

The ssl installs an SSL certificate that Thunderbird / Iceweasel doesn't accept because of DH key being too short. And then some other crap. I solved some of it before blogging so I forget what I did to kind of get it working.

# DH_BITS=2048 mkdhparams

Of course, then I got thinking, there are free SSL certificate issuers around, time to see if I can get a proper cert.

So off to https://startssl.com/ and see what to do. Once one has a login - local SSL cert installed in the browser - make sure you extract it (I haven't done that yet so I can't say how).

You will have to validate control of the domain by receiving an email at one of the ones listed as contacts in the WHOIS for the domain (e.g. postmaster@example.com)

Then you can go to the Tool Box and start the process of generating a:

(o) Web Server SSL/TLS Certificate

This takes you to https://startssl.com/Certificates/ApplySSLCert where you can add up to 5 domain names, which should be enough for most private stuff. And a Certificate Request generated with:

# openssl req -newkey rsa:2048 -keyout yourname.key.enc -out yourname.csr

the .key.enc has a keyphrase, you will need to produce a decrypted version

# openssl rsa -in server.key.enc -out server.key

Paste the .csr into the form on the same page and a new SSL cert is generated.

The Certificate List link will give you the wherewithall to download a server.pem file for your certificate. This is not enough to use in courier.

# cat server.key server.pem > /etc/courier/imapd.pem
# /etc/init.d/courier-imap-ssl restart

And that was that, Thunderbird was happy to connect, K9 Mail on my phone needed an "accept certificate" pressing but it accepted it. I guess one can add StartSSL CA certs to these things too.

plan9port on Debian Jessie and now in 2020 Ubuntu / Pop_OS!

Update 26 Aug 2019, I don't know if this is too many now but I added one more library on for it to work. June 2020 - I'm now using Pop_OS! which is Ubuntu based - lots of new GCC warnings

apt-get install build-essential git libx11-dev libfreetype6-dev libfontconfig1-dev libxext-dev libxt-dev

# cd /usr/local
# git clone https://github.com/9fans/plan9port plan9
# cd plan9
# sh INSTALL

Xen Config stuff

This is information I don't want to lose. I got it from http://blog.hostduplex.com/2011/02/24/reconfigure-dns-on-xenserver-5-6/ but it is :

1. Run a console
2. # xe pif-list
   Take note of the uuid of your management interface.
3. # xe pif-reconfigure-ip uuid=youruuid mode=static IP=ipaddresshere netmask=subnetmask gateway=yourgateway DNS=dnsserver1,dnsserver2

Now you'll need some memory. From http://serverfault.com/questions/292560/citrix-xenserver-how-to-add-ram-to-a-guest-vm-without-xencenter

# xe vm-list name-label=NAME params=all |grep memory
...
# xe vm-memory-limits-set dynamic-max=300000000 dynamic-min=300000000 static-max=300000000 static-min=16777216 name-label=Local-PerformanceVM

Connect to your machine # xe console uuid=UUID

Ok The whole shebang

Make the Jessie template
Then name a VM to use it :
# xe vm-install template=Debian\ Jessie\ 8.1\ \(64-bit\) new-name-label=$VMNAME
# xe vm-param-set uuid=$VMUUID other-config:install-repository=http://ftp.uk.debian.org/debian
# xe network-list bridge=xenbr0 --minimal
# xe vif-create vm-uuid=$VMUUID network-uuid=$NETUUID mac=random device=0
# xe vm-memory-limits-set dynamic-max=4000000000 dynamic-min=3000000000 static-max=4000000000 static-min=167772160 name-label=$VMNAME
# xe vm-start name-label=$VMNAME

ucspi-tcp installing on Centos and others

This is the same error
/usr/bin/ld: errno: TLS definition in /lib/libc.so.6 section .tbss mismatches non-TLS reference in envdir.o
as this guy but my solution doesn't need you to edit files manually !

echo '/^gcc s!$! -include /usr/include/errno.h! w q' | ed cc-conf