tenchiken

Mine would be when I had a 48 bay disk array / JBOD fail on me… badly. After a storm, it killed the larger card that allowed for me to get many of the drives into a PCIe 16x slot, and I was relegated down to only getting ~8 disks per box made from spare hardware. A single box I got 16 going. Add to this mix an SSD for mid-line caching.

These were all running bcache on top of mdraid… One single mount.

Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.

I just HAD to solve this one though.

In come several spare chassis / mobo etc… get a bunch of drives powered and on /dev/ , move to the next.

A couple spare gigabit switches…

several gigabit NICs…

two explicit paths for each machine…

a bit of iSCSI magic, and one machine now had the physical disks all exposed to it… mdadm --assemble blah blah, bit of UUID chaos…

It’s surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.

Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of “I can lose this and don’t care” vs “this box gets RAID10 and/or offsite backups nightly”.

back up your critical stuff people!

Mitch McConnell announces IS discharge from rehabilitation center

I’ll take 2 of em.

Replace the guy with just moneybags on their way to get emptied.

root@kamidake:~# for i in /dev/sd? ; do echo $i; smartctl -a $i |grep Hours| awk ’ { print $10/8760}'; done

/dev/sda

15.6055

/dev/sdb

15.6979

/dev/sdc

6.04909

/dev/sdd

7.35708

/dev/sde

10.618

/dev/sdf

7.28071

/dev/sdg

10.5689

/dev/sdh

10.6264

/dev/sdi

2.28311

/dev/sdj

10.7498

/dev/sdk

10.5832

/dev/sdl

7.35502

/dev/sdm

10.643

/dev/sdn

10.8016

/dev/sdo

10.7558

/dev/sdp

9.11416

root@kamidake:~#

These are running ZFS and I have 2 other boxes with similar, most in the 10-12 year range. Spinning drives.

Good stuff.

lol.

lmfao even.

I’m the admin of one of the blocked instances. Yay!