Multicore Shell Scripting

Processing lots of files at once in a process level embarassingly parallel way is v. handy. I've been meaning to get round to doing this shell script spawner that will use a fixed number of processes launching, by default, one process for each CPU core the OS reports.

It reads one line at a time and executes it until N are running then it waits until they have all finished and then spawns N more.

It's the most horrible way of doing it from an I/O point of view, hotspot bonanza!!

So maybe it's best to echo 'sleep 2; gm mogrify -resize 5000x3000 foo.jpg bar.png'
I don't know, we'll see

#!/usr/local/plan9/bin/rc

fn os_cpus { # platform dependent, number of cpus, counting from 1
        # Linux
        grep '^processor' /proc/cpuinfo |
                tail -n 1 |
                sed 's/[^0-9]//g' |
                awk '/./ {print $0 + 1; exit} {print "1"}'
}

fn cpus {
        if(~ $#1 0)
                os_cpus # default
        if not
                echo $1
}

cpuN = `{cpus $1}
while() {
        for(i in `{seq 1 $cpuN}) {
                if(cmd = `{read}) { # brace required :) clever rc parser
                        $cmd &
                }
                if not
                        kill $pid
        }
        wait
}

I just ran this

{ for(f in f*png) echo gm convert $f -interlace None i^$f } | conquer

Worked a treat