[Cialug] Looking for a simple parallelization command

[Thomas Kula]

On Tue, Nov 11, 2008 at 10:03:43AM -0800, John Lengeling wrote:
> I have a script which sequentially performs some operations across
> multiple databases.   Since this is running on a 32 core server, I would
> like to investigate running a number of these operations in parallel.
> Are there any simple utilities out there which when given a list of
> commands will run them parallel?

It's not necessarily a simple utility, but our AFS backups here 
are driven by that old warhorse, make, which allows us to do some
parts in parallel. If you have a fixed number of databases (or a
number of databases that doesn't change all that often) you could
do something like this in a Makefile:

default:
	make -j 4 jobs

jobs: job1 job2 job3 job4 job5 ...

job1:
	command to run against db 1

job2:
	command to run against db 2

...

Make handles having N jobs running (in this example, N happens to
be 4) in parallel, and starting a new job when one has finished.
Someone who is better than I am with Makefiles can probably make
that somewhat generic, even.


-- 
Thomas L. Kula | kula at tproa.net | http://kula.tproa.net/
Mathom House in Midtown, The People's Republic of Ames