Brian McCallister: "So, ruby threads have issues. Multiprocessing, on the other hand, works great."

This is the same basic structure the Odeo crawler uses (though thats a 3-tier system). The wisdom of Unix rears its hoary head once again. (and dRB is just neat!)