[rancid] Parrallel Execution of Rancid

[Callum Scott]

Hi All,

I've been searching around for an answer to this and can't seem to find
anything.

My organisation currently has a large number of sites with a varying number
of network devices each of which we would like to back up with rancid. Im
using dotwaffle's patched version to include git support, though this
shouldn't make a difference for this issue.

Because I am sorting by SITE I have over 1000 rancid groups. Some have only
5 networking devices, whilst others will have tens of networking devices.

My problem is that the rancid-run is taking a massive amount of time (in
the order of days). I have tried playing around with the PAR_COUNT to
increase concurrency and also reduce the MAX_ROUNDS to reduce the amount of
time spent on each device. It looks to me that the PAR_COUNT only comes in
during the execution of the control_rancid script, which means concurrency
only hits in within the GROUP.

I'd like a way to run rancid_run on the groups in parallel. Am I missing
something obvious here? Does anyone have any ideas on how to achieve this?

Kind Regards
Callum

[Alan McKinnon]

Post by Callum Scott
Hi All,
I've been searching around for an answer to this and can't seem to find
anything.
My organisation currently has a large number of sites with a varying
number of network devices each of which we would like to back up with
rancid. Im using dotwaffle's patched version to include git support,
though this shouldn't make a difference for this issue.
Because I am sorting by SITE I have over 1000 rancid groups. Some have
only 5 networking devices, whilst others will have tens of networking
devices.
My problem is that the rancid-run is taking a massive amount of time (in
the order of days). I have tried playing around with the PAR_COUNT to
increase concurrency and also reduce the MAX_ROUNDS to reduce the amount
of time spent on each device. It looks to me that the PAR_COUNT only
comes in during the execution of the control_rancid script, which means
concurrency only hits in within the GROUP.
I'd like a way to run rancid_run on the groups in parallel. Am I
missing something obvious here? Does anyone have any ideas on how to
achieve this?

Per the man page:

rancid-run <groupname>

You could organize your crontabs so that several instances of rancid-run
run in parallel, each with their own unique list of groups that are
handled in series. With 1000 groups this could get interesting.

Or you could write a wrapper script that tries to do something
intelligent. The main information you need in LIST_OF_GROUPS in
rancid.conf and the number of up devices in each router.db

AFAIK such a script isn't out there yet.

Also, have you considered consolidating your devices somehow to reduce
the number of groups? I don't know how you separate things out but if
you are doing it by physical location, would it not work if you grouped
them by town, state or even continent?