[rancid] How to implement a diff filter?

[James Bensley]

Hi Listee's,

I have a problem which I'm sure is common, I wish to filter out some
email updates from Rancid.

Here are two very common examples for me of diff's that are emailed to
me regularly, that I really don't need to know about :)

[1] http://pastebin.com/raw.php?i=8KviGwAb
[2] http://pastebin.com/raw.php?i=Ff38Gpvp

What I would like is to understand the order in which the various
parts of Rancid run when indexing Cisco devices, so that I can
interrupt it.
If we suppose updates happen as follows;
[1] Connect to each device (one at a time) and execute the serious of
enumerating commands preconfigured in the "@CommandTable"
[2] Diff the current details with the last retrieved set, and if
differences exist, commit as a new entry to the archive
[3] After looping through all devices, email out the diff's for any
devices who's archive revision has updated as a combined listing

If that were how Rancid runs, I'd ideally like to insert a call to a
script in section 3, which would parse the diff to my script. My
script would look at a folder I have populated with diff's like those
above, I don't want to be notified about, and not parse them onto to
the final email that Rancid sends out when a match is found.

Has anyone done this already perhaps? Ideally, I would like to
understand more about this order of events, which scripts and binaries
are performing which tasks, so I can implement such a feature for my
self.

Thanks all for reading and I appreciate any feed back any one can give.

Cheers,

[heasley]

Post by James Bensley
Hi Listee's,
I have a problem which I'm sure is common, I wish to filter out some
email updates from Rancid.
Here are two very common examples for me of diff's that are emailed to
me regularly, that I really don't need to know about :)
[1] http://pastebin.com/raw.php?i=8KviGwAb
[2] http://pastebin.com/raw.php?i=Ff38Gpvp
What I would like is to understand the order in which the various
parts of Rancid run when indexing Cisco devices, so that I can
interrupt it.

control_rancid runs for each group
rancid script
execs login script
control_rancid diffs & checks in changes

you have to change the rancid script.

rancid-3.0 will have a manner to add/remove commands you do/dont want to run,
define your own device types, or add your own filters without altering the
base stuff.

Post by James Bensley
If we suppose updates happen as follows;
[1] Connect to each device (one at a time) and execute the serious of
[2] Diff the current details with the last retrieved set, and if
differences exist, commit as a new entry to the archive
[3] After looping through all devices, email out the diff's for any
devices who's archive revision has updated as a combined listing
If that were how Rancid runs, I'd ideally like to insert a call to a
script in section 3, which would parse the diff to my script. My
script would look at a folder I have populated with diff's like those
above, I don't want to be notified about, and not parse them onto to
the final email that Rancid sends out when a match is found.
Has anyone done this already perhaps? Ideally, I would like to
understand more about this order of events, which scripts and binaries
are performing which tasks, so I can implement such a feature for my
self.
Thanks all for reading and I appreciate any feed back any one can give.
Cheers,
--
James.
http://www.jamesbensley.co.cc/
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

[j***@crc.u-strasbg.fr]

On Fri, Feb 17, 2012 at 08:37:29PM +0000, James Bensley wrote:
[...]

Post by James Bensley
If that were how Rancid runs, I'd ideally like to insert a call to a
script in section 3, which would parse the diff to my script. My
script would look at a folder I have populated with diff's like those
above, I don't want to be notified about, and not parse them onto to
the final email that Rancid sends out when a match is found.
Has anyone done this already perhaps?

We do this by redefining PATH to intercept the call to sendmail
in rancid-run :

export PATH=/my/dir:$PATH
rancid-run

A script named sendmail in /my/dir reads stdin and could apply any changes
to the message: add or remove some lines, send an email or not etc.

See:

https://github.com/pdav/netmagis/blob/master/topo/src/start-rancid
https://github.com/pdav/netmagis/blob/master/topo/src/sendmail

--
Jean

[Lee]

Post by j***@crc.u-strasbg.fr
[...]

Post by James Bensley
If that were how Rancid runs, I'd ideally like to insert a call to a
script in section 3, which would parse the diff to my script. My
script would look at a folder I have populated with diff's like those
above, I don't want to be notified about, and not parse them onto to
the final email that Rancid sends out when a match is found.
Has anyone done this already perhaps?

We do this by redefining PATH to intercept the call to sendmail
export PATH=/my/dir:$PATH
rancid-run
A script named sendmail in /my/dir reads stdin and could apply any changes
to the message: add or remove some lines, send an email or not etc.

I like having everything in the saved configs, but don't like mailing
out diffs with sensitive information (eg. lines containing " password
7 ") so I added

~/bin/sanitize.sh $TMP.diff >$TMP.diff2
/bin/mv $TMP.diff2 $TMP.diff

to control_rancid just before where it mails out the diffs.

If enough people are modifying the diff output it might be worthwhile
to add something like

PRE_EMAIL_SCRIPT=""

in /etc/rancid.conf and in bin/control_rancid add

if [ "X$PRE_EMAIL_SCRIPT" != "X" -a -x $PRE_EMAIL_SCRIPT ]; then
export MAILFILE="$TMP.diff"
$PRE_EMAIL_SCRIPT
fi

just before
# Mail out the diffs (if there are any).

Lee

[James Bensley]

Better late than never ;)

Thanks to everyone for your input on this, the info provided was
exactly the direction I needed.

I have implemented a diff filter now. It's very basic but that's all
it needed to be. I wanted my rancid update emails to change from this
[1], to this [2]. So any section where there wasn't a configuration
change, don't tell me about it (although do commit it to the
repository).

The simple pattern here is that updates without config changes have
the same number of line start with "-" and "- !" and the same number
of lines starting with "+" and "+ !". The script simple uses sed to
cut out sections from the email update diff file that match this
simple pattern. The script is here;
http://null.53bits.co.uk/index.php?page=rancid-filter

As pointed out, you need to make a minor modification to
control_rancid in bin/ directory to call the filter script. Thanks
again everyone, you have aided me in the never ending battle of
reducing the number of emails I receive!

Cheers,
James.

[1] http://pastebin.com/raw.php?i=hJX2mrmX
[2] http://pastebin.com/raw.php?i=FMY5C6dp