[rancid] Repetitive RANCID Checks

[Mark Tinka]

Hi all.

We upgraded to Perl-5.18 today on FreeBSD-10, and rebuilt
all ports that depend on Perl, including RANCID.

Since then, RANCID has been continuously e-mailing the same
changes at every interval, even though those changes have
not been actually effected.

The changes are essentially re-ordered prefix list entries
for Cisco IOS-based systems.

Anyone know what this is, before I go hacking at this?

RANCID version is 2.3.8. Thanks.

Cheers,

Mark.

[Alan McKinnon]

Post by Mark Tinka
Hi all.
We upgraded to Perl-5.18 today on FreeBSD-10, and rebuilt
all ports that depend on Perl, including RANCID.
Since then, RANCID has been continuously e-mailing the same
changes at every interval, even though those changes have
not been actually effected.
The changes are essentially re-ordered prefix list entries
for Cisco IOS-based systems.
Anyone know what this is, before I go hacking at this?
RANCID version is 2.3.8. Thanks.
Cheers,
Mark.

There's a setting for this in rancid.conf:

SORTACL

what is yours set to? The upgrade may have overwritten your config file.

[Mark Tinka]

Post by Alan McKinnon
SORTACL
what is yours set to?

# if ACLSORT is NO, access-lists will NOT be sorted.
#ACLSORT=YES; export ACLSORT

As you can see, currently commented out.

My guess is uncommenting this and toggling between "YES" and
"NO" is worth a try, non?

Post by Alan McKinnon
The upgrade may have overwritten
your config file.

I only upgraded Perl. I've been running RANCID-2.3.8 for a
long time now, and haven't had this issue. Would Perl do
anything to RANCID?

Cheers,

Mark.

[Alan McKinnon]

Post by Mark Tinka

Post by Alan McKinnon
SORTACL
what is yours set to?

# if ACLSORT is NO, access-lists will NOT be sorted.
#ACLSORT=YES; export ACLSORT
As you can see, currently commented out.
My guess is uncommenting this and toggling between "YES" and
"NO" is worth a try, non?

As shipped, the commented value is the default. Toggling it is worth a
try, but is not likely to change anything for you as you haven't changed it

Post by Mark Tinka

Post by Alan McKinnon
The upgrade may have overwritten
your config file.

I only upgraded Perl. I've been running RANCID-2.3.8 for a
long time now, and haven't had this issue. Would Perl do
anything to RANCID?

Ah, my mistake. I read your mail as saying you'd also upgraded rancid.
Perl would only change rancid's behaviour if perl features that rancid
uses now behave differently.

A few routine questions?

Is there anything in your logs about this?

The rancid code that deals with prefix lists is

/^ip(v6)?
prefix-list\s+(\S+)\s+seq\s+(\d+)\s+(permit|deny)\s+(\S+)(\/.*)$/
&& ProcessHistory("PACL $2 $4","$aclsort","$5",
"ip$1 prefix-list $2 $4 $5$6\n")
&& next;

Is yours still unchanged?

if you run "perl -cw <filename>" o each perl script in your rancid bin/
directory, does perlc omplain about any of them?

[Mark Tinka]

Post by Alan McKinnon
As shipped, the commented value is the default. Toggling
it is worth a try, but is not likely to change anything
for you as you haven't changed it

So I initially set "ACLSORT=YES" and that didn't help.

Toggling it to "ACLSORT=NO" seems to be helping. I haven't
received another annoying update in almost 3hrs.

Post by Alan McKinnon
Ah, my mistake. I read your mail as saying you'd also
upgraded rancid. Perl would only change rancid's
behaviour if perl features that rancid uses now behave
differently.

That's my thinking.

Post by Alan McKinnon
Is there anything in your logs about this?

I didn't find anything interesting when I checked yesterday,
when it began.

Post by Alan McKinnon
The rancid code that deals with prefix lists is
/^ip(v6)?
prefix-list\s+(\S+)\s+seq\s+(\d+)\s+(permit|deny)\s+(\S+)
(\/.*)$/ && ProcessHistory("PACL $2 $4","$aclsort","$5",
"ip$1 prefix-list $2 $4 $5$6\n") && next;
Is yours still unchanged?

Yes, this still looks good.

Post by Alan McKinnon
if you run "perl -cw <filename>" o each perl script in
your rancid bin/ directory, does perlc omplain about any
of them?

I'll try this if I get another update. But nothing in 3hrs,
so it's looking better than the last 24hrs.

Mark.

[Alan McKinnon]

Post by Mark Tinka

Post by Alan McKinnon
As shipped, the commented value is the default. Toggling
it is worth a try, but is not likely to change anything
for you as you haven't changed it

So I initially set "ACLSORT=YES" and that didn't help.
Toggling it to "ACLSORT=NO" seems to be helping. I haven't
received another annoying update in almost 3hrs.

That's interesting because it's the wrong way round :-)

My gut feel hunch is that sorting inside ProcessHistory() isn't working
properly since the perl upgrade; the code is randomly reordering things
whereas what's in show run is already consistent.

However I don't see anything in that sub or the sorting routines that
5.18 would complain about.

We'd probably need a sample of your show run to properly debug further
as this looks awfully like a data issue

[Mark Tinka]

Post by Alan McKinnon
We'd probably need a sample of your show run to properly
debug further as this looks awfully like a data issue

Do you want to see the RANCID diffs against what is actually
on the device for the affected section of the configuration?

Mark.

[Alan McKinnon]

Post by Mark Tinka
Hi all.
We upgraded to Perl-5.18 today on FreeBSD-10, and rebuilt
all ports that depend on Perl, including RANCID.
Since then, RANCID has been continuously e-mailing the same
changes at every interval, even though those changes have
not been actually effected.
The changes are essentially re-ordered prefix list entries
for Cisco IOS-based systems.
Anyone know what this is, before I go hacking at this?
RANCID version is 2.3.8. Thanks.

I see what the bug is here, the solution probably needs list discussion
first.

With rancid-2.3.8 and perl-5.18, ipv6 prefix-lists are always output in
random order if ACLSORT=yes

The cause is twofold:

1. "sub ipaddrval" is ipv4-specific and cannot deal with ipv6 addresses
properly, so sortbyipaddr() doesn't work as intended. I believe that
%history in ProcessHistory() is not being sorted at all

2. A recent perl version re-implemented hashes to guarantee that keys
are randomly ordered internally and different between runs of the same
script. The previous implementation would order the same data
consistently between invocations.

So ipv6 prefix-list ordering just happened to work with <perl-5.18
because %history just happened to be consistently ordered.

Solutions:

1. Implement ipv6 sorting in sortbyipaddr(). I really want to find an
excuse not to do this (being essentially a lazy bum at heart)

2. Why are we sorting these lists by ip address? Surely sorting 1) by
prefix name then 2) by seq makes more sense? This is what my NetOps
colleagues would like to see happen.

[Mark Tinka]

Post by Alan McKinnon
2. Why are we sorting these lists by ip address? Surely
sorting 1) by prefix name then 2) by seq makes more
sense? This is what my NetOps colleagues would like to
see happen.

Well, yes - sorting by prefix name and sequence number would
make more sense to me, as an operator.

That said, I've always noticed that RANCID does not include
the sequence number of prefix lists in the final output that
gets saved and/or sent out via e-mails. Is this something
that is problematic to implement? Thanks.

Cheers,

Mark.

[heasley]

Post by Alan McKinnon
I see what the bug is here, the solution probably needs list discussion
first.
With rancid-2.3.8 and perl-5.18, ipv6 prefix-lists are always output in
random order if ACLSORT=yes
1. "sub ipaddrval" is ipv4-specific and cannot deal with ipv6 addresses
properly, so sortbyipaddr() doesn't work as intended. I believe that
%history in ProcessHistory() is not being sorted at all
2. A recent perl version re-implemented hashes to guarantee that keys
are randomly ordered internally and different between runs of the same
script. The previous implementation would order the same data
consistently between invocations.

bugger. thanks for that. another user was asking about this problem and
i was having trouble recreating it.

Post by Alan McKinnon
So ipv6 prefix-list ordering just happened to work with <perl-5.18
because %history just happened to be consistently ordered.
1. Implement ipv6 sorting in sortbyipaddr(). I really want to find an
excuse not to do this (being essentially a lazy bum at heart)
2. Why are we sorting these lists by ip address? Surely sorting 1) by
prefix name then 2) by seq makes more sense? This is what my NetOps
colleagues would like to see happen.

the sequences just tend to create noise, esp. when the whole list is
reloaded (therefore renumbered), and hides the actual change or makes it
less obvious.

[Mark Tinka]

Post by heasley
the sequences just tend to create noise, esp. when the
whole list is reloaded (therefore renumbered), and hides
the actual change or makes it less obvious.

Could you kindly elaborate, Heasley? Thanks.

Mark.

[Alan McKinnon]

Post by heasley

Post by Alan McKinnon
I see what the bug is here, the solution probably needs list discussion
first.
With rancid-2.3.8 and perl-5.18, ipv6 prefix-lists are always output in
random order if ACLSORT=yes
1. "sub ipaddrval" is ipv4-specific and cannot deal with ipv6 addresses
properly, so sortbyipaddr() doesn't work as intended. I believe that
%history in ProcessHistory() is not being sorted at all
2. A recent perl version re-implemented hashes to guarantee that keys
are randomly ordered internally and different between runs of the same
script. The previous implementation would order the same data
consistently between invocations.

bugger. thanks for that. another user was asking about this problem and
i was having trouble recreating it.

To recreate it reliably, have a sort function (the ones with automagic
$a and $b) return a 0.

This never happens with sortbyipaddr and ipv4 addresses as they are
always different and you always get a reliable sort

Post by heasley

Post by Alan McKinnon
So ipv6 prefix-list ordering just happened to work with <perl-5.18
because %history just happened to be consistently ordered.
1. Implement ipv6 sorting in sortbyipaddr(). I really want to find an
excuse not to do this (being essentially a lazy bum at heart)
2. Why are we sorting these lists by ip address? Surely sorting 1) by
prefix name then 2) by seq makes more sense? This is what my NetOps
colleagues would like to see happen.

the sequences just tend to create noise, esp. when the whole list is
reloaded (therefore renumbered), and hides the actual change or makes it
less obvious.

I think we should document more explicitly what ACLSORT achieves. I was
under the impression for years that it ensures lists are sorted in
evaluation order if show run were to display them mixed them up a bit.
Then I read the code in detail :-)

When does a list reload occur? My NetOps colleagues maintain it only
happens when the router itself is rebooted, and they don't care about
that (the diff after a reboot is already quite large)

[Mark Tinka]

Post by Alan McKinnon
When does a list reload occur? My NetOps colleagues
maintain it only happens when the router itself is
rebooted, and they don't care about that (the diff after
a reboot is already quite large)

That's right - in the context of RANCID, prefix lists are
not constantly reloaded while the router is running, only
after a reboot of the device occurs.

Mark.

[Alan McKinnon]

Post by Mark Tinka

Post by Alan McKinnon
When does a list reload occur? My NetOps colleagues
maintain it only happens when the router itself is
rebooted, and they don't care about that (the diff after
a reboot is already quite large)

That's right - in the context of RANCID, prefix lists are
not constantly reloaded while the router is running, only
after a reboot of the device occurs.
Mark.

Thanks for that confirmation, I suspected as such