[rancid] sorting output from custom commands...

[Jon Peatfield]

I added a quick hack to our rancid (2.3.4) setup to record the output of
running:

show cdp neighbors detail

to hrancid and rancid (we only have Cisco and HP procurve switches).

On the newer versions of the HP porocurve firmware (all 2610-xx models)
each (cisco 79xx) phone device shows up as two device-ids, and it appears
that every time a handset is connected those devices appear in a random
order.

So at the moment we get spurious change messages swapping the order of the
two entries (both on the same port).

I'm not very familiar with how ProcessHistory works, so I don't know if
there is a trick I can use to get it to do the sorting for me. I did
already write a piece of code to store the data in a hash by device-id and
only call ProcessHistory when we notice that the port in the output has
changed, but I lost that by accidentally zapping the directory I was
testing in (oops). I can re-do that work easily enough but since other
stuff has to handle output in different orders I wonder if I'm wasting my
time...

-- Jon

[john heasley]

Post by Jon Peatfield
I added a quick hack to our rancid (2.3.4) setup to record the output of
show cdp neighbors detail
to hrancid and rancid (we only have Cisco and HP procurve switches).
On the newer versions of the HP porocurve firmware (all 2610-xx models)
each (cisco 79xx) phone device shows up as two device-ids, and it appears
that every time a handset is connected those devices appear in a random
order.
So at the moment we get spurious change messages swapping the order of
the two entries (both on the same port).
I'm not very familiar with how ProcessHistory works, so I don't know if
there is a trick I can use to get it to do the sorting for me. I did
already write a piece of code to store the data in a hash by device-id
and only call ProcessHistory when we notice that the port in the output
has changed, but I lost that by accidentally zapping the directory I was
testing in (oops). I can re-do that work easily enough but since other
stuff has to handle output in different orders I wonder if I'm wasting my
time...

ProcessHistory(tag, command, command_arg, data)

data is added the buffer, by command with command_arg (if specified),
unless tag or command has changed since the last call, in which case
the buffer would be flushed before calling command.

a good example, where $aclsort = "ipsort" (the function)

# order access-lists
/^access-list\s+(\d\d?)\s+(\S+)\s+(\S+)/ &&
ProcessHistory("ACL $1 $2","$aclsort","$3","$_") && next;

[Jon Peatfield]

Post by john heasley
ProcessHistory(tag, command, command_arg, data)
data is added the buffer, by command with command_arg (if specified),
unless tag or command has changed since the last call, in which case
the buffer would be flushed before calling command.
a good example, where $aclsort = "ipsort" (the function)
# order access-lists
/^access-list\s+(\d\d?)\s+(\S+)\s+(\S+)/ &&
ProcessHistory("ACL $1 $2","$aclsort","$3","$_") && next;

So since the output is multi-line per section/device I guess I still need
to collect together the lines for each device-id so we can pass that in
the string that the sort function uses...

I'm including a few some bits of output from a switch:

fl10-v2# show cdp nei det

CDP neighbors information

Port : 1
Device ID : SEP001FCA369E19
Address Type : IP
Address : 172.18.18.179
Platform : SIP11.8-5-3SCisco IP Phone 7911
Capability :
Device Port : Port 1
Version : SIP11.8-5-3SCisco IP Phone 7911

------------------------------------------------------------------------------

Port : 1
Device ID : 01 ac 12 12 b3
Address Type : IP
Address : 172.18.18.179
Platform : Cisco IP Phone 7911G,V5, SIP11.8-5-3S
Capability : Switch
Device Port : 001FCA369E19:P1
Version : Cisco IP Phone 7911G,V5, SIP11.8-5-3S

------------------------------------------------------------------------------

Port : 2
Device ID : SEP001FCA369FE8
Address Type : IP
Address : 172.18.18.181
Platform : SIP11.8-5-3SCisco IP Phone 7911
Capability :
Device Port : Port 1
Version : SIP11.8-5-3SCisco IP Phone 7911

------------------------------------------------------------------------------

Port : 2
Device ID : 01 ac 12 12 b5
Address Type : IP
Address : 172.18.18.181
Platform : Cisco IP Phone 7911G,V5, SIP11.8-5-3S
Capability : Switch
Device Port : 001FCA369FE8:P1
Version : Cisco IP Phone 7911G,V5, SIP11.8-5-3S
...


For each block of lines corresponding to a device I need to call
ProcessHistory with the device-id as something to sort on...

Or in code something like:

my $currport=' ';
my $deviceid='';
my $cdpdblock='';
while (<INPUT>) {

... usual-tests and skip junk...

if (/^\s*Port\s*:\s*(.*)/) {
my $newport=$1;
# send any previous device-block
ProcessHistory("COMMENTS$currport","keysort","IO $deviceid",$cdpdblock);
$currport=$newport;
$cdpdblock='';
$deviceid='';
} elsif (/^\s*Device ID\s*:\s*(.*)/) {
$deviceid=$1;
}
# Accumulate all the lines into the current block...
print STDERR " Adding to cdp block (port=$currport dev=$deviceid): $_\n";
$cdpdblock.=";CDP: $_\n";
}

# And deal with the last block...
$cdpdblock.=";\n";
ProcessHistory("COMMENTS$currport","keysort","IO $deviceid",$cdpdblock);
...

which seems to work - at least for my simple tests.

I may have it use "keyrsort" (like keysort but backwards) simply because
that seems to be closer to the unsorted output I get from more of our
switches, or take the bigger hit once.

Many thanks.

[john heasley]

Post by Jon Peatfield
CDP neighbors information
Port : 1
Device ID : SEP001FCA369E19
Address Type : IP
Address : 172.18.18.179
Platform : SIP11.8-5-3SCisco IP Phone 7911
Device Port : Port 1
Version : SIP11.8-5-3SCisco IP Phone 7911
------------------------------------------------------------------------------
Port : 1
Device ID : 01 ac 12 12 b3
Address Type : IP
Address : 172.18.18.179
Platform : Cisco IP Phone 7911G,V5, SIP11.8-5-3S
Capability : Switch
Device Port : 001FCA369E19:P1
Version : Cisco IP Phone 7911G,V5, SIP11.8-5-3S

though this looks legitimate; some of the phones have two device. i'd
keep track of the port's seen (or just last port) and skip duplicates.

Post by Jon Peatfield
if (/^\s*Port\s*:\s*(.*)/) {
my $newport=$1;

if port == lastport
read until ^$
else
read, processhistory, until ^$

[Jon Peatfield]

Post by john heasley

Post by Jon Peatfield
CDP neighbors information
Port : 1
Device ID : SEP001FCA369E19
Address Type : IP
Address : 172.18.18.179
Platform : SIP11.8-5-3SCisco IP Phone 7911
Device Port : Port 1
Version : SIP11.8-5-3SCisco IP Phone 7911
------------------------------------------------------------------------------
Port : 1
Device ID : 01 ac 12 12 b3
Address Type : IP
Address : 172.18.18.179
Platform : Cisco IP Phone 7911G,V5, SIP11.8-5-3S
Capability : Switch
Device Port : 001FCA369E19:P1
Version : Cisco IP Phone 7911G,V5, SIP11.8-5-3S

though this looks legitimate; some of the phones have two device. i'd
keep track of the port's seen (or just last port) and skip duplicates.

Post by Jon Peatfield
if (/^\s*Port\s*:\s*(.*)/) {
my $newport=$1;

if port == lastport
read until ^$
else
read, processhistory, until ^$

I may be misunderstanding you, but that sounds like you are suggesting
skipping all but the first device-id for each port.

Since the multiple device-ids on a port are listed in an apparently random
order (but the order stays until the device is unplugged or reset), just
reporting the first device on a port will cause reporting flaps (just like
I was getting by not sorting them).

The current code I'm testing seems to give the answers I want though it
does assume that all devices for port P will be consecutive in the
output...

There is another thing I do plan to fix which is that for a few seconds
after a handset is plugged in we get a 3rd device-id reported - apparently
a fixed device-id for all handsets. Presumably this is a quirk of the
data that these Cisco handsets send in their first cdp frame, or the HP
not understanding something.

I plan to simply filter those out since the information isn't useful to me
and we only see them if rancid happens to poll the switch right after a
handset is connected.

I'll be happy to contribute what I end up with if anyone else wants this.

-- Jon

[Jethro R Binks]

Post by Jon Peatfield
I plan to simply filter those out since the information isn't useful to
me and we only see them if rancid happens to poll the switch right after
a handset is connected.
I'll be happy to contribute what I end up with if anyone else wants this.

I'm interested, although working in a different environment (H3C/3Com and
Mitel handsets).

Also, I think your discussions about sorting some of the lines in the
command output helps me with something else I was idling around with a
while ago, so I've saved that and will take another look with the comments
in mind. I never did really understand properly what ProcessHistory does,
behyond the very basic, so that brief explanation from John was useful in
itself.

Jethro.

. . . . . . . . . . . . . . . . . . . . . . . . .
Jethro R Binks, Computing Officer
Information Services, The University Of Strathclyde, Glasgow, UK

The University of Strathclyde is a charitable body, registered in
Scotland, number SC015263.

[john heasley]

Post by Jon Peatfield

Post by john heasley

Post by Jon Peatfield
CDP neighbors information
Port : 1
Device ID : SEP001FCA369E19
Address Type : IP
Address : 172.18.18.179
Platform : SIP11.8-5-3SCisco IP Phone 7911
Device Port : Port 1
Version : SIP11.8-5-3SCisco IP Phone 7911
------------------------------------------------------------------------------
Port : 1
Device ID : 01 ac 12 12 b3
Address Type : IP
Address : 172.18.18.179
Platform : Cisco IP Phone 7911G,V5, SIP11.8-5-3S
Capability : Switch
Device Port : 001FCA369E19:P1
Version : Cisco IP Phone 7911G,V5, SIP11.8-5-3S

though this looks legitimate; some of the phones have two device. i'd
keep track of the port's seen (or just last port) and skip duplicates.

Post by Jon Peatfield
if (/^\s*Port\s*:\s*(.*)/) {
my $newport=$1;

if port == lastport
read until ^$
else
read, processhistory, until ^$

I may be misunderstanding you, but that sounds like you are suggesting
skipping all but the first device-id for each port.

yes

Post by Jon Peatfield
Since the multiple device-ids on a port are listed in an apparently
random order (but the order stays until the device is unplugged or
reset), just reporting the first device on a port will cause reporting
flaps (just like I was getting by not sorting them).

missed that. what you had is appropriate, or use

processhistory("cdp", "keysort", "port $port $capability", data)
or
processhistory("cdp", "keysort", "port $port $deviceid", data)

and let keysort sort the entries.

[Jon Peatfield]

On Wed, 29 Sep 2010, Jon Peatfield wrote:

...

The current code I'm testing seems to give the answers I want though it does
assume that all devices for port P will be consecutive in the output...

...

I'll be happy to contribute what I end up with if anyone else wants this.

Some time has passed and I realized yesterday that I'd not actually sent
the version of the patches that we have been using.

So here they are, the first is very simple and is just for cisco boxes
where we simply have to remove the holdtime from the output, and the
second is for HP procurve and is slightly more complex since it does the
sorting or the device-ids shown on each port.

Note that this is running 'show cdp neighbors detail' so the output can be
quite long and you may well get lots of flapping if devices come/go so I'm
sure that this isn't for everyone!

Hopefully they might be of use to at least someone.

Although these patches say 2.3.4 in the filenames I've tested that they
apply cleanly to 2.3.6 and since yesterday that is what we are now using.

-- Jon