[rancid] Issue with nx-os after switch upgrades

[John Livingston]

I have a set of MDS switches that yesterday were upgraded from SAN-OS 3.3 to NX-OS 5.2 and now the nxrancid script is failing with an "end of run not found" error message as shown below:

ibmbc1-fc2: missed cmd(s): show running-config
ibmbc1-fc2: missed cmd(s): show running-config
ibmbc1-fc2: End of run not found
ibmbc1-fc2: End of run not found
ibmbc1-fc2: clean: 0, end: 0
!PROC_LOGS: ^

Any recommendations or advise on how to fix this? I have another 10 or so switches to go this weekend and don't want to lose backups on these. Currently running version 2.3.8 of rancid and using the stock nxrancid script.

Thanks,
John

[Alan McKinnon]

On Wed, 4 Mar 2015 17:52:43 +0000

Post by John Livingston
I have a set of MDS switches that yesterday were upgraded from SAN-OS
3.3 to NX-OS 5.2 and now the nxrancid script is failing with an "end
ibmbc1-fc2: missed cmd(s): show running-config
ibmbc1-fc2: missed cmd(s): show running-config
ibmbc1-fc2: End of run not found
ibmbc1-fc2: End of run not found
ibmbc1-fc2: clean: 0, end: 0
!PROC_LOGS: ^
Any recommendations or advise on how to fix this? I have another 10
or so switches to go this weekend and don't want to lose backups on
these. Currently running version 2.3.8 of rancid and using the stock
nxrancid script.

John,

There's no generic catch-all process or fix for this situation, and the
printed output is essentialy useless. All it says is "something went
wrong" but we already know that. Essentially, nxrancid can't make
complete sense of the output and it's not finding where various parts
(individual commands or perhaps everything) begin and end. We need more
information:

nxrancid -d <device>

where <device> is the name in router.db - this will at least show what
commands, if any, are being recognised. Then we will need to dig deeper
and find out why.

Then, one of the first things printed with -d is the full clogin
command used by nxrancid. Run that, deleting out all the commands
except show running-config and see what's going on.

In my experience, I've found things like this:

- Incorrect end-of-line characters at the end of a command output
- No blank line adter the end of a command output when the parser
expects one
- Prompt redefined to something nxrancid can't parse
- Odd characters in the login banner (including the notorious " and >)

Alan

[John Livingston]

Thanks guys! Here is the full output of the nxrancid -d script:


[***@host1 ~]$ cd bin
[***@host1 ~]$ nxrancid -d rv-ibmbc1-fc2
executing clogin -t 90 -c"term no monitor-force;show version;show version build-info all;show license;show license usage;show license host-id;show system redundancy status;show environment clock;show environment fan;show environment fex all fan;show environment temperature;show environment power;show boot;dir bootflash:;dir debug:;dir logflash:;dir slot0:;dir usb1:;dir usb2:;dir volatile:;show module;show module xbar;show inventory;show vtp status;show vlan;show debug;show cores vdc-all;show processes log vdc-all;show running-config" rv-ibmbc1-fc2
PROMPT MATCH: rv-ibmbc1-fc2#
HIT COMMAND:rv-ibmbc1-fc2# term no monitor-force
In RunCommand: rv-ibmbc1-fc2# term no monitor-force
HIT COMMAND:rv-ibmbc1-fc2# show version
In ShowVersion: rv-ibmbc1-fc2# show version
TYPE = NXOS
HIT COMMAND:rv-ibmbc1-fc2# show version build-info all
In ShowVersionBuild: rv-ibmbc1-fc2# show version build-info all
HIT COMMAND:rv-ibmbc1-fc2# show license
In ShowLicense: rv-ibmbc1-fc2# show license
HIT COMMAND:rv-ibmbc1-fc2# show license usage
In ShowLicense: rv-ibmbc1-fc2# show license usage
HIT COMMAND:rv-ibmbc1-fc2# show license host-id
In ShowLicense: rv-ibmbc1-fc2# show license host-id
HIT COMMAND:rv-ibmbc1-fc2# show system redundancy status
In ShowRedundancy: rv-ibmbc1-fc2# show system redundancy status
HIT COMMAND:rv-ibmbc1-fc2# show environment clock
In ShowEnv: rv-ibmbc1-fc2# show environment clock
HIT COMMAND:rv-ibmbc1-fc2# show environment fan
In ShowEnv: rv-ibmbc1-fc2# show environment fan
HIT COMMAND:rv-ibmbc1-fc2# show environment fex all fan
In ShowEnv: rv-ibmbc1-fc2# show environment fex all fan
HIT COMMAND:rv-ibmbc1-fc2# show environment temperature
In ShowEnvTemp: rv-ibmbc1-fc2# show environment temperature
HIT COMMAND:rv-ibmbc1-fc2# show environment power
In ShowEnvPower: rv-ibmbc1-fc2# show environment power
HIT COMMAND:rv-ibmbc1-fc2# show boot
In ShowBoot: rv-ibmbc1-fc2# show boot
HIT COMMAND:rv-ibmbc1-fc2# dir bootflash:
In DirSlotN: rv-ibmbc1-fc2# dir bootflash:
HIT COMMAND:rv-ibmbc1-fc2# dir debug:
In DirSlotN: rv-ibmbc1-fc2# dir debug:
HIT COMMAND:rv-ibmbc1-fc2# dir logflash:
In DirSlotN: rv-ibmbc1-fc2# dir logflash:
HIT COMMAND:rv-ibmbc1-fc2# dir slot0:
In DirSlotN: rv-ibmbc1-fc2# dir slot0:
HIT COMMAND:rv-ibmbc1-fc2# dir usb1:
In DirSlotN: rv-ibmbc1-fc2# dir usb1:
HIT COMMAND:rv-ibmbc1-fc2# dir usb2:
In DirSlotN: rv-ibmbc1-fc2# dir usb2:
HIT COMMAND:rv-ibmbc1-fc2# dir volatile:
In DirSlotN: rv-ibmbc1-fc2# dir volatile:
HIT COMMAND:rv-ibmbc1-fc2# show module
In ShowModule: rv-ibmbc1-fc2# show module
HIT COMMAND:rv-ibmbc1-fc2# show module xbar
In ShowModule: rv-ibmbc1-fc2# show module xbar
HIT COMMAND:rv-ibmbc1-fc2# show inventory
In ShowInventory: rv-ibmbc1-fc2# show inventory
HIT COMMAND:rv-ibmbc1-fc2# show vtp status
In ShowVTP: rv-ibmbc1-fc2# show vtp status
HIT COMMAND:rv-ibmbc1-fc2# show vlan
In ShowVLAN: rv-ibmbc1-fc2# show vlan
HIT COMMAND:rv-ibmbc1-fc2# show debug
In ShowDebug: rv-ibmbc1-fc2# show debug
HIT COMMAND:rv-ibmbc1-fc2# show cores vdc-all
In ShowCores: rv-ibmbc1-fc2# show cores vdc-all
HIT COMMAND:rv-ibmbc1-fc2# show processes log vdc-all
In ShowProcLog: rv-ibmbc1-fc2# show processes log vdc-all
rv-ibmbc1-fc2: show processes log vdc-all failed: -1
rv-ibmbc1-fc2: missed cmd(s): show running-config
rv-ibmbc1-fc2: missed cmd(s): show running-config
rv-ibmbc1-fc2: End of run not found
rv-ibmbc1-fc2: End of run not found
rv-ibmbc1-fc2: clean: 0, end: 0
!PROC_LOGS: ^
[***@host1 ~]$


And here's the output of clogin with just the "show running-config" command:



[***@host1 ~]$clogin -t 90 -c:"show running-config" rv-ibmbc1-fc2
rv-ibmbc1-fc2
spawn ssh -c 3des -x -l user rv-ibmbc1-fc2
MDS Switch
Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
rv-ibmbc1-fc2#
rv-ibmbc1-fc2# terminal length 0
rv-ibmbc1-fc2# terminal width 132
rv-ibmbc1-fc2# terminal-length 0
^
% Invalid command at '^' marker.
rv-ibmbc1-fc2# :show running-config
^
% Invalid command at '^' marker.
rv-ibmbc1-fc2#exit
Connection to rv-ibmbc1-fc2 closed.
[***@host1 ~]$

Any help you can provide is much appreciated.

Thanks,
John

[Dan Anderson]

Sorry. Should have included the diff:

--- nxrancid.orig 2014-09-24 10:02:46.017969251 -0400
+++ nxrancid 2014-09-24 10:03:06.039305060 -0400
@@ -660,7 +660,7 @@
next if (/^(\s*|\s*$cmd\s*)$/);
return(1) if /Line has invalid autocommand /;
return(1) if /(Invalid input detected|Type help or )/;
- return(-1) if (/\% Invalid command at /);
+ return(1) if (/\% Invalid command at /);
return(-1) if (/\% Permission denied/);
return(-1) if (/command authorization failed/i);

I had a similar problem with nxrancid. I ended up modifying the script in
the ShowVersion sub to return(1) for Invalid input detected rather than -1.
The issue is that some of the commands will only show up with certain
hardware ("show vdc all" doesn't work on anything other than a Nexus
7000/7700) or unless certain features have been enabled ("show fex" only
works when "feature fex" has been enabled). It's a hack, but I thought that
it was the easiest solution without creating a command list for every
variable of hardware/configuration.
On Wed, Mar 4, 2015 at 5:08 PM, John Livingston <

Post by John Livingston
executing clogin -t 90 -c"term no monitor-force;show version;show version
build-info all;show license;show license usage;show license host-id;show
system redundancy status;show environment clock;show environment fan;show
environment fex all fan;show environment temperature;show environment
power;show boot;dir bootflash:;dir debug:;dir logflash:;dir slot0:;dir
usb1:;dir usb2:;dir volatile:;show module;show module xbar;show
inventory;show vtp status;show vlan;show debug;show cores vdc-all;show
processes log vdc-all;show running-config" rv-ibmbc1-fc2
PROMPT MATCH: rv-ibmbc1-fc2#
HIT COMMAND:rv-ibmbc1-fc2# term no monitor-force
In RunCommand: rv-ibmbc1-fc2# term no monitor-force
HIT COMMAND:rv-ibmbc1-fc2# show version
In ShowVersion: rv-ibmbc1-fc2# show version
TYPE = NXOS
HIT COMMAND:rv-ibmbc1-fc2# show version build-info all
In ShowVersionBuild: rv-ibmbc1-fc2# show version build-info all
HIT COMMAND:rv-ibmbc1-fc2# show license
In ShowLicense: rv-ibmbc1-fc2# show license
HIT COMMAND:rv-ibmbc1-fc2# show license usage
In ShowLicense: rv-ibmbc1-fc2# show license usage
HIT COMMAND:rv-ibmbc1-fc2# show license host-id
In ShowLicense: rv-ibmbc1-fc2# show license host-id
HIT COMMAND:rv-ibmbc1-fc2# show system redundancy status
In ShowRedundancy: rv-ibmbc1-fc2# show system redundancy status
HIT COMMAND:rv-ibmbc1-fc2# show environment clock
In ShowEnv: rv-ibmbc1-fc2# show environment clock
HIT COMMAND:rv-ibmbc1-fc2# show environment fan
In ShowEnv: rv-ibmbc1-fc2# show environment fan
HIT COMMAND:rv-ibmbc1-fc2# show environment fex all fan
In ShowEnv: rv-ibmbc1-fc2# show environment fex all fan
HIT COMMAND:rv-ibmbc1-fc2# show environment temperature
In ShowEnvTemp: rv-ibmbc1-fc2# show environment temperature
HIT COMMAND:rv-ibmbc1-fc2# show environment power
In ShowEnvPower: rv-ibmbc1-fc2# show environment power
HIT COMMAND:rv-ibmbc1-fc2# show boot
In ShowBoot: rv-ibmbc1-fc2# show boot
HIT COMMAND:rv-ibmbc1-fc2# show module
In ShowModule: rv-ibmbc1-fc2# show module
HIT COMMAND:rv-ibmbc1-fc2# show module xbar
In ShowModule: rv-ibmbc1-fc2# show module xbar
HIT COMMAND:rv-ibmbc1-fc2# show inventory
In ShowInventory: rv-ibmbc1-fc2# show inventory
HIT COMMAND:rv-ibmbc1-fc2# show vtp status
In ShowVTP: rv-ibmbc1-fc2# show vtp status
HIT COMMAND:rv-ibmbc1-fc2# show vlan
In ShowVLAN: rv-ibmbc1-fc2# show vlan
HIT COMMAND:rv-ibmbc1-fc2# show debug
In ShowDebug: rv-ibmbc1-fc2# show debug
HIT COMMAND:rv-ibmbc1-fc2# show cores vdc-all
In ShowCores: rv-ibmbc1-fc2# show cores vdc-all
HIT COMMAND:rv-ibmbc1-fc2# show processes log vdc-all
In ShowProcLog: rv-ibmbc1-fc2# show processes log vdc-all
rv-ibmbc1-fc2: show processes log vdc-all failed: -1
rv-ibmbc1-fc2: missed cmd(s): show running-config
rv-ibmbc1-fc2: missed cmd(s): show running-config
rv-ibmbc1-fc2: End of run not found
rv-ibmbc1-fc2: End of run not found
rv-ibmbc1-fc2: clean: 0, end: 0
!PROC_LOGS: ^
rv-ibmbc1-fc2
spawn ssh -c 3des -x -l user rv-ibmbc1-fc2
MDS Switch
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
rv-ibmbc1-fc2#
rv-ibmbc1-fc2# terminal length 0
rv-ibmbc1-fc2# terminal width 132
rv-ibmbc1-fc2# terminal-length 0
^
% Invalid command at '^' marker.
rv-ibmbc1-fc2# :show running-config
^
% Invalid command at '^' marker.
rv-ibmbc1-fc2#exit
Connection to rv-ibmbc1-fc2 closed.
Any help you can provide is much appreciated.
Thanks,
John
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

--
Dan

[John Livingston]

This post might be inappropriate. Click to display it.

[Alan McKinnon]

On Wed, 4 Mar 2015 17:38:39 -0500

Post by Dan Anderson
--- nxrancid.orig 2014-09-24 10:02:46.017969251 -0400
+++ nxrancid 2014-09-24 10:03:06.039305060 -0400
@@ -660,7 +660,7 @@
next if (/^(\s*|\s*$cmd\s*)$/);
return(1) if /Line has invalid autocommand /;
return(1) if /(Invalid input detected|Type help or )/;
- return(-1) if (/\% Invalid command at /);
+ return(1) if (/\% Invalid command at /);
return(-1) if (/\% Permission denied/);
return(-1) if (/command authorization failed/i);

+1 to this

I recall now doing this for the Nexuses I had back in the day. Solved
many problems just like for John.

I never could understand why Invalid Command would be a fatal error -
if the device can't do it, the script should just move past it and deal
with the output it *can* handle.

Alan

[Jethro R Binks]

Now you mention it, I have done likewise for the comware stuff. In fact I
actually abstracted into subroutines to minimise repetition. Now my
command processing subs all look something like this near the start:

while (<$INPUT>) {
tr/\015//d;
last if(/^\s*$prompt/);
chomp;
$_ = filter_lines($_);

return(1) if command_not_valid($_);
return(-1) if command_not_auth($_);
next if skip_pattern($_);

and then I have:

# Some commands are not supported on some models or versions
# of code.
# Remove the associated error messages, and rancid will ensure that
# these are not treated as "missed" commands
sub command_not_valid {
my ($l) = (@_);

if ( $l =~
/% Too many parameters found at '\^' position/ ||
/% Unrecognized command found at '\^' position/ ||
/% Incomplete command found at '\^' position./ ||
/(% )?Wrong parameter found at '\^' position/ ||
/% Wrong device .+/
) {
return(1);
} else {
return(0);
}
}

# Some commands are not authorized under the current
# user's permissions
sub command_not_auth {
my ($l) = (@_);

if ( $l =~
/Permission denied\./
) {
return(1);
} else {
return(0);
}
}

# Some output lines are always skipped
sub skip_pattern {
my ($l) = (@_);

if ( $l =~
/^\s+\^$/
) {
return(1);
} else {
return(0);
}
}

Post by Alan McKinnon
On Wed, 4 Mar 2015 17:38:39 -0500

Post by Dan Anderson
--- nxrancid.orig 2014-09-24 10:02:46.017969251 -0400
+++ nxrancid 2014-09-24 10:03:06.039305060 -0400
@@ -660,7 +660,7 @@
next if (/^(\s*|\s*$cmd\s*)$/);
return(1) if /Line has invalid autocommand /;
return(1) if /(Invalid input detected|Type help or )/;
- return(-1) if (/\% Invalid command at /);
+ return(1) if (/\% Invalid command at /);
return(-1) if (/\% Permission denied/);
return(-1) if (/command authorization failed/i);

+1 to this
I recall now doing this for the Nexuses I had back in the day. Solved
many problems just like for John.
I never could understand why Invalid Command would be a fatal error -
if the device can't do it, the script should just move past it and deal
with the output it *can* handle.
Alan
--
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

. . . . . . . . . . . . . . . . . . . . . . . . .
Jethro R Binks, Network Manager,
Information Services Directorate, University Of Strathclyde, Glasgow, UK

The University of Strathclyde is a charitable body, registered in
Scotland, number SC015263.

[heasley]

Post by Alan McKinnon
I never could understand why Invalid Command would be a fatal error -
if the device can't do it, the script should just move past it and deal
with the output it *can* handle.

because sometimes the command *should* exist. its has caught o/s bugs in
the past. if it may exist, then we return 1 when it is missing.

[Dan Anderson]

What happens when you run nxrancid -d <device>?

Post by John Livingston
I have a set of MDS switches that yesterday were upgraded from SAN-OS 3.3
to NX-OS 5.2 and now the nxrancid script is failing with an "end of run not
ibmbc1-fc2: missed cmd(s): show running-config
ibmbc1-fc2: missed cmd(s): show running-config
ibmbc1-fc2: End of run not found
ibmbc1-fc2: End of run not found
ibmbc1-fc2: clean: 0, end: 0
!PROC_LOGS: ^
Any recommendations or advise on how to fix this? I have another 10 or so
switches to go this weekend and don't want to lose backups on these.
Currently running version 2.3.8 of rancid and using the stock nxrancid
script.
Thanks,
John
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss