[rancid] Juniper CLI prompts out of sync causing frequent changes

[Chris Wopat]

Hi folks,

Last year I commented on an issue we're seeing across many Juniper devices.
I neglected to follow up on Heasley's response then but are seeing it a lot
more frequently now, perhaps related to some OS upgrades or something else.


Thread was here:

http://www.shrubbery.net/pipermail/rancid-discuss/2017-October/009916.html
http://www.shrubbery.net/pipermail/rancid-discuss/2017-October/009922.html

Looking at the last week or so of these we've had, they're on devices
running 14.1X53-D4*, which is primaraily QFX5100 but also a few EX4200.

Here's output from a single diff, its like this on various commands nearly
every run:



Index: configs/r-kettlemoraine-hub
===================================================================
retrieving revision 1.144
diff -u -4 -r1.144 r-kettlemoraine-hub
@@ -1,7 +1,8 @@
#RANCID-CONTENT-TYPE: juniper
#
# r-kettlemoraine-hub> show chassis clocks
+ # show chassis environment
# r-kettlemoraine-hub> show chassis environment
# Class Item Status
# Power FPC 0 Power Supply 0 OK
# FPC 0 Power Supply 1 OK
Index: configs/r-lacrossecity-hub
===================================================================
retrieving revision 1.108
diff -u -4 -r1.108 r-lacrossecity-hub
@@ -15,9 +15,8 @@
# FPC 0 Fan 2 OK
# FPC 0 Fan 3 OK
#
# r-lacrossecity-hub> show chassis firmware
- # show chassis fpc detail
# Part Type Version
# FPC 0 uboot U-Boot 1.1.6 (Jun 5 2012 -
02:24:53) 1.0.0
# loader FreeBSD/PowerPC U-Boot bootstrap
loader 2.4
#
Index: configs/r-platteville-hub
===================================================================
retrieving revision 1.274
diff -u -4 -r1.274 r-platteville-hub
@@ -1,7 +1,8 @@
#RANCID-CONTENT-TYPE: juniper
#
# r-platteville-hub> show chassis clocks
+ # show chassis environment
# r-platteville-hub> show chassis environment
# Class Item Status
# Power FPC 0 Power Supply 0 OK
# FPC 0 Power Supply 1 OK

Heasley, you chimed in saying the prompt may be out of sync. While I don't
quite know what that means, you suggested sending output of:

eval `rancid -Ct juniper device`

Here that is, finally:

jlogin -t 120 -c 'show chassis clocks;show chassis environment;show chassis
firmware;show chassis fpc detail;show chassis hardware detail;show chassis
hardware models;show chassis routing-engine;show chassis scb;show chassis
sfm detail;show chassis ssb;show chassis feb detail;show chassis feb;show
chassis cfeb;show chassis alarms;show system license;show system
boot-messages;show system core-dumps;show version detail;show version
invoke-on other-routing-engine;show configuration;file checksum md5
/var/db/scripts/*/*;file list recursive /var/db/scripts/' r-platteville-hub

You may notice some additions at the end which help us track some slax
scripts:

file checksum md5 /var/db/scripts/*/*;file list recursive /var/db/scripts/'

we've had those in place for quite some time (before this) so I'm unsure if
those are related.

Cheers,
Chris

[heasley]

Post by Chris Wopat
Hi folks,
Last year I commented on an issue we're seeing across many Juniper devices.
I neglected to follow up on Heasley's response then but are seeing it a lot
more frequently now, perhaps related to some OS upgrades or something else.
http://www.shrubbery.net/pipermail/rancid-discuss/2017-October/009916.html
http://www.shrubbery.net/pipermail/rancid-discuss/2017-October/009922.html
Looking at the last week or so of these we've had, they're on devices
running 14.1X53-D4*, which is primaraily QFX5100 but also a few EX4200.
Here's output from a single diff, its like this on various commands nearly
Index: configs/r-kettlemoraine-hub
===================================================================
retrieving revision 1.144
diff -u -4 -r1.144 r-kettlemoraine-hub
@@ -1,7 +1,8 @@
#RANCID-CONTENT-TYPE: juniper
#
# r-kettlemoraine-hub> show chassis clocks
+ # show chassis environment
# r-kettlemoraine-hub> show chassis environment
# Class Item Status
# Power FPC 0 Power Supply 0 OK
# FPC 0 Power Supply 1 OK
Index: configs/r-lacrossecity-hub
===================================================================
retrieving revision 1.108
diff -u -4 -r1.108 r-lacrossecity-hub
@@ -15,9 +15,8 @@
# FPC 0 Fan 2 OK
# FPC 0 Fan 3 OK
#
# r-lacrossecity-hub> show chassis firmware
- # show chassis fpc detail
# Part Type Version
# FPC 0 uboot U-Boot 1.1.6 (Jun 5 2012 -
02:24:53) 1.0.0
# loader FreeBSD/PowerPC U-Boot bootstrap
loader 2.4
#
Index: configs/r-platteville-hub
===================================================================
retrieving revision 1.274
diff -u -4 -r1.274 r-platteville-hub
@@ -1,7 +1,8 @@
#RANCID-CONTENT-TYPE: juniper
#
# r-platteville-hub> show chassis clocks
+ # show chassis environment
# r-platteville-hub> show chassis environment
# Class Item Status
# Power FPC 0 Power Supply 0 OK
# FPC 0 Power Supply 1 OK
Heasley, you chimed in saying the prompt may be out of sync. While I don't
eval `rancid -Ct juniper device`
jlogin -t 120 -c 'show chassis clocks;show chassis environment;show chassis
firmware;show chassis fpc detail;show chassis hardware detail;show chassis
hardware models;show chassis routing-engine;show chassis scb;show chassis
sfm detail;show chassis ssb;show chassis feb detail;show chassis feb;show
chassis cfeb;show chassis alarms;show system license;show system
boot-messages;show system core-dumps;show version detail;show version
invoke-on other-routing-engine;show configuration;file checksum md5
/var/db/scripts/*/*;file list recursive /var/db/scripts/' r-platteville-hub
You may notice some additions at the end which help us track some slax
file checksum md5 /var/db/scripts/*/*;file list recursive /var/db/scripts/'
we've had those in place for quite some time (before this) so I'm unsure if
those are related.

not likely. I presume you have a banner with something that looks like a
prompt; like https://www.juniper.net/documentation/software/junos/junos93/swconfig-system-basics/configuring-a-system-login-message.html.
if not, you'll have to share output with me.

eval `rancid -t juniper -C hostname` &> output

[Chris Wopat]

We actually do not have a banner, but your mention of that reminds me that
indeed, when doing some updates recently we enabled login-tip (
https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/login-tip-edit-system.html/)
which is almost certainly putting something random in a banner-ish area
upon each login.

Here are 3 random examples from the same device:

========================================================
Password:
--- JUNOS 14.1X53-D47.3 built 2018-05-10 21:38:01 UTC
JUNOS tip:
Use the 'no-more' CLI pipe to disable the CLI's more capability and
let the multiple pages of output scroll without stopping.


--- JUNOS 14.1X53-D47.3 built 2018-05-10 21:38:01 UTC
JUNOS tip:
Use ESC-/ in the CLI to expand strings into matching words from the
command line history.

Password:
--- JUNOS 14.1X53-D47.3 built 2018-05-10 21:38:01 UTC
JUNOS tip:
Use the TAB key to autocomplete interface names in operational mode.
========================================================

Since they liberally use dashes, single quotes, forward slashes and so on-
this seems to be the likely culprit?

Would it be possible to filter out anything between "JUNOS tip" and the
first valid prompt so we have a chance of leaving them enabled?


--Chris

Post by Chris Wopat

Post by Chris Wopat
Hi folks,
Last year I commented on an issue we're seeing across many Juniper

devices.

Post by Chris Wopat
I neglected to follow up on Heasley's response then but are seeing it a

lot

Post by Chris Wopat
more frequently now, perhaps related to some OS upgrades or something

else.

Post by Chris Wopat
http://www.shrubbery.net/pipermail/rancid-discuss/2017-

October/009916.html

Post by Chris Wopat
http://www.shrubbery.net/pipermail/rancid-discuss/2017-

October/009922.html

Post by Chris Wopat
Looking at the last week or so of these we've had, they're on devices
running 14.1X53-D4*, which is primaraily QFX5100 but also a few EX4200.
Here's output from a single diff, its like this on various commands

nearly

Post by Chris Wopat
Index: configs/r-kettlemoraine-hub
===================================================================
retrieving revision 1.144
diff -u -4 -r1.144 r-kettlemoraine-hub
@@ -1,7 +1,8 @@
#RANCID-CONTENT-TYPE: juniper
#
# r-kettlemoraine-hub> show chassis clocks
+ # show chassis environment
# r-kettlemoraine-hub> show chassis environment
# Class Item Status
# Power FPC 0 Power Supply 0 OK
# FPC 0 Power Supply 1 OK
Index: configs/r-lacrossecity-hub
===================================================================
retrieving revision 1.108
diff -u -4 -r1.108 r-lacrossecity-hub
@@ -15,9 +15,8 @@
# FPC 0 Fan 2 OK
# FPC 0 Fan 3 OK
#
# r-lacrossecity-hub> show chassis firmware
- # show chassis fpc detail
# Part Type Version
# FPC 0 uboot U-Boot 1.1.6 (Jun 5 2012 -
02:24:53) 1.0.0
# loader FreeBSD/PowerPC U-Boot bootstrap
loader 2.4
#
Index: configs/r-platteville-hub
===================================================================
retrieving revision 1.274
diff -u -4 -r1.274 r-platteville-hub
@@ -1,7 +1,8 @@
#RANCID-CONTENT-TYPE: juniper
#
# r-platteville-hub> show chassis clocks
+ # show chassis environment
# r-platteville-hub> show chassis environment
# Class Item Status
# Power FPC 0 Power Supply 0 OK
# FPC 0 Power Supply 1 OK
Heasley, you chimed in saying the prompt may be out of sync. While I

don't

Post by Chris Wopat
eval `rancid -Ct juniper device`
jlogin -t 120 -c 'show chassis clocks;show chassis environment;show

chassis

Post by Chris Wopat
firmware;show chassis fpc detail;show chassis hardware detail;show

chassis

Post by Chris Wopat
hardware models;show chassis routing-engine;show chassis scb;show chassis
sfm detail;show chassis ssb;show chassis feb detail;show chassis feb;show
chassis cfeb;show chassis alarms;show system license;show system
boot-messages;show system core-dumps;show version detail;show version
invoke-on other-routing-engine;show configuration;file checksum md5
/var/db/scripts/*/*;file list recursive /var/db/scripts/'

r-platteville-hub

Post by Chris Wopat
You may notice some additions at the end which help us track some slax
file checksum md5 /var/db/scripts/*/*;file list recursive

/var/db/scripts/'

Post by Chris Wopat
we've had those in place for quite some time (before this) so I'm unsure

if

Post by Chris Wopat
those are related.

not likely. I presume you have a banner with something that looks like a
prompt; like https://www.juniper.net/documentation/software/junos/
junos93/swconfig-system-basics/configuring-a-system-login-message.html.
if not, you'll have to share output with me.
eval `rancid -t juniper -C hostname` &> output

[heasley]

Post by Chris Wopat
We actually do not have a banner, but your mention of that reminds me that
indeed, when doing some updates recently we enabled login-tip (
https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/login-tip-edit-system.html/)
which is almost certainly putting something random in a banner-ish area
upon each login.
========================================================
--- JUNOS 14.1X53-D47.3 built 2018-05-10 21:38:01 UTC
Use the 'no-more' CLI pipe to disable the CLI's more capability and
let the multiple pages of output scroll without stopping.
--- JUNOS 14.1X53-D47.3 built 2018-05-10 21:38:01 UTC
Use ESC-/ in the CLI to expand strings into matching words from the
command line history.
--- JUNOS 14.1X53-D47.3 built 2018-05-10 21:38:01 UTC
Use the TAB key to autocomplete interface names in operational mode.
========================================================
Since they liberally use dashes, single quotes, forward slashes and so on-
this seems to be the likely culprit?

maybe; those do not look like prompts.

Post by Chris Wopat
Would it be possible to filter out anything between "JUNOS tip" and the
first valid prompt so we have a chance of leaving them enabled?

it is all about reliably identifying the valid prompt. if i could tell
users and vendors not to use [\][[:space:])(_*\\<>] in their prompts, life
would be easier - the regex would simply be
^[^PROMPTTERMINALCHAR\r\n ]+PROMPTTERMINALCHAR

jlogin is looking for '>'. can you reproduce it reliably with a particular
device? if you can share (with me only) the output of
jlogin -d -c 'show version' hostname 2> output
i expect that i can fix it, but i may need more output, like:
jlogin -d -c 'show chassis clocks;show chassis environment;show chassis firmware;show chassis fpc detail' hostname 2> output

Post by Chris Wopat
--Chris

Post by Chris Wopat

Post by Chris Wopat
Hi folks,
Last year I commented on an issue we're seeing across many Juniper

devices.

Post by Chris Wopat
I neglected to follow up on Heasley's response then but are seeing it a

lot

Post by Chris Wopat
more frequently now, perhaps related to some OS upgrades or something

else.

Post by Chris Wopat
http://www.shrubbery.net/pipermail/rancid-discuss/2017-

October/009916.html

Post by Chris Wopat
http://www.shrubbery.net/pipermail/rancid-discuss/2017-

October/009922.html

Post by Chris Wopat
Looking at the last week or so of these we've had, they're on devices
running 14.1X53-D4*, which is primaraily QFX5100 but also a few EX4200.
Here's output from a single diff, its like this on various commands

nearly

Post by Chris Wopat
Index: configs/r-kettlemoraine-hub
===================================================================
retrieving revision 1.144
diff -u -4 -r1.144 r-kettlemoraine-hub
@@ -1,7 +1,8 @@
#RANCID-CONTENT-TYPE: juniper
#
# r-kettlemoraine-hub> show chassis clocks
+ # show chassis environment
# r-kettlemoraine-hub> show chassis environment
# Class Item Status
# Power FPC 0 Power Supply 0 OK
# FPC 0 Power Supply 1 OK
Index: configs/r-lacrossecity-hub
===================================================================
retrieving revision 1.108
diff -u -4 -r1.108 r-lacrossecity-hub
@@ -15,9 +15,8 @@
# FPC 0 Fan 2 OK
# FPC 0 Fan 3 OK
#
# r-lacrossecity-hub> show chassis firmware
- # show chassis fpc detail
# Part Type Version
# FPC 0 uboot U-Boot 1.1.6 (Jun 5 2012 -
02:24:53) 1.0.0
# loader FreeBSD/PowerPC U-Boot bootstrap
loader 2.4
#
Index: configs/r-platteville-hub
===================================================================
retrieving revision 1.274
diff -u -4 -r1.274 r-platteville-hub
@@ -1,7 +1,8 @@
#RANCID-CONTENT-TYPE: juniper
#
# r-platteville-hub> show chassis clocks
+ # show chassis environment
# r-platteville-hub> show chassis environment
# Class Item Status
# Power FPC 0 Power Supply 0 OK
# FPC 0 Power Supply 1 OK
Heasley, you chimed in saying the prompt may be out of sync. While I

don't

Post by Chris Wopat
eval `rancid -Ct juniper device`
jlogin -t 120 -c 'show chassis clocks;show chassis environment;show

chassis

Post by Chris Wopat
firmware;show chassis fpc detail;show chassis hardware detail;show

chassis

Post by Chris Wopat
hardware models;show chassis routing-engine;show chassis scb;show chassis
sfm detail;show chassis ssb;show chassis feb detail;show chassis feb;show
chassis cfeb;show chassis alarms;show system license;show system
boot-messages;show system core-dumps;show version detail;show version
invoke-on other-routing-engine;show configuration;file checksum md5
/var/db/scripts/*/*;file list recursive /var/db/scripts/'

r-platteville-hub

Post by Chris Wopat
You may notice some additions at the end which help us track some slax
file checksum md5 /var/db/scripts/*/*;file list recursive

/var/db/scripts/'

Post by Chris Wopat
we've had those in place for quite some time (before this) so I'm unsure

if

Post by Chris Wopat
those are related.

not likely. I presume you have a banner with something that looks like a
prompt; like https://www.juniper.net/documentation/software/junos/
junos93/swconfig-system-basics/configuring-a-system-login-message.html.
if not, you'll have to share output with me.
eval `rancid -t juniper -C hostname` &> output