[rancid] CentOS 6.6, recent upgrade, now all CatOS devices broken

[Brooks Bridges]

Had a system with Rancid installed from the EPEL repos. All switches
were configured and everything worked great. Someone ran a yum update on
the system yesterday, and it upgraded the package to
"rancid-3.2-1.el6.x86_64", which broke all kinds of crap.

The first indication was the Socket module bug, which I got corrected,
however now all CatOS devices in the network are "down" and cannot get
the configs pulled from them. After adding a bit of debugging, the logs
show:

smfits1.domain.com: missed cmd(s): write term all,show port ifindex,show
module,dir sup-microcode:,dir sup-bootflash:,dir bootflash:,dir
slot0:,show flash,show running-config,write term,show boot,dir
slot1:,show inventory raw
smfits1.domain.com - Clean_run is: 0 <--- added debugging statement
smfits1.domain.com - found_end is: 0 <--- added debugging statement
smfits1.domain.com: End of run not found

After many hours of digging, I've determined that the issue appears to
be that clogin is not properly recognizing the end of the output of the
commands.

Debug log can be viewed at http://pastebin.ca/3027707 showing this
behavior when running "show version", where it gets to the end of the
output from the command and then just hangs there. It appears that none
of the regexes are being matched when the command prompt pops back up
after the command completes (perhaps it's only looking for a 'more'
prompt?).

Since the FTP site still doesn't let you download anything from the .old
directory (and doesn't show any indication that it's going to be fixed
anytime soon) I can't get my hands on the old version to compare it
against. My lack of perl, tcl, and expect skills are quite an obstacle
here as well.

Any pointers in how to hammer this thing back into submission?

[Brooks Bridges]

Had a system with Rancid installed from the EPEL repos. All switches
were configured and everything worked great. Someone ran a yum update
on the system yesterday, and it upgraded the package to
"rancid-3.2-1.el6.x86_64", which broke all kinds of crap.

The first indication was the Socket module bug, which I got corrected,
however now all CatOS devices in the network are "down" and cannot get
the configs pulled from them. After adding a bit of debugging, the logs
show:

smfits1.domain.com: missed cmd(s): write term all,show port ifindex,show
module,dir sup-microcode:,dir sup-bootflash:,dir bootflash:,dir
slot0:,show flash,show running-config,write term,show boot,dir
slot1:,show inventory raw
smfits1.domain.com - Clean_run is: 0 <--- added debugging statement
smfits1.domain.com - found_end is: 0 <--- added debugging statement
smfits1.domain.com: End of run not found

After many hours of digging, I've determined that the issue appears to
be that clogin is not properly recognizing the end of the output of the
commands. Debug log is attached showing this behavior when running
"show version", where it gets to the end of the output from the command
and then just hangs there. It appears that none of the regexes are
being matched when the command prompt pops back up after the command
completes (perhaps it's only looking for a 'more' prompt?).

Since the FTP site still doesn't let you download anything from the .old
directory (and doesn't show any indication that it's going to be fixed
anytime soon) I can't get my hands on the old version to compare it
against. My lack of perl, tcl, and expect skills are quite an obstacle
here as well.

Any pointers in how to hammer this thing back into submission?

[Joseph Bernard]

What version were you using previously?

Thanks,
Joseph B.

-----Original Message-----
From: Rancid-discuss [mailto:rancid-discuss-***@shrubbery.net] On Behalf Of Brooks Bridges
Sent: Sunday, June 14, 2015 12:51 AM
To: rancid-***@shrubbery.net
Subject: [rancid] CentOS 6.6, recent upgrade, now all CatOS devices broken

Had a system with Rancid installed from the EPEL repos. All switches were configured and everything worked great. Someone ran a yum update on the system yesterday, and it upgraded the package to "rancid-3.2-1.el6.x86_64", which broke all kinds of crap.

The first indication was the Socket module bug, which I got corrected, however now all CatOS devices in the network are "down" and cannot get the configs pulled from them. After adding a bit of debugging, the logs
show:

smfits1.domain.com: missed cmd(s): write term all,show port ifindex,show module,dir sup-microcode:,dir sup-bootflash:,dir bootflash:,dir slot0:,show flash,show running-config,write term,show boot,dir slot1:,show inventory raw
smfits1.domain.com - Clean_run is: 0 <--- added debugging statement
smfits1.domain.com - found_end is: 0 <--- added debugging statement
smfits1.domain.com: End of run not found

After many hours of digging, I've determined that the issue appears to be that clogin is not properly recognizing the end of the output of the commands. Debug log is attached showing this behavior when running "show version", where it gets to the end of the output from the command and then just hangs there. It appears that none of the regexes are being matched when the command prompt pops back up after the command completes (perhaps it's only looking for a 'more' prompt?).

Since the FTP site still doesn't let you download anything from the .old directory (and doesn't show any indication that it's going to be fixed anytime soon) I can't get my hands on the old version to compare it against. My lack of perl, tcl, and expect skills are quite an obstacle here as well.

Any pointers in how to hammer this thing back into submission?

--
-Brooks Bridges

[Brooks Bridges]

That's the thing, I don't really remember and I can't seem to find anything in the EPEL repo that says what the previous version was.


-Brooks Bridges

Post by Joseph Bernard
What version were you using previously?
Thanks,
Joseph B.
-----Original Message-----
Behalf Of Brooks Bridges
Sent: Sunday, June 14, 2015 12:51 AM
Subject: [rancid] CentOS 6.6, recent upgrade, now all CatOS devices broken
Had a system with Rancid installed from the EPEL repos. All switches
were configured and everything worked great. Someone ran a yum update
on the system yesterday, and it upgraded the package to
"rancid-3.2-1.el6.x86_64", which broke all kinds of crap.
The first indication was the Socket module bug, which I got corrected,
however now all CatOS devices in the network are "down" and cannot get
the configs pulled from them. After adding a bit of debugging, the logs
smfits1.domain.com: missed cmd(s): write term all,show port
ifindex,show module,dir sup-microcode:,dir sup-bootflash:,dir
bootflash:,dir slot0:,show flash,show running-config,write term,show
boot,dir slot1:,show inventory raw
smfits1.domain.com - Clean_run is: 0 <--- added debugging statement
smfits1.domain.com - found_end is: 0 <--- added debugging statement
smfits1.domain.com: End of run not found
After many hours of digging, I've determined that the issue appears to
be that clogin is not properly recognizing the end of the output of the
commands. Debug log is attached showing this behavior when running
"show version", where it gets to the end of the output from the command
and then just hangs there. It appears that none of the regexes are
being matched when the command prompt pops back up after the command
completes (perhaps it's only looking for a 'more' prompt?).
Since the FTP site still doesn't let you download anything from the
.old directory (and doesn't show any indication that it's going to be
fixed anytime soon) I can't get my hands on the old version to compare
it against. My lack of perl, tcl, and expect skills are quite an
obstacle here as well.
Any pointers in how to hammer this thing back into submission?
--
-Brooks Bridges
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

[Lance Vermilion]

http://www.cyberciti.biz/faq/yum-history-command/

That should help you to figure out what was installed.

Yum history package-list rancid

Then

Yum history info #

Post by Brooks Bridges
That's the thing, I don't really remember and I can't seem to find
anything in the EPEL repo that says what the previous version was.
-Brooks Bridges

Post by Joseph Bernard
What version were you using previously?
Thanks,
Joseph B.
-----Original Message-----
Sent: Sunday, June 14, 2015 12:51 AM
Subject: [rancid] CentOS 6.6, recent upgrade, now all CatOS devices broken
Had a system with Rancid installed from the EPEL repos. All switches were configured and everything worked great. Someone ran a yum update on the system yesterday, and it upgraded the package to "rancid-3.2-1.el6.x86_64", which broke all kinds of crap.
The first indication was the Socket module bug, which I got corrected, however now all CatOS devices in the network are "down" and cannot get the configs pulled from them. After adding a bit of debugging, the logs
smfits1.domain.com: missed cmd(s): write term
all,show port ifindex,show module,dir sup-microcode:,dir sup-bootflash:,dir bootflash:,dir slot0:,show flash,show running-config,write term,show boot,dir slot1:,show inventory raw
smfits1.domain.com - Clean_run is: 0 <--- added debugging statement
smfits1.domain.com - found_end is: 0 <--- added debugging statement
smfits1.domain.com: End of run not found
After many hours of digging, I've determined that the issue appears to be that clogin is not properly recognizing the end of the output of the commands. Debug log is attached showing this behavior when running "show version", where it gets to the end of the output from the command and then just hangs there. It appears that none of the regexes are being matched when the command prompt pops back up after the command completes (perhaps it's only looking for a 'more' prompt?).
Since the FTP site still doesn't let you download anything from the .old directory (and doesn't show any indication that it's going to be fixed anytime soon) I can't get my hands on the old version to compare it against. My lack of perl, tcl, and expect skills are quite an obstacle here as well.
Any pointers in how to hammer this thing back into submission?
--
-Brooks Bridges
------------------------------
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

[Brooks Bridges]

TIL about yum history. Neat. Thanks!

So yeah, rancid-3.1-3.el6.x86_64 was the previous version. I'll grab
that version from Joseph's link and diff it against the current one and
see what I can find.

-Brooks Bridges

Post by Lance Vermilion
http://www.cyberciti.biz/faq/yum-history-command/
That should help you to figure out what was installed.
Yum history package-list rancid
Then
Yum history info #
That's the thing, I don't really remember and I can't seem to find
anything in the EPEL repo that says what the previous version was.
-Brooks Bridges
What version were you using previously?
Thanks,
Joseph B.
-----Original Message-----
Sent: Sunday, June 14, 2015 12:51 AM
Subject: [rancid] CentOS 6.6, recent upgrade, now all CatOS devices broken
Had a system with Rancid installed from the EPEL repos. All switches were configured and everything worked great. Someone ran a yum update on the system yesterday, and it upgraded the package to "rancid-3.2-1.el6.x86_64", which broke all kinds of crap.
The first indication was the Socket module bug, which I got corrected, however now all CatOS devices in the network are "down" and cannot get the configs pulled from them. After adding a bit of debugging, the logs
smfits1.domain.com <http://smfits1.domain.com>: missed cmd(s): write term
all,show port ifindex,show module,dir sup-microcode:,dir sup-bootflash:,dir bootflash:,dir slot0:,show flash,show running-config,write term,show boot,dir slot1:,show inventory raw
smfits1.domain.com <http://smfits1.domain.com> - Clean_run is: 0 <--- added debugging statement
smfits1.domain.com <http://smfits1.domain.com> - found_end is: 0 <--- added debugging statement
smfits1.domain.com <http://smfits1.domain.com>: End of run not found
After many hours of digging, I've determined that the issue appears to be that clogin is not properly recognizing the end of the output of the commands. Debug log is attached showing this behavior when running "show version", where it gets to the end of the output from the command and then just hangs there. It appears that none of the regexes are being matched when the command prompt pops back up after the command completes (perhaps it's only looking for a 'more' prompt?).
Since the FTP site still doesn't let you download anything from the .old directory (and doesn't show any indication that it's going to be fixed anytime soon) I can't get my hands on the old version to compare it against. My lack of perl, tcl, and expect skills are quite an obstacle here as well.
Any pointers in how to hammer this thing back into submission?
--
-Brooks Bridges
------------------------------------------------------------------------
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss
_______________________________________________ Rancid-discuss
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

[Brooks Bridges]

Yeah, there's definitely some pretty significant changes in how the
prompt is being handled.

From 3.1-3.el6.x86_64:
# match cisco config mode prompts too, such as router(config-if)#,
# but catalyst does not change in this fashion.
regsub -all {^(.{1,11}).*([#>])$} $prompt
{\1([^#>\r\n]+)?[#>](\\([^)\\r\\n]+\\))?} reprompt
} else {
set reprompt $prompt
}

From 3.2-1.el6.x86_64:
# match cisco config mode prompts too, such as router(config-if)#,
# but catalyst does not change in this fashion.
regsub -lineanchor -- {^(.{1,11}).*([#>])$} $prompt {\1} reprompt
regsub -all -- {[\\]$} $reprompt {} reprompt
append reprompt {([^#>\r\n]+)?[#>](\\([^)\\r\\n]+\\))?}
} else {
set reprompt $prompt
}

Now we're into "Hell if I know regex well enough to figure that out"
territory. Anyone care to take a crack at it? I'm more than happy to
be the guinea pig.

-Brooks Bridges

Post by Brooks Bridges
TIL about yum history. Neat. Thanks!
So yeah, rancid-3.1-3.el6.x86_64 was the previous version. I'll grab
that version from Joseph's link and diff it against the current one
and see what I can find.
-Brooks Bridges

Post by Lance Vermilion
http://www.cyberciti.biz/faq/yum-history-command/
That should help you to figure out what was installed.
Yum history package-list rancid
Then
Yum history info #
That's the thing, I don't really remember and I can't seem to
find anything in the EPEL repo that says what the previous
version was.
-Brooks Bridges
What version were you using previously?
Thanks,
Joseph B.
-----Original Message-----
Sent: Sunday, June 14, 2015 12:51 AM
Subject: [rancid] CentOS 6.6, recent upgrade, now all CatOS devices broken
Had a system with Rancid installed from the EPEL repos. All switches were configured and everything worked great. Someone ran a yum update on the system yesterday, and it upgraded the package to "rancid-3.2-1.el6.x86_64", which broke all kinds of crap.
The first indication was the Socket module bug, which I got corrected, however now all CatOS devices in the network are "down" and cannot get the configs pulled from them. After adding a bit of debugging, the logs
smfits1.domain.com <http://smfits1.domain.com>: missed cmd(s): write term
all,show port ifindex,show module,dir sup-microcode:,dir sup-bootflash:,dir bootflash:,dir slot0:,show flash,show running-config,write term,show boot,dir slot1:,show inventory raw
smfits1.domain.com <http://smfits1.domain.com> - Clean_run is: 0 <--- added debugging statement
smfits1.domain.com <http://smfits1.domain.com> - found_end is: 0 <--- added debugging statement
smfits1.domain.com <http://smfits1.domain.com>: End of run not found
After many hours of digging, I've determined that the issue appears to be that clogin is not properly recognizing the end of the output of the commands. Debug log is attached showing this behavior when running "show version", where it gets to the end of the output from the command and then just hangs there. It appears that none of the regexes are being matched when the command prompt pops back up after the command completes (perhaps it's only looking for a 'more' prompt?).
Since the FTP site still doesn't let you download anything from the .old directory (and doesn't show any indication that it's going to be fixed anytime soon) I can't get my hands on the old version to compare it against. My lack of perl, tcl, and expect skills are quite an obstacle here as well.
Any pointers in how to hammer this thing back into submission?
--
-Brooks Bridges
------------------------------------------------------------------------
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss
_______________________________________________ Rancid-discuss
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

[Joseph Bernard]

I think this was the previous version:

https://kojipkgs.fedoraproject.org/packages/rancid/3.1/3.el6/x86_64/

Thanks,
Joseph B.

From: Rancid-discuss [mailto:rancid-discuss-***@shrubbery.net] On Behalf Of Brooks Bridges
Sent: Sunday, June 14, 2015 12:45 PM
To: rancid-***@shrubbery.net
Subject: Re: [rancid] CentOS 6.6, recent upgrade, now all CatOS devices broken

That's the thing, I don't really remember and I can't seem to find anything in the EPEL repo that says what the previous version was.


-Brooks Bridges

On June 14, 2015 9:42:13 AM PDT, Joseph Bernard <***@clemson.edu<mailto:***@clemson.edu>> wrote:

What version were you using previously?

Thanks,
Joseph B.

-----Original Message-----
From: Rancid-discuss [mailto:rancid-discuss-***@shrubbery.net] On Behalf Of Brooks Bridges
Sent: Sunday, June 14, 2015 12:51 AM
To: rancid-***@shrubbery.net<mailto:rancid-***@shrubbery.net>
Subject: [rancid] CentOS 6.6, recent upgrade, now all CatOS devices broken

Had a system with Rancid installed from the EPEL repos. All switches were configured and everything worked great. Someone ran a yum update on the system yesterday, and it upgraded the package to "rancid-3.2-1.el6.x86_64", which broke all kinds of crap.

The first indication was the Socket module bug, which I got corrected, however now all CatOS devices in the network are "down" and cannot get the configs pulled from them. After adding a bit of debugging, the logs
show:

smfits1.domain.com<http://smfits1.domain.com>: missed cmd(s): write te!

rm

all,show port ifindex,show module,dir sup-microcode:,dir sup-bootflash:,dir bootflash:,dir slot0:,show flash,show running-config,write term,show boot,dir slot1:,show inventory raw
smfits1.domain.com<http://smfits1.domain.com> - Clean_run is: 0 <--- added debugging statement
smfits1.domain.com<http://smfits1.domain.com> - found_end is: 0 <--- added debugging statement
smfits1.domain.com<http://smfits1.domain.com>: End of run not found

After many hours of digging, I've determined that the issue appears to be that clogin is not properly recognizing the end of the output of the commands. Debug log is attached showing this behavior when running "show version", where it gets to the end of the output from the command and then just hangs there. It appears that none of the regexes are being matched when the command prompt pops back up after the command completes (perhaps it's only looking for a 'more' prom!

pt?).

Since the FTP site still doesn't let you download anything from the .old directory (and doesn't show any indication that it's going to be fixed anytime soon) I can't get my hands on the old version to compare it against. My lack of perl, tcl, and expect skills are quite an obstacle here as well.

Any pointers in how to hammer this thing back into submission?