[rancid] FreeBSD 7.0 + Expect 5.43.0 -- Do I need to patch expect?

Post by Eric Cables
I've been running into this problem quite frequently, about once a week or
so rancid will hang for no apparent reason, and until I kill the process it
will remain hung. I've read in the archives that Linux & Solaris have a
problem with expect that requires a patch, but does this also include
FreeBSD?

Yes, some combinations of recent tcl / expect seem to have trouble on fbsd.
I'm not sure why, but the expect was pulled from distribution site for
defects. the problem is that rolling back (via ports) seems to resurrect
an old problem which itself had been correct in ports by rolling back, IIRC.

I do not use freebsd and haven't had time to debug the problem. ports is,
imho, too anxious to move forward with new tcl/expect versions and both are
too delicate for that.

Eric Cables

2008-11-14 19:25:29 UTC

Are there any debugging options available to try and isolate the root cause
of the problem? I could probably write up a shell script that finds hung
processes and kills them, but that's certainly not the best option. :-)

--
Eric Cables

Post by Eric Cables
I've been running into this problem quite frequently, about once a week

Post by Eric Cables
so rancid will hang for no apparent reason, and until I kill the process

Post by Eric Cables
will remain hung. I've read in the archives that Linux & Solaris have a
problem with expect that requires a patch, but does this also include
FreeBSD?

Yes, some combinations of recent tcl / expect seem to have trouble on fbsd.
I'm not sure why, but the expect was pulled from distribution site for
defects. the problem is that rolling back (via ports) seems to resurrect
an old problem which itself had been correct in ports by rolling back, IIRC.
I do not use freebsd and haven't had time to debug the problem. ports is,
imho, too anxious to move forward with new tcl/expect versions and both are
too delicate for that.

john heasley

2008-11-14 23:26:00 UTC

I have no idea what the problem is. I expect it'l require a mix of
ktrace, gcore/gdb, and tcpdump to figure out what is causing tcl/expect
to hang.

Post by Eric Cables
Are there any debugging options available to try and isolate the root cause
of the problem? I could probably write up a shell script that finds hung
processes and kills them, but that's certainly not the best option. :-)
--
Eric Cables

Post by Eric Cables
I've been running into this problem quite frequently, about once a week

Post by Eric Cables
so rancid will hang for no apparent reason, and until I kill the process

Post by Eric Cables
will remain hung. I've read in the archives that Linux & Solaris have a
problem with expect that requires a patch, but does this also include
FreeBSD?

Yes, some combinations of recent tcl / expect seem to have trouble on fbsd.
I'm not sure why, but the expect was pulled from distribution site for
defects. the problem is that rolling back (via ports) seems to resurrect
an old problem which itself had been correct in ports by rolling back, IIRC.
I do not use freebsd and haven't had time to debug the problem. ports is,
imho, too anxious to move forward with new tcl/expect versions and both are
too delicate for that.

Eric Cables

2008-11-14 23:47:39 UTC

Is this something I can do when I discover the process is hung, or something
that has to be prepared before it hangs? For example, if I come back into
the office on Monday to find RANCID hung, is there anything I can do to
collect forensics as to what caused it to hang?

--
Eric Cables

Post by john heasley
I have no idea what the problem is. I expect it'l require a mix of
ktrace, gcore/gdb, and tcpdump to figure out what is causing tcl/expect
to hang.

Post by Eric Cables
Are there any debugging options available to try and isolate the root

cause

Post by Eric Cables
of the problem? I could probably write up a shell script that finds hung
processes and kills them, but that's certainly not the best option. :-)
--
Eric Cables

Post by Eric Cables
I've been running into this problem quite frequently, about once a

week

Post by john heasley
or

Post by Eric Cables
so rancid will hang for no apparent reason, and until I kill the

process

Post by john heasley
it

Post by Eric Cables
will remain hung. I've read in the archives that Linux & Solaris

have a

Post by Eric Cables
problem with expect that requires a patch, but does this also include
FreeBSD?

Yes, some combinations of recent tcl / expect seem to have trouble on

fbsd.

Post by john heasley
I'm not sure why, but the expect was pulled from distribution site for
defects. the problem is that rolling back (via ports) seems to

resurrect

Post by john heasley
an old problem which itself had been correct in ports by rolling back, IIRC.
I do not use freebsd and haven't had time to debug the problem. ports

is,

Post by john heasley
imho, too anxious to move forward with new tcl/expect versions and both

are

Post by john heasley
too delicate for that.

john heasley

2008-11-18 19:27:30 UTC

Post by Eric Cables
Is this something I can do when I discover the process is hung, or something
that has to be prepared before it hangs? For example, if I come back into
the office on Monday to find RANCID hung, is there anything I can do to
collect forensics as to what caused it to hang?

I don't believe so. Looking back, Randy has already provided enough to
identify the problem and access to his machine. So, I need to find the
time to debug it.

If anyone with the knowledge to debug it wants to work on it; my notes
indicate that we found that globs are broken. since, there may have been
other changes with fbsd ports that have changed that.

Sam Munzani

2008-11-18 19:35:52 UTC

I don't believe so. Looking back, Randy has already provided enough to
identify the problem and access to his machine. So, I need to find the
time to debug it.
If anyone with the knowledge to debug it wants to work on it; my notes
indicate that we found that globs are broken. since, there may have been
other changes with fbsd ports that have changed that.
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

Eric Cables

2008-11-18 21:17:59 UTC

Well, given the alert on expect's homepage, I'm not sure upgrading is
the best option..

Alert: Version 5.44 of Expect has been recalled due to unresolved
problems. Please use 5.43.0. All links below point to 5.43.0 - Don

I had similar issues a few months ago. Instead of spending time on debug, I chose to use the source from shruberry.net(2.3.2a7 version) instead of ports and used expect version 5.44.1.7. Everything seems to work with this combination.
Thanks,
Sam
Is this something I can do when I discover the process is hung, or something
that has to be prepared before it hangs? For example, if I come back into
the office on Monday to find RANCID hung, is there anything I can do to
collect forensics as to what caused it to hang?
I don't believe so. Looking back, Randy has already provided enough to
identify the problem and access to his machine. So, I need to find the
time to debug it.
If anyone with the knowledge to debug it wants to work on it; my notes
indicate that we found that globs are broken. since, there may have been
other changes with fbsd ports that have changed that.
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

john heasley

2008-11-22 08:27:18 UTC

Post by Sam Munzani
I had similar issues a few months ago. Instead of spending time on
debug, I chose to use the source from shruberry.net(2.3.2a7 version)
instead of ports and used expect version 5.44.1.7. Everything seems to
work with this combination.

I find that confusing. I wouldn't expect (pun not intended) that the
version of rancid would matter here, unless you'd gone back quite far.
I suspect it may have found a different version of expect/tcl.

To answer eric, there was a "normal" and devel version of the expect fbsd
port. Neither version worked properly. I haven't looked at them recently,
so they may have changed.

Post by Sam Munzani
Thanks,
Sam

I don't believe so. Looking back, Randy has already provided enough to
identify the problem and access to his machine. So, I need to find the
time to debug it.
If anyone with the knowledge to debug it wants to work on it; my notes
indicate that we found that globs are broken. since, there may have been
other changes with fbsd ports that have changed that.
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

Russell Jackson

2008-11-24 18:11:10 UTC

I've been running rancid on FreeBSD 6.2 without issue for some time. I haven't had any
hangs with respect to expect. I haven't tried it on FreeBSD 7.0 yet.

$ uname -a
FreeBSD svn.csub.edu 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jul 13 19:38:42 PDT 2000
***@thor.csub.edu:/usr/obj/usr/src/sys/THOR i386

$ pkg_info | egrep 'tcl|rancid'
expect-nox11-5.44.1.7 A sophisticated scripter based on tcl/tk
rancid-local-2.3.2a7_1 Really Awesome New Cisco confIg Differ
tcl-8.4.19,1 Tool Command Language

I'm running a locally patched version of the rancid port to silence cisco diff noise (STP
costs et al).

$ diff -ur net-mgmt/rancid-devel local/rancid
Only in local/rancid: .svn
diff -ur net-mgmt/rancid-devel/Makefile local/rancid/Makefile
--- net-mgmt/rancid-devel/Makefile Mon Dec 31 06:28:08 2007
+++ local/rancid/Makefile Mon Feb 25 15:56:16 2008
@@ -10,7 +10,6 @@
PORTREVISION= 1
CATEGORIES= net-mgmt
MASTER_SITES= ftp://ftp.shrubbery.net/pub/rancid/
-PKGNAMESUFFIX= -devel

MAINTAINER= ***@bsd.hu
COMMENT= Really Awesome New Cisco confIg Differ
Only in local/rancid/files: .svn
Only in local/rancid/files: patch-bin::cat5rancid.in
Only in local/rancid/files: patch-bin::clogin.in
Only in local/rancid/files: patch-bin::rancid.in
Only in local/rancid: pkg-message

--
Russell A. Jackson <***@csub.edu>
Network Analyst
California State University, Bakersfield

The greatest productive force is human selfishness.
-- Robert Heinlein

Eric Cables

2008-12-04 02:10:37 UTC

This problem has made a resurgence this week, crashing out every night and
sending the CPU on the box that RANCID runs on sky high. I end up killing
the hung expect processes manually, but instead of staying constant it
appears to be getting worse (dang growing environments).

Any suggestions are welcome...

-- Eric Cables

Post by Eric Cables
I've been running into this problem quite frequently, about once a week

Post by Eric Cables
so rancid will hang for no apparent reason, and until I kill the process

Post by Eric Cables
will remain hung. I've read in the archives that Linux & Solaris have a
problem with expect that requires a patch, but does this also include
FreeBSD?

I've been running rancid on FreeBSD 6.2 without issue for some time. I haven't had any
hangs with respect to expect. I haven't tried it on FreeBSD 7.0 yet.
$ uname -a
FreeBSD svn.csub.edu 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jul 13 19:38:42 PDT 2000
$ pkg_info | egrep 'tcl|rancid'
expect-nox11-5.44.1.7 A sophisticated scripter based on tcl/tk
rancid-local-2.3.2a7_1 Really Awesome New Cisco confIg Differ
tcl-8.4.19,1 Tool Command Language
I'm running a locally patched version of the rancid port to silence cisco diff noise (STP
costs et al).
$ diff -ur net-mgmt/rancid-devel local/rancid
Only in local/rancid: .svn
diff -ur net-mgmt/rancid-devel/Makefile local/rancid/Makefile
--- net-mgmt/rancid-devel/Makefile Mon Dec 31 06:28:08 2007
+++ local/rancid/Makefile Mon Feb 25 15:56:16 2008
@@ -10,7 +10,6 @@
PORTREVISION= 1
CATEGORIES= net-mgmt
MASTER_SITES= ftp://ftp.shrubbery.net/pub/rancid/
-PKGNAMESUFFIX= <ftp://ftp.shrubbery.net/pub/rancid/-PKGNAMESUFFIX=>-devel
COMMENT= Really Awesome New Cisco confIg Differ
Only in local/rancid/files: .svn
Only in local/rancid/files: patch-bin::cat5rancid.in
Only in local/rancid/files: patch-bin::clogin.in
Only in local/rancid/files: patch-bin::rancid.in
Only in local/rancid: pkg-message
--
Network Analyst
California State University, Bakersfield
The greatest productive force is human selfishness.
-- Robert Heinlein
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

Eric Cables

2008-12-04 02:25:41 UTC

Would upgrading to tcl8.5 be a potential solution? Reading through the old
expect threads it indicated that the problem may be with Tcl itself. Right
now the system is running tcl-8.4.19.

-- Eric Cables

Post by Eric Cables
This problem has made a resurgence this week, crashing out every night and
sending the CPU on the box that RANCID runs on sky high. I end up killing
the hung expect processes manually, but instead of staying constant it
appears to be getting worse (dang growing environments).
Any suggestions are welcome...
-- Eric Cables

Post by Eric Cables
I've been running into this problem quite frequently, about once a week

Post by Eric Cables
so rancid will hang for no apparent reason, and until I kill the process

Post by Eric Cables
will remain hung. I've read in the archives that Linux & Solaris have a
problem with expect that requires a patch, but does this also include
FreeBSD?

I've been running rancid on FreeBSD 6.2 without issue for some time. I haven't had any
hangs with respect to expect. I haven't tried it on FreeBSD 7.0 yet.
$ uname -a
FreeBSD svn.csub.edu 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jul 13 19:38:42 PDT 2000
$ pkg_info | egrep 'tcl|rancid'
expect-nox11-5.44.1.7 A sophisticated scripter based on tcl/tk
rancid-local-2.3.2a7_1 Really Awesome New Cisco confIg Differ
tcl-8.4.19,1 Tool Command Language
I'm running a locally patched version of the rancid port to silence cisco diff noise (STP
costs et al).
$ diff -ur net-mgmt/rancid-devel local/rancid
Only in local/rancid: .svn
diff -ur net-mgmt/rancid-devel/Makefile local/rancid/Makefile
--- net-mgmt/rancid-devel/Makefile Mon Dec 31 06:28:08 2007
+++ local/rancid/Makefile Mon Feb 25 15:56:16 2008
@@ -10,7 +10,6 @@
PORTREVISION= 1
CATEGORIES= net-mgmt
MASTER_SITES= ftp://ftp.shrubbery.net/pub/rancid/
-PKGNAMESUFFIX= <ftp://ftp.shrubbery.net/pub/rancid/-PKGNAMESUFFIX=>-devel
COMMENT= Really Awesome New Cisco confIg Differ
Only in local/rancid/files: .svn
Only in local/rancid/files: patch-bin::cat5rancid.in
Only in local/rancid/files: patch-bin::clogin.in
Only in local/rancid/files: patch-bin::rancid.in
Only in local/rancid: pkg-message
--
Network Analyst
California State University, Bakersfield
The greatest productive force is human selfishness.
-- Robert Heinlein
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

Peter Serwe

2008-12-04 02:56:01 UTC

I'm using tcl and tk 8.4.19 and the patched version of expect 5.40.1
from shrubbery.net
on two different installations. I would recommend rebuilding tcl, tk,
expect, and then rancid
and I'm relatively certain the behavior will disappear.

This is, by the way, on a pair of CentOS 5.2 systems. One of them was
4.7 until a recent
drive failure without good backups caused me to rebuild it into a 5.2
box on a new drive and
rebuild all of the software on it.

Peter

Post by Eric Cables
Would upgrading to tcl8.5 be a potential solution? Reading through the old
expect threads it indicated that the problem may be with Tcl itself. Right
now the system is running tcl-8.4.19.
-- Eric Cables

I've been running rancid on FreeBSD 6.2 without issue for some time. I haven't had any
hangs with respect to expect. I haven't tried it on FreeBSD 7.0 yet.
$ uname -a
FreeBSD svn.csub.edu 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jul 13 19:38:42 PDT 2000
$ pkg_info | egrep 'tcl|rancid'
expect-nox11-5.44.1.7 A sophisticated scripter based on tcl/tk
rancid-local-2.3.2a7_1 Really Awesome New Cisco confIg Differ
tcl-8.4.19,1 Tool Command Language
I'm running a locally patched version of the rancid port to silence cisco
diff noise (STP
costs et al).
$ diff -ur net-mgmt/rancid-devel local/rancid
Only in local/rancid: .svn
diff -ur net-mgmt/rancid-devel/Makefile local/rancid/Makefile
--- net-mgmt/rancid-devel/Makefile Mon Dec 31 06:28:08 2007
+++ local/rancid/Makefile Mon Feb 25 15:56:16 2008
@@ -10,7 +10,6 @@
PORTREVISION= 1
CATEGORIES= net-mgmt
MASTER_SITES= ftp://ftp.shrubbery.net/pub/rancid/
-PKGNAMESUFFIX= -devel
COMMENT= Really Awesome New Cisco confIg Differ
Only in local/rancid/files: .svn
Only in local/rancid/files: patch-bin::cat5rancid.in
Only in local/rancid/files: patch-bin::clogin.in
Only in local/rancid/files: patch-bin::rancid.in
Only in local/rancid: pkg-message
--
Network Analyst
California State University, Bakersfield
The greatest productive force is human selfishness.
-- Robert Heinlein
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

--
$B%T!<%?!<(B

Eric Cables

2008-12-04 03:37:51 UTC

Well, I went ahead and rebuilt Tcl and Expect, to their latest available
versions, and things appear to be working now. I'm not sure whether it ws
Tcl or Expect causing the problems, but the versions I'm running now seem
stable (two full rancid-run completions so far).

expect-nox11-5.44.1.7
tcl-8.5.4

-- Eric Cables

Post by Peter Serwe
I'm using tcl and tk 8.4.19 and the patched version of expect 5.40.1
from shrubbery.net
on two different installations. I would recommend rebuilding tcl, tk,
expect, and then rancid
and I'm relatively certain the behavior will disappear.
This is, by the way, on a pair of CentOS 5.2 systems. One of them was
4.7 until a recent
drive failure without good backups caused me to rebuild it into a 5.2
box on a new drive and
rebuild all of the software on it.
Peter

Post by Eric Cables
Would upgrading to tcl8.5 be a potential solution? Reading through the

old

Post by Eric Cables
expect threads it indicated that the problem may be with Tcl itself.

Right

Post by Eric Cables
now the system is running tcl-8.4.19.
-- Eric Cables

Post by Eric Cables
This problem has made a resurgence this week, crashing out every night

and

Post by Eric Cables
sending the CPU on the box that RANCID runs on sky high. I end up

killing

Post by Eric Cables
the hung expect processes manually, but instead of staying constant it
appears to be getting worse (dang growing environments).
Any suggestions are welcome...
-- Eric Cables

Post by Eric Cables
I've been running into this problem quite frequently, about once a

week

Post by Eric Cables
or
so rancid will hang for no apparent reason, and until I kill the process it
will remain hung. I've read in the archives that Linux & Solaris

have

Post by Eric Cables
a
problem with expect that requires a patch, but does this also include
FreeBSD?

cisco