[rancid] run_commands not liking eof/end of session

Forgot to mention jrancid version:

## $Id: jlogin.in,v 1.69 2009/04/16 21:22:57 heas Exp $

Appreciate the help/direction,
Josh

Post by Josh Rogers
I was having trouble with rancid reporting 'hung' and lockfiles
remaining, so I installed the patched/hack expect/tcl in hopes of
resolving it. Ever since, I'm seeing the following error below.
Seems like eof isn't being handled properly when disconnecting, and it
errors out? Any ideas what is leading to this?
$ jlogin -t 120 -c"show chassis clocks;show chassis environment;show
chassis firmware;show chassis fpc detail;show chassis hardware
detail;show chassis routing-engine;show chassis scb;show chassis sfm
detail;show chassis ssb;show chassis feb detail;show chassis feb;show
chassis cfeb;show chassis alarms;show system license;show system
boot-messages;show system core-dumps;show version detail;show
interfaces description;show configuration;show configuration | display
set" rtr2
spawn ssh -c 3des -x -l robot rtr2
<output of show commands>
{master}
Connection to rtr2 closed.
bad spawn_id (process died earlier?)
    while executing
"expect -nobrace {
} { exp_continue } timeout { catch {close}; catch {wait};
                        return 0
                        } eof { return 0 }"
    invoked from within
"expect {
    "\n"                    { exp_continue }
    timeout                    { catch {close}; catch {wait};
                        return 0
                        }
    eof                    { return 0 }
    }"
    (procedure "run_commands" line 21)
    invoked from within
"run_commands $prompt $command"
    ("foreach" body line 68)
    invoked from within
"foreach router [lrange $argv $i end] {
    set router [string tolower $router]
    send_user "$router\n"
    set prompt ">"
    # Figure out usernam..."
    (file "/usr/local/bin/jlogin" line 461)
$ expect -v
expect version 5.43.0
$ which tclsh
$ ls -l /usr/bin/tclsh
lrwxrwxrwx 1 root root 23 Nov 29 16:04 /usr/bin/tclsh -> /etc/alternatives/tclsh
$ ls -l /etc/alternatives/tclsh
lrwxrwxrwx 1 root root 23 Mar 29 11:05 /etc/alternatives/tclsh ->
/usr/local/bin/tclsh8.4
# Run commands given on the command line.
proc run_commands { prompt command } {
    global in_proc
    set in_proc 1
    send "set cli complete-on-space off\r"
    expect -re $prompt {}
    send "set cli screen-length 0\r"
    expect -re $prompt {}
    set commands [split $command \;]
    set num_commands [llength $commands]
    for {set i 0} {$i < $num_commands} { incr i} {
        send "[lindex $commands $i]\r"
        expect {
            -re "^\[^\n\r *]*$prompt $" {}
            -re "^\[^\n\r]*$prompt."    { exp_continue }
            -re "(\r\n|\n)"                     { exp_continue }
        }
    }
    send "quit\r"
    expect {
        "\n"                                    { exp_continue }
        timeout                                 { catch {close}; catch {wait};
                                                  return 0
                                                }
        eof                                     { return 0 }
    }
    set in_proc 0
}
Thanks much,
Josh

john heasley

2011-03-29 22:59:29 UTC

Post by Josh Rogers
## $Id: jlogin.in,v 1.69 2009/04/16 21:22:57 heas Exp $

i dont see anything the commit history that should have affected this.
unless you've modified it in some manner...

Post by Josh Rogers
Appreciate the help/direction,
Josh

Post by Josh Rogers
I was having trouble with rancid reporting 'hung' and lockfiles
remaining, so I installed the patched/hack expect/tcl in hopes of
resolving it.

good.

Post by Josh Rogers
Ever since, I'm seeing the following error below.
Seems like eof isn't being handled properly when disconnecting, and it
errors out?? Any ideas what is leading to this?

what version of expect? on what O/S? the expect_after or the expect after
the quite should deal with the process exiting; it should receive EOF. i'd
suspect a change in expect, perhaps.

Post by Josh Rogers
$ jlogin -t 120 -c"show chassis clocks;show chassis environment;show
chassis firmware;show chassis fpc detail;show chassis hardware
detail;show chassis routing-engine;show chassis scb;show chassis sfm
detail;show chassis ssb;show chassis feb detail;show chassis feb;show
chassis cfeb;show chassis alarms;show system license;show system
boot-messages;show system core-dumps;show version detail;show
interfaces description;show configuration;show configuration | display
set" rtr2
spawn ssh -c 3des -x -l robot rtr2
<output of show commands>
{master}
Connection to rtr2 closed.
bad spawn_id (process died earlier?)
??? while executing
"expect -nobrace {
} { exp_continue } timeout { catch {close}; catch {wait};
??? ??? ??? ??? ??? ??? ? return 0
??? ??? ??? ??? ??? ??? } eof { return 0 }"
??? invoked from within
"expect {
??? "\n"??? ??? ??? ??? ??? { exp_continue }
??? timeout??? ??? ??? ??? ??? { catch {close}; catch {wait};
??? ??? ??? ??? ??? ??? ? return 0
??? ??? ??? ??? ??? ??? }
??? eof??? ??? ??? ??? ??? { return 0 }
??? }"
??? (procedure "run_commands" line 21)
??? invoked from within
"run_commands $prompt $command"
??? ("foreach" body line 68)
??? invoked from within
"foreach router [lrange $argv $i end] {
??? set router [string tolower $router]
??? send_user "$router\n"
??? set prompt ">"
??? # Figure out usernam..."
??? (file "/usr/local/bin/jlogin" line 461)
$ expect -v
expect version 5.43.0
$ which tclsh
$ ls -l /usr/bin/tclsh
lrwxrwxrwx 1 root root 23 Nov 29 16:04 /usr/bin/tclsh -> /etc/alternatives/tclsh
$ ls -l /etc/alternatives/tclsh
lrwxrwxrwx 1 root root 23 Mar 29 11:05 /etc/alternatives/tclsh ->
/usr/local/bin/tclsh8.4
# Run commands given on the command line.
proc run_commands { prompt command } {
??? global in_proc
??? set in_proc 1
??? send "set cli complete-on-space off\r"
??? expect -re $prompt? {}
??? send "set cli screen-length 0\r"
??? expect -re $prompt? {}
??? set commands [split $command \;]
??? set num_commands [llength $commands]
??? for {set i 0} {$i < $num_commands} { incr i} {
??????? send "[lindex $commands $i]\r"
??????? expect {
??????????? -re "^\[^\n\r *]*$prompt $" {}
??????????? -re "^\[^\n\r]*$prompt."??? { exp_continue }
??????????? -re "(\r\n|\n)"???????????????????? { exp_continue }
??????? }
??? }
??? send "quit\r"
??? expect {
??????? "\n"??????????????????????????????????? { exp_continue }
??????? timeout???????????????????????????????? { catch {close}; catch {wait};
????????????????????????????????????????????????? return 0
??????????????????????????????????????????????? }
??????? eof???????????????????????????????????? { return 0 }
??? }
??? set in_proc 0
}
Thanks much,
Josh

_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

Josh Rogers

2011-03-30 03:04:02 UTC

Post by Josh Rogers
$ expect -v
expect version 5.43.0

Ubuntu 10.04. Originally I had the ubuntu maintainers version of
expect/tcl installed, but the above was downloaded and compiled from
shrubbery.net. When looking closer, it appears that ssh is being
spawned twice?

$ jlogin rtr1 "show system users"
rtr1
spawn ssh -c 3des -x -l robot rtr1
The authenticity of host 'rtr1 (192.168.240.1)' can't be established.
RSA key fingerprint is <>.
Are you sure you want to continue connecting (yes/no)?
Host rtr1 added to the list of known hosts.
yes
Warning: Permanently added 'rtr1,192.168.240.1' (RSA) to the list of
known hosts.

***@rtr1's password:
--- JUNOS 9.2R3.5 built 2009-01-15 04:36:25 UTC
{master}
***@rtr1> #Interacting?

{master}
***@rtr1> q
show system users
spawn ssh -c 3des -x -l robot show system users
bad spawn_id (process died earlier?)
while executing
"expect -nobrace -re {(Connection refused|Secure connection [^
]+ refused|Connection closed by)} {
catch {close}; catch {wait};
if !$progs {
..."
invoked from within
"expect {
-re "(Connection refused|Secure connection \[^\n\r]+
refused|Connection closed by)" {
catch {close}; catch {wait};
if !$progs {
..."
(procedure "login" line 75)
invoked from within
"login $router $loginname $passwd $cmethod $cyphertype $identfile"
("foreach" body line 62)
invoked from within
"foreach router [lrange $argv $i end] {
set router [string tolower $router]
send_user "$router\n"

set prompt ">"

# Figure out usernam..."
(file "/usr/local/bin/jlogin" line 461)

I do not believe jlogin is modified, however I have modified jrancid
(which isn't even called above), and I didn't have this issue on the
previous version of expect (5.44.1.14) and tcl (8.5.8-2)

Here is my jlogin for reference: http://pastebin.com/FeqYqYPq

Thanks again for helping with this before I have to rollback expect and tcl.

-Josh

Post by Josh Rogers
## $Id: jlogin.in,v 1.69 2009/04/16 21:22:57 heas Exp $

i dont see anything the commit history that should have affected this.
unless you've modified it in some manner...

Post by Josh Rogers
Appreciate the help/direction,
Josh

Post by Josh Rogers
I was having trouble with rancid reporting 'hung' and lockfiles
remaining, so I installed the patched/hack expect/tcl in hopes of
resolving it.

good.

Post by Josh Rogers
Ever since, I'm seeing the following error below.
Seems like eof isn't being handled properly when disconnecting, and it
errors out?? Any ideas what is leading to this?

what version of expect? on what O/S? the expect_after or the expect after
the quite should deal with the process exiting; it should receive EOF. i'd
suspect a change in expect, perhaps.

_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

Josh Rogers

2011-03-30 04:21:22 UTC

Once again, with the -c switch (nevermind the comment about spawning ssh twice)

$ jlogin -c "show system users" rtr1
rtr1
spawn ssh -c 3des -x -l robot rtr1

***@rtr1's password:
--- JUNOS 9.2R3.5 built 2009-01-15 04:36:25 UTC
{master}
***@rtr1>

{master}
***@rtr1> set cli complete-on-space off
Disabling complete-on-space

{master}
***@rtr1> set cli screen-length 0
Screen length set to 0

{master}
***@rtr1> show system users
11:18PM up 750 days, 7:02, 2 users, load averages: 0.72, 0.68, 0.68
USER TTY FROM LOGIN@ IDLE WHAT
robot p1 web01 11:18PM - -cli (cli)

{master}
***@rtr1> bad spawn_id (process died earlier?)
while executing
"expect -nobrace {
} { exp_continue } timeout { catch {close}; catch {wait};
return 0
} eof { return 0 }"
invoked from within
"expect {
"\n" { exp_continue }
timeout { catch {close}; catch {wait};
return 0
}
eof { return 0 }
}"
(procedure "run_commands" line 21)
invoked from within
"run_commands $prompt $command"
("foreach" body line 68)
invoked from within
"foreach router [lrange $argv $i end] {
set router [string tolower $router]
send_user "$router\n"

set prompt ">"

# Figure out usernam..."
(file "/usr/local/bin/jlogin" line 461)

Post by Josh Rogers
$ expect -v
expect version 5.43.0

Ubuntu 10.04. Originally I had the ubuntu maintainers version of
expect/tcl installed, but the above was downloaded and compiled from
shrubbery.net. When looking closer, it appears that ssh is being
spawned twice?
$ jlogin rtr1 "show system users"
rtr1
spawn ssh -c 3des -x -l robot rtr1
The authenticity of host 'rtr1 (192.168.240.1)' can't be established.
RSA key fingerprint is <>.
Are you sure you want to continue connecting (yes/no)?
Host rtr1 added to the list of known hosts.
yes
Warning: Permanently added 'rtr1,192.168.240.1' (RSA) to the list of
known hosts.
--- JUNOS 9.2R3.5 built 2009-01-15 04:36:25 UTC
{master}
{master}
show system users
spawn ssh -c 3des -x -l robot show system users
bad spawn_id (process died earlier?)
while executing
"expect -nobrace -re {(Connection refused|Secure connection [^
]+ refused|Connection closed by)} {
catch {close}; catch {wait};
if !$progs {
..."
invoked from within
"expect {
-re "(Connection refused|Secure connection \[^\n\r]+
refused|Connection closed by)" {
catch {close}; catch {wait};
if !$progs {
..."
(procedure "login" line 75)
invoked from within
"login $router $loginname $passwd $cmethod $cyphertype $identfile"
("foreach" body line 62)
invoked from within
"foreach router [lrange $argv $i end] {
set router [string tolower $router]
send_user "$router\n"
set prompt ">"
# Figure out usernam..."
(file "/usr/local/bin/jlogin" line 461)
I do not believe jlogin is modified, however I have modified jrancid
(which isn't even called above), and I didn't have this issue on the
previous version of expect (5.44.1.14) and tcl (8.5.8-2)
Here is my jlogin for reference: http://pastebin.com/FeqYqYPq
Thanks again for helping with this before I have to rollback expect and tcl.
-Josh

Post by Josh Rogers
## $Id: jlogin.in,v 1.69 2009/04/16 21:22:57 heas Exp $

i dont see anything the commit history that should have affected this.
unless you've modified it in some manner...

Post by Josh Rogers
Appreciate the help/direction,
Josh

Post by Josh Rogers
I was having trouble with rancid reporting 'hung' and lockfiles
remaining, so I installed the patched/hack expect/tcl in hopes of
resolving it.

good.

Post by Josh Rogers
Ever since, I'm seeing the following error below.
Seems like eof isn't being handled properly when disconnecting, and it
errors out?? Any ideas what is leading to this?

what version of expect? on what O/S? the expect_after or the expect after
the quite should deal with the process exiting; it should receive EOF. i'd
suspect a change in expect, perhaps.

_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

Josh Rogers

2011-03-30 04:53:19 UTC

I was questioning whether I had made any modifications to jlogin (I
typically make a copy of the original and link to a new version if I
do, which I had not), so I replaced my 1.69 jlogin with a 1.62 jlogin
I found in a debian repository. Same results as before. If this is a
result of any modifications I've made, it would be to a file besides
jlogin.

I left the original expect/tcl installed, and just updated symlinks to
the new ones:

$ ls -la /usr/bin/@(tclsh|expect)
lrwxrwxrwx 1 root root 24 Mar 29 11:12 /usr/bin/expect ->
/etc/alternatives/expect
lrwxrwxrwx 1 root root 23 Nov 29 16:04 /usr/bin/tclsh -> /etc/alternatives/tclsh
$ ls -la /etc/alternatives/@(tclsh|expect)
lrwxrwxrwx 1 root root 21 Mar 29 23:49 /etc/alternatives/expect ->
/usr/local/bin/expect
lrwxrwxrwx 1 root root 23 Mar 29 23:49 /etc/alternatives/tclsh ->
/usr/local/bin/tclsh8.4

I would expect this to work just fine.

Thanks again, for the extra set(s) of eyes.

Post by Josh Rogers
Once again, with the -c switch (nevermind the comment about spawning ssh twice)
$ jlogin -c "show system users" rtr1
rtr1
spawn ssh -c 3des -x -l robot rtr1
--- JUNOS 9.2R3.5 built 2009-01-15 04:36:25 UTC
{master}
{master}
Disabling complete-on-space
{master}
Screen length set to 0
{master}
11:18PM up 750 days, 7:02, 2 users, load averages: 0.72, 0.68, 0.68
robot p1 web01 11:18PM - -cli (cli)
{master}
while executing
"expect -nobrace {
} { exp_continue } timeout { catch {close}; catch {wait};
return 0
} eof { return 0 }"
invoked from within
"expect {
"\n" { exp_continue }
timeout { catch {close}; catch {wait};
return 0
}
eof { return 0 }
}"
(procedure "run_commands" line 21)
invoked from within
"run_commands $prompt $command"
("foreach" body line 68)
invoked from within
"foreach router [lrange $argv $i end] {
set router [string tolower $router]
send_user "$router\n"
set prompt ">"
# Figure out usernam..."
(file "/usr/local/bin/jlogin" line 461)

Post by Josh Rogers
$ expect -v
expect version 5.43.0

Ubuntu 10.04. Originally I had the ubuntu maintainers version of
expect/tcl installed, but the above was downloaded and compiled from
shrubbery.net. When looking closer, it appears that ssh is being
spawned twice?
$ jlogin rtr1 "show system users"
rtr1
spawn ssh -c 3des -x -l robot rtr1
The authenticity of host 'rtr1 (192.168.240.1)' can't be established.
RSA key fingerprint is <>.
Are you sure you want to continue connecting (yes/no)?
Host rtr1 added to the list of known hosts.
yes
Warning: Permanently added 'rtr1,192.168.240.1' (RSA) to the list of
known hosts.
--- JUNOS 9.2R3.5 built 2009-01-15 04:36:25 UTC
{master}
{master}
show system users
spawn ssh -c 3des -x -l robot show system users
bad spawn_id (process died earlier?)
while executing
"expect -nobrace -re {(Connection refused|Secure connection [^
]+ refused|Connection closed by)} {
catch {close}; catch {wait};
if !$progs {
..."
invoked from within
"expect {
-re "(Connection refused|Secure connection \[^\n\r]+
refused|Connection closed by)" {
catch {close}; catch {wait};
if !$progs {
..."
(procedure "login" line 75)
invoked from within
"login $router $loginname $passwd $cmethod $cyphertype $identfile"
("foreach" body line 62)
invoked from within
"foreach router [lrange $argv $i end] {
set router [string tolower $router]
send_user "$router\n"
set prompt ">"
# Figure out usernam..."
(file "/usr/local/bin/jlogin" line 461)
I do not believe jlogin is modified, however I have modified jrancid
(which isn't even called above), and I didn't have this issue on the
previous version of expect (5.44.1.14) and tcl (8.5.8-2)
Here is my jlogin for reference: http://pastebin.com/FeqYqYPq
Thanks again for helping with this before I have to rollback expect and tcl.
-Josh

Post by Josh Rogers
## $Id: jlogin.in,v 1.69 2009/04/16 21:22:57 heas Exp $

i dont see anything the commit history that should have affected this.
unless you've modified it in some manner...

Post by Josh Rogers
Appreciate the help/direction,
Josh

Post by Josh Rogers
I was having trouble with rancid reporting 'hung' and lockfiles
remaining, so I installed the patched/hack expect/tcl in hopes of
resolving it.

good.

Post by Josh Rogers
Ever since, I'm seeing the following error below.
Seems like eof isn't being handled properly when disconnecting, and it
errors out?? Any ideas what is leading to this?

what version of expect? on what O/S? the expect_after or the expect after
the quite should deal with the process exiting; it should receive EOF. i'd
suspect a change in expect, perhaps.