[rancid] duplicated characters

[Corey Edwards]

After a recent upgrade from Debian 5.0 to 6.0 I began seeing duplicate
characters in config diffs. Here's an example:

- access-list 102 deny udp any any eq 135
+ access-list 102 denny udp any any eq 135
- access-list 110 deny tcp any any eq 445 log
+ access-list 110 deny tcp any any eq 445 log

The next round of updates will revert the change back. I haven't found
any pattern to which character it will be or even which router. It's
only happening with my Cisco devices, not any of our others (Foundry
switches, Mikrotiks or Lucent TNTs). I'm using 2.3.1, but a fresh
install of 2.3.6 on the same server does the same thing. 2.3.6 on my
Ubuntu 10.04 laptop works just fine.

To verify whether the routers themselves were errant, I did a packet
capture. That confirmed that the data from the router is correct. I
turned on debugging in clogin. WriteTerm shows the duplicate characters.
So then I put in a puts in run_commands in rancid and that also showed
duplicate characters in $expect_out(buffer).

I found this thread from 2004 which hinted at it being an issue with the
telnet client. The OP eventually switched to ssh which in this case
isn't an option.

http://www.gossamer-threads.com/lists/rancid/users/720

The incessant emails are driving me batty and making the cvs history
significantly less useful. I'll debug whatever needs to be debugged but
I'm fresh out of ideas of where to look. I started looking at expect but
was quickly lost in the code.

Any ideas?

Corey

[john heasley]

Post by Corey Edwards
After a recent upgrade from Debian 5.0 to 6.0 I began seeing duplicate
- access-list 102 deny udp any any eq 135
+ access-list 102 denny udp any any eq 135
- access-list 110 deny tcp any any eq 445 log
+ access-list 110 deny tcp any any eq 445 log
The next round of updates will revert the change back. I haven't found
any pattern to which character it will be or even which router. It's
only happening with my Cisco devices, not any of our others (Foundry
switches, Mikrotiks or Lucent TNTs). I'm using 2.3.1, but a fresh
install of 2.3.6 on the same server does the same thing. 2.3.6 on my
Ubuntu 10.04 laptop works just fine.
To verify whether the routers themselves were errant, I did a packet
capture. That confirmed that the data from the router is correct. I
turned on debugging in clogin. WriteTerm shows the duplicate characters.
So then I put in a puts in run_commands in rancid and that also showed
duplicate characters in $expect_out(buffer).
I found this thread from 2004 which hinted at it being an issue with the
telnet client. The OP eventually switched to ssh which in this case
isn't an option.
http://www.gossamer-threads.com/lists/rancid/users/720

i dont know what the fbsd fix was for this, but they fixed it somehow in
their tcl or expect port.

perhaps its related to a problem that came up in rancid 2.3.5 something
changed in debian's libc that caused problems with string handling
functions, which i believe was related to internationalization.

or could be a bug in buffer handling in expect or tcl.

either way, if you can capture clogin -d -c 'show running' o/p where this
occurs, it should at least reveal where the bug is.

you can also try rebuilding tcl and expect w/o internationalization.

Post by Corey Edwards
The incessant emails are driving me batty and making the cvs history
significantly less useful. I'll debug whatever needs to be debugged but
I'm fresh out of ideas of where to look. I started looking at expect but
was quickly lost in the code.
Any ideas?
Corey
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

[Corey Edwards]

Post by john heasley

Post by Corey Edwards
After a recent upgrade from Debian 5.0 to 6.0 I began seeing duplicate
- access-list 102 deny udp any any eq 135
+ access-list 102 denny udp any any eq 135
- access-list 110 deny tcp any any eq 445 log
+ access-list 110 deny tcp any any eq 445 log
I found this thread from 2004 which hinted at it being an issue with the
telnet client. The OP eventually switched to ssh which in this case
isn't an option.
http://www.gossamer-threads.com/lists/rancid/users/720

i dont know what the fbsd fix was for this, but they fixed it somehow in
their tcl or expect port.
perhaps its related to a problem that came up in rancid 2.3.5 something
changed in debian's libc that caused problems with string handling
functions, which i believe was related to internationalization.
or could be a bug in buffer handling in expect or tcl.

I moved the RANCID install to a separate server running Debian 6.0 and
it's working just fine, so it appears to be something left over from the
upgrade from 5.0. I just can't imagine what it is.

Post by john heasley
either way, if you can capture clogin -d -c 'show running' o/p where this
occurs, it should at least reveal where the bug is.

I captured this output, but unfortunately it was rife with passwords. I
trimmed it down to what I hope is still a useful summary. In this
example the duplicated character showed up in the word "permitt".

http://pastebin.com/epUCPr2p

Post by john heasley
you can also try rebuilding tcl and expect w/o internationalization.

I could not find any way to do this.

Corey

[john heasley]

Post by Corey Edwards

Post by john heasley
either way, if you can capture clogin -d -c 'show running' o/p where this
occurs, it should at least reveal where the bug is.

I captured this output, but unfortunately it was rife with passwords. I
trimmed it down to what I hope is still a useful summary. In this
example the duplicated character showed up in the word "permitt".
http://pastebin.com/epUCPr2p

-expect: does "access-list 1 permit x.x.x.101\r\naccess-li" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no
+expect: does "access-list 1 permitt x.x.x.101\r\naccess-l" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no

its occuring inside tcl or expect. i suspect its the same problem that i
had with hpuifilter when some linux users started upgrading. I dont know
what the cause is, but i'd look for old versions of libc and internationization
libraries that tcl or expect are linked with.

Post by Corey Edwards

Post by john heasley
you can also try rebuilding tcl and expect w/o internationalization.

I could not find any way to do this.

hrm, there doesnt appear to be an easy way to disable it in expect.

[Corey Edwards]

Post by john heasley

Post by Corey Edwards

Post by john heasley
either way, if you can capture clogin -d -c 'show running' o/p where this
occurs, it should at least reveal where the bug is.

I captured this output, but unfortunately it was rife with passwords. I
trimmed it down to what I hope is still a useful summary. In this
example the duplicated character showed up in the word "permitt".
http://pastebin.com/epUCPr2p

-expect: does "access-list 1 permit x.x.x.101\r\naccess-li" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no
+expect: does "access-list 1 permitt x.x.x.101\r\naccess-l" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no
its occuring inside tcl or expect. i suspect its the same problem that i
had with hpuifilter when some linux users started upgrading. I dont know
what the cause is, but i'd look for old versions of libc and internationization
libraries that tcl or expect are linked with.

I ran clogin through strace and looked at all the files that were
touched. A comparison between the working server and the upgraded server
revealed some kerberos packages (libkrb5-dev, krb5-config,
krb5-clients). After removing those, everything is working perfectly. I
guess a kerberized version of telnet must have been triggering this bug.

Corey

[john heasley]

Post by Corey Edwards

Post by john heasley
-expect: does "access-list 1 permit x.x.x.101\r\naccess-li" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no
+expect: does "access-list 1 permitt x.x.x.101\r\naccess-l" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no
its occuring inside tcl or expect. i suspect its the same problem that i
had with hpuifilter when some linux users started upgrading. I dont know
what the cause is, but i'd look for old versions of libc and internationization
libraries that tcl or expect are linked with.

I ran clogin through strace and looked at all the files that were
touched. A comparison between the working server and the upgraded server
revealed some kerberos packages (libkrb5-dev, krb5-config,
krb5-clients). After removing those, everything is working perfectly. I
guess a kerberized version of telnet must have been triggering this bug.

ah, there is a telnet that is installed with the heimdal package on bsd
that weird stuff too; unsolicited, it prints to the controlling tty in
rather random way, which confuses the login process if the timing is
right.