[rancid] MikroTik missed cmd(s): system package print detail without-paging

[IT-Info]

Hello,

I've got an issue specifically with MikroTik devices. When RANCID runs, it skips a bunch of MikroTik devices with this type of error:
mikrotik01: missed cmd(s): system package print detail without-paging

If I log in and run rancid-run manually, sometimes it will work and sometimes it won't, but it never works from cron. These MikroTik devices are those that have been replaced due to failure or other reason. I delete the key in .ssh/known_keys, the new key is added upon next run, but then this error appears. Since we've replaced many MikroTiks over the past 6 months, the list of MikroTik devices with this issue is growing - I've got about 20 now.

I upgraded RANCID from 2.3.8 to 3.1, but that didn't help the issue. I'm aware of the issue detailed at http://www.shrubbery.net/pipermail/rancid-discuss/2013-August/007010.html. This issue is similar, but I don't know the solution and I can replicate it after physically replacing a MikroTik device.

I can manually run the command that RANICID errors on without any issues, so it doesn't appear to be a timeout issue:
$ bin/mtlogin -c "system package print detail without-paging" mikrotik01

Has anyone seen this issue or have any advice for me? It's frustrating that when a MikroTik is replaced, it's no longer able to be backed up in RANCID due to this error. Otherwise, RANCID works great, especially with our Juniper and Cisco devices.

Thanks in advance!
Brian

[Tim Warnock]

-----Original Message-----
Behalf Of IT-Info
Sent: Thursday, 8 January 2015 5:28 AM
Subject: [rancid] MikroTik missed cmd(s): system package print detail
without-paging
Hello,
I've got an issue specifically with MikroTik devices. When RANCID runs, it
mikrotik01: missed cmd(s): system package print detail without-paging
If I log in and run rancid-run manually, sometimes it will work and
sometimes it won't, but it never works from cron. These MikroTik devices
are those that have been replaced due to failure or other reason. I delete
the key in .ssh/known_keys, the new key is added upon next run, but then
this error appears. Since we've replaced many MikroTiks over the past 6
months, the list of MikroTik devices with this issue is growing - I've got
about 20 now.

Are you running from cron as the same user as run by hand?

Are these boxes running full tables? How long does an /export take if you run it by hand?

Does a /system package print detail without-paging complete successfully if you run it via ssh?

Is your /system identity dns style or does it have other characters/symbols?

[IT-Info]

Hi Tim,

When I test it manually, I always test using the rancid user, which is what cron also runs as.

No, they're not running full tables.

Yes, a /system package print detail without-paging completes successfully and quickly via ssh.

Yes, the /system identity is composed of alphanumeric characters, as well as periods and dashes.

Again, it seems that the only MikroTiks that have issues are those that have been replaced. I can pretty much guarantee that if a MikroTik router fails and gets replaced, it will no longer get backed up automatically in RANCID due to the error. This has not been the case for Juniper or Cisco devices, those seem to work fine when they're replaced.

Thank you,
Brian

-----Original Message-----
From: Rancid-discuss [mailto:rancid-discuss-***@shrubbery.net] On Behalf Of Tim Warnock
Sent: Wednesday, January 7, 2015 3:17 PM
To: rancid-***@shrubbery.net
Subject: Re: [rancid] MikroTik missed cmd(s): system package print detail without-paging

-----Original Message-----
Behalf Of IT-Info
Sent: Thursday, 8 January 2015 5:28 AM
Subject: [rancid] MikroTik missed cmd(s): system package print detail
without-paging
Hello,
I've got an issue specifically with MikroTik devices. When RANCID runs, it
mikrotik01: missed cmd(s): system package print detail without-paging
If I log in and run rancid-run manually, sometimes it will work and
sometimes it won't, but it never works from cron. These MikroTik devices
are those that have been replaced due to failure or other reason. I delete
the key in .ssh/known_keys, the new key is added upon next run, but then
this error appears. Since we've replaced many MikroTiks over the past 6
months, the list of MikroTik devices with this issue is growing - I've got
about 20 now.

Are you running from cron as the same user as run by hand?

Are these boxes running full tables? How long does an /export take if you run it by hand?

Does a /system package print detail without-paging complete successfully if you run it via ssh?

Is your /system identity dns style or does it have other characters/symbols?

[heasley]

Post by IT-Info
mikrotik01: missed cmd(s): system package print detail without-paging
If I log in and run rancid-run manually, sometimes it will work and sometimes it won't, but it never works from cron. These MikroTik devices are those that have been replaced due to failure or other reason. I delete the key in .ssh/known_keys, the new key is added upon next run, but then this error appears. Since we've replaced many MikroTiks over the past 6 months, the list of MikroTik devices with this issue is growing - I've got about 20 now.
I upgraded RANCID from 2.3.8 to 3.1, but that didn't help the issue. I'm aware of the issue detailed at http://www.shrubbery.net/pipermail/rancid-discuss/2013-August/007010.html. This issue is similar, but I don't know the solution and I can replicate it after physically replacing a MikroTik device.
$ bin/mtlogin -c "system package print detail without-paging" mikrotik01

Would you try this patch? please lmk if it fixes the problem. if it doesnt,
perhaps you can give me remote access to one.

Index: mtlogin.in
===================================================================
--- mtlogin.in (revision 3005)
+++ mtlogin.in (working copy)
@@ -457,6 +457,14 @@
source_password_file $password_file
set in_proc 0
set exitval 0
+# if we have dont have a tty, we need some additional terminal settings
+if [catch {stty -echo} reason] {
+ # no tty, ie: cron
+ set spawnopts "-nottycopy"
+ set stty_init "cols 132"
+} else {
+ catch {stty echo} reason
+}
foreach router [lrange $argv $i end] {
set router [string tolower $router]
send_user "$router\n"

[IT-Info]

Hi Heasley,

I appologize, but how do I apply this patch? Does this need to be appended to the bin/mtlogin.in source file and then recompiled? Can you clarify for me?

Thanks,
Brian

-----Original Message-----
From: heasley [mailto:***@shrubbery.net]
Sent: Wednesday, January 7, 2015 3:37 PM
To: IT-Info
Cc: rancid-***@shrubbery.net
Subject: Re: [rancid] MikroTik missed cmd(s): system package print detail without-paging

Post by IT-Info
mikrotik01: missed cmd(s): system package print detail without-paging
If I log in and run rancid-run manually, sometimes it will work and sometimes it won't, but it never works from cron. These MikroTik devices are those that have been replaced due to failure or other reason. I delete the key in .ssh/known_keys, the new key is added upon next run, but then this error appears. Since we've replaced many MikroTiks over the past 6 months, the list of MikroTik devices with this issue is growing - I've got about 20 now.
I upgraded RANCID from 2.3.8 to 3.1, but that didn't help the issue. I'm aware of the issue detailed at http://www.shrubbery.net/pipermail/rancid-discuss/2013-August/007010.html. This issue is similar, but I don't know the solution and I can replicate it after physically replacing a MikroTik device.
$ bin/mtlogin -c "system package print detail without-paging" mikrotik01

Would you try this patch? please lmk if it fixes the problem. if it doesnt,
perhaps you can give me remote access to one.

Index: mtlogin.in
===================================================================
--- mtlogin.in (revision 3005)
+++ mtlogin.in (working copy)
@@ -457,6 +457,14 @@
source_password_file $password_file
set in_proc 0
set exitval 0
+# if we have dont have a tty, we need some additional terminal settings
+if [catch {stty -echo} reason] {
+ # no tty, ie: cron
+ set spawnopts "-nottycopy"
+ set stty_init "cols 132"
+} else {
+ catch {stty echo} reason
+}
foreach router [lrange $argv $i end] {
set router [string tolower $router]
send_user "$router\n"

[IT-Info]

After patching bin/mtlogin, there seems to be improvement. The devices that were experiencing the error are now being backed up properly.

The logs show many of these messages, though. Is this normal?
are you disconnected or in a batch, at, or cron script?stty: impossible in this context


Thanks,
Brian

[Matthias Cramer]

Hi

This patch seams to fix it for me as well. Can now backup 12 more routers. Thanks.

Regards

Matthias

Post by IT-Info
After patching bin/mtlogin, there seems to be improvement. The devices that were experiencing the error are now being backed up properly.
The logs show many of these messages, though. Is this normal?
are you disconnected or in a batch, at, or cron script?stty: impossible in this context
Thanks,
Brian
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

[heasley]

Post by IT-Info
After patching bin/mtlogin, there seems to be improvement. The devices that were experiencing the error are now being backed up properly.
The logs show many of these messages, though. Is this normal?
are you disconnected or in a batch, at, or cron script?stty: impossible in this context

that is coming from expect...though i do not understand why. catch should
have prevented it. what version of expect is this?

[IT-Info]

Hi Heasley,

expect version 5.44.1.15

Thanks,
Brian

[heasley]

Post by heasley

Post by IT-Info
After patching bin/mtlogin, there seems to be improvement. The devices that were experiencing the error are now being backed up properly.
The logs show many of these messages, though. Is this normal?
are you disconnected or in a batch, at, or cron script?stty: impossible in this context

that is coming from expect...though i do not understand why. catch should
have prevented it. what version of expect is this?
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

Please try this alternative patch:

Index: bin/mtlogin.in
===================================================================
--- bin/mtlogin.in (revision 3012)
+++ bin/mtlogin.in (working copy)
@@ -457,6 +457,15 @@
source_password_file $password_file
set in_proc 0
set exitval 0
+# http://www.shrubbery.net/pipermail/rancid-discuss/2015-January/007984.html
+# if we have dont have a tty, we need some additional terminal settings
+if [catch {open /dev/tty w} ttyid] {
+ # no tty, ie: cron
+ set spawnopts "-nottycopy"
+ set stty_init "cols 132"
+} else {
+ catch {close ttyid} reason
+}
foreach router [lrange $argv $i end] {
set router [string tolower $router]
send_user "$router\n"

[IT-Info]

Hi Heasley,


This patch got rid of that message in the logs. Everything seems to be working well. Thanks for all the help!


Brian

-----Original Message-----
From: heasley [mailto:***@shrubbery.net]
Sent: Friday, January 9, 2015 5:48 PM
To: IT-Info; heasley
Cc: rancid-***@shrubbery.net
Subject: Re: [rancid] MikroTik missed cmd(s): system package print detail without-paging

Post by heasley

Post by IT-Info
After patching bin/mtlogin, there seems to be improvement. The devices that were experiencing the error are now being backed up properly.
The logs show many of these messages, though. Is this normal?
are you disconnected or in a batch, at, or cron script?stty: impossible in this context

that is coming from expect...though i do not understand why. catch should
have prevented it. what version of expect is this?
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

Please try this alternative patch:

Index: bin/mtlogin.in
===================================================================
--- bin/mtlogin.in (revision 3012)
+++ bin/mtlogin.in (working copy)
@@ -457,6 +457,15 @@
source_password_file $password_file
set in_proc 0
set exitval 0
+# http://www.shrubbery.net/pipermail/rancid-discuss/2015-January/007984.html
+# if we have dont have a tty, we need some additional terminal settings
+if [catch {open /dev/tty w} ttyid] {
+ # no tty, ie: cron
+ set spawnopts "-nottycopy"
+ set stty_init "cols 132"
+} else {
+ catch {close ttyid} reason
+}
foreach router [lrange $argv $i end] {
set router [string tolower $router]
send_user "$router\n"