[rancid] Rollback functionality and potential pitfalls

[Ramon]

Hello all,

I am currently looking in to a way to implement rollback functionality
using rancid. What would it take to reverse bad changes and restore the
config of a cisco device back to a previous saved version?

I ask because just pushing the original config on top of the modified
version would not remove or flush out certain statements, possibly leaving
duplicates that would have to be removed manually.

My first idea was to create a negative file, by generating a "no statement"
for every line in the new config that does not match the old config. After
pushing the negative file and removing the changes I would push the old
config to restore any of the deleted statements.

Feedback on any foreseeable issues such as possible hierarchical problems
(interfaces, acls, bgp) would be very welcomed.

Thanks,
Ramon

[Miles Lott]

Seems like it might be better to just overwrite running or startup
configs using copy from tftp. Or maybe I am missing your point.

Post by Ramon
Hello all,
I am currently looking in to a way to implement rollback functionality
using rancid. What would it take to reverse bad changes and restore
the config of a cisco device back to a previous saved version?
I ask because just pushing the original config on top of the modified
version would not remove or flush out certain statements, possibly
leaving duplicates that would have to be removed manually.
My first idea was to create a negative file, by generating a "no
statement" for every line in the new config that does not match the
old config. After pushing the negative file and removing the changes I
would push the old config to restore any of the deleted statements.
Feedback on any foreseeable issues such as possible hierarchical
problems (interfaces, acls, bgp) would be very welcomed.
Thanks,
Ramon

--------------------------------------------------------------------------------
Privileged, Proprietary and/or Confidential Information may be contained in
this electronic message. If you are not the intended recipient, you should
not copy it, re-transmit it, use it or disclose its contents, but should
kindly return to the sender immediately and delete your copy from your system.
Gulf Interstate Engineering Company does not accept responsibility for
changes made to this electronic message or its attachments.

[andrew.brennan+]

One complication would be cases where a command's default state was one that
isn't normally displayed in the configs. You could do the CVS diff between
good/bad configs and then parse for +/- to remove/add any change that was
shown. Then, parse the "no shutdown" so that you weren't doing something like
"no no shutdown" (remove 'no' when present, add 'no' for other diff'd lines).

That *might* address the non-displayed default config lines, but I'm not sure
I've thought through this completely yet. You'll still have issues where lines
are eliminated from the RANCID store (passwds, etc.).

The other option would be to create a copy of the old/good config file and copy
it directly to startup-config for the next reboot. You still have an issue
with lines eliminated by RANCID, though.

andrew.

Post by Ramon
Hello all,
I am currently looking in to a way to implement rollback functionality using
rancid. What would it take to reverse bad changes and restore the config of
a cisco device back to a previous saved version?
I ask because just pushing the original config on top of the modified
version would not remove or flush out certain statements, possibly leaving
duplicates that would have to be removed manually.
My first idea was to create a negative file, by generating a "no statement"
for every line in the new config that does not match the old config. After
pushing the negative file and removing the changes I would push the old
config to restore any of the deleted statements.
Feedback on any foreseeable issues such as possible hierarchical problems
(interfaces, acls, bgp) would be very welcomed.
Thanks,
Ramon

[heasley]

Post by andrew.brennan+
One complication would be cases where a command's default state was one that
isn't normally displayed in the configs. You could do the CVS diff between

many commands/blocks now have a default command (default interface ...). I
dont know how well it works (i expect poorly), but if you can tolerate service
interrupts that it will cause, that will at least provide a clean slate on
which to apply the portion from the saved config.

[Skye Hagen]

You may want to look into the Cisco IOS 'configure replace' command. This
will replace the running configuration with another configuration. It is NOT
a merge operation like 'copy ... Running-config'. It can work using tftp,
ftp, scp, etc.

Configure replace will work with whole configurations only. If you are
removing passwords in RANCID, you would need to replace them beforehand,
otherwise they will be missing when you replace the config.

Skye.

Post by andrew.brennan+
One complication would be cases where a command's default state was one that
isn't normally displayed in the configs. You could do the CVS diff between
good/bad configs and then parse for +/- to remove/add any change that was
shown. Then, parse the "no shutdown" so that you weren't doing something like
"no no shutdown" (remove 'no' when present, add 'no' for other diff'd lines).
That *might* address the non-displayed default config lines, but I'm not sure
I've thought through this completely yet. You'll still have issues where
lines
are eliminated from the RANCID store (passwds, etc.).
The other option would be to create a copy of the old/good config file and
copy
it directly to startup-config for the next reboot. You still have an issue
with lines eliminated by RANCID, though.
andrew.

Post by Ramon
Hello all,
I am currently looking in to a way to implement rollback functionality using
rancid. What would it take to reverse bad changes and restore the config of
a cisco device back to a previous saved version?
I ask because just pushing the original config on top of the modified
version would not remove or flush out certain statements, possibly leaving
duplicates that would have to be removed manually.
My first idea was to create a negative file, by generating a "no statement"
for every line in the new config that does not match the old config. After
pushing the negative file and removing the changes I would push the old
config to restore any of the deleted statements.
Feedback on any foreseeable issues such as possible hierarchical problems
(interfaces, acls, bgp) would be very welcomed.
Thanks,
Ramon

_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

[shouldbe q931]

Post by Skye Hagen
You may want to look into the Cisco IOS 'configure replace' command. This
will replace the running configuration with another configuration. It is NOT
a merge operation like 'copy ... Running-config'. It can work using tftp,
ftp, scp, etc.
Configure replace will work with whole configurations only. If you are
removing passwords in RANCID, you would need to replace them beforehand,
otherwise they will be missing when you replace the config.
Skye.

Thank you for making me feel like an idiot for not knowing about configure
replace :-)

Cheers

[Ramon]

heasley; that is a good idea and I have definitely have tried that... the
problem I am having however with the banner at this point is with clogin or
"expect" perhaps: newlines (line breaks) after each line in the banner
statement dont return any meaningful character such as ">", "#" to the
clogin script, so it just halts.

I dont think it is a bug, but it would be very good to be able use the -x
option with clogin and push a complete config files (including banner).
Perhaps the solution is for clogin to push lines after a banner statement
differently "knowing" that until it reaches the delimiter character it can
continue to paste lines without expecting the hash.

shouldb, excellent idea to write the date and message about rollback. That
is a easy and useful one-liner (the limit a this point)

I also got very interested when I found about configure replace... my
success rate using on production so far was 1 out of 3. Yesterday I saw
another (undocumented new) failure:

"Error: could not write to output buffer incremental-temp.cfg"

Post by shouldbe q931

Post by Skye Hagen
You may want to look into the Cisco IOS 'configure replace' command. This
will replace the running configuration with another configuration. It is NOT
a merge operation like 'copy ... Running-config'. It can work using tftp,
ftp, scp, etc.
Configure replace will work with whole configurations only. If you are
removing passwords in RANCID, you would need to replace them beforehand,
otherwise they will be missing when you replace the config.
Skye.

Thank you for making me feel like an idiot for not knowing about configure
replace :-)
Cheers
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

[heasley]

Post by Ramon
heasley; that is a good idea and I have definitely have tried that... the
problem I am having however with the banner at this point is with clogin or
"expect" perhaps: newlines (line breaks) after each line in the banner
statement dont return any meaningful character such as ">", "#" to the
clogin script, so it just halts.
I dont think it is a bug, but it would be very good to be able use the -x
option with clogin and push a complete config files (including banner).

you should be able to send \n instead. or, just eliminate or ignore the
banner.

Post by Ramon
I also got very interested when I found about configure replace... my
success rate using on production so far was 1 out of 3. Yesterday I saw

afaik, its a relatively new feature for ios.

[shouldbe q931]

Post by Ramon
Hello all,
I am currently looking in to a way to implement rollback functionality
using rancid. What would it take to reverse bad changes and restore the
config of a cisco device back to a previous saved version?
I ask because just pushing the original config on top of the modified
version would not remove or flush out certain statements, possibly leaving
duplicates that would have to be removed manually.
My first idea was to create a negative file, by generating a "no
statement" for every line in the new config that does not match the old
config. After pushing the negative file and removing the changes I would
push the old config to restore any of the deleted statements.
Feedback on any foreseeable issues such as possible hierarchical problems
(interfaces, acls, bgp) would be very welcomed.
Thanks,
Ramon

A few problems that I thought of

If you're doing a "conf t" and then sending the updates, you need to be
able to parse and escape "special" characters, such as the ? in line three
below

1 ip ddns update method dyn.he.net
2 HTTP
3 add
http://dynamicrouter.domain.com:***@ipv4.dyn.dns.he.net/nic/update?hostname=
<h>&myip=<a>

The thought of escaping characters lead me onto motd lines...

banner motd ^C
This is not the router you are looking for
^C

And finally, the order of NAT rules on an ASA can be critical, this would
mean either removing all of them (affecting all traffic) and then adding
all of them, or being able to parse all of the NAT rule lines to add the
sequence to the rollback, such as

no nat 14
nat (inside,outside) 14 source static i-server
i-server.domain.comservice tcp-in-https tcp-in-https

I'm sure there are lots of other similar cases

If you can cope with a reload, then would go with either a tftp boot, or
erase startup, tftp copy to startup and then reload.

As well as rancid, I also like to archive to a FTP server, so when TFTP
isn't an option (such as a remote site router, I can simply go

erase start
copy ftp://user:***@host/path/file startup

And then reload

For a client that bought CSM, I manage their ASA firewalls with CSM, after
a particularly bad experience with the built in CSM rollback, I now do the
changes and deploy normally rather than a rollback deploy if changes need
to be "reverted".

Cheers

[Ramon]

shouldbe, thanks.... that is exactly what I was looking for. Characters "?"
, "$" and "^C" all produce unwanted expansion. The "?" throws a WORD in the
example you cite, breaking the config script. The "$" in a secret or
password string breaks out the clogin process with because it tries to read
it as a variable:

can't read "1": no such variable
while executing
"subst -nocommands [lindex $commands $i]"
(procedure "run_commands" line 26)
invoked from within
"run_commands $prompt $command"
("foreach" body line 186)
invoked from within
"foreach router [lrange $argv $i end] {
set router [string tolower $router]
# attempt at platform switching.
set platform ""
send_user ..."
(file "/usr/local/rancid/bin/clogin" line 740)

And the "^C" on the motd enters interactive mode and eventually times out:

#banner login ^C
Enter TEXT message. End with the character '^'.

Error: TIMEOUT reached


I took the ASA out of the scope yesterday when I noticed what a mess that
would be. So right now I am only trying to make this work for Cisco routers
and switches (2911's and 2950's specifically). Router reload in completely
out of question, and configure replace has proved to be risky at times
(when the command is scripted you have to use force, and it has trimmed
correct vlans and left old ones in place).

I'm going to dig thru clogin code to see if I can come up with a workaround
for the parsing problems mentioned above, any help appreciated!

Ramon

Post by shouldbe q931

Post by Ramon
Hello all,
I am currently looking in to a way to implement rollback functionality
using rancid. What would it take to reverse bad changes and restore the
config of a cisco device back to a previous saved version?
I ask because just pushing the original config on top of the modified
version would not remove or flush out certain statements, possibly leaving
duplicates that would have to be removed manually.
My first idea was to create a negative file, by generating a "no
statement" for every line in the new config that does not match the old
config. After pushing the negative file and removing the changes I would
push the old config to restore any of the deleted statements.
Feedback on any foreseeable issues such as possible hierarchical problems
(interfaces, acls, bgp) would be very welcomed.
Thanks,
Ramon

A few problems that I thought of
If you're doing a "conf t" and then sending the updates, you need to be
able to parse and escape "special" characters, such as the ? in line three
below
1 ip ddns update method dyn.he.net
2 HTTP
3 add
<h>&myip=<a>
The thought of escaping characters lead me onto motd lines...
banner motd ^C
This is not the router you are looking for
^C
And finally, the order of NAT rules on an ASA can be critical, this would
mean either removing all of them (affecting all traffic) and then adding
all of them, or being able to parse all of the NAT rule lines to add the
sequence to the rollback, such as
no nat 14
nat (inside,outside) 14 source static i-server i-server.domain.comservice tcp-in-https tcp-in-https
I'm sure there are lots of other similar cases
If you can cope with a reload, then would go with either a tftp boot, or
erase startup, tftp copy to startup and then reload.
As well as rancid, I also like to archive to a FTP server, so when TFTP
isn't an option (such as a remote site router, I can simply go
erase start
And then reload
For a client that bought CSM, I manage their ASA firewalls with CSM, after
a particularly bad experience with the built in CSM rollback, I now do the
changes and deploy normally rather than a rollback deploy if changes need
to be "reverted".
Cheers

[Ramon]

This post might be inappropriate. Click to display it.

[heasley]

Post by Ramon
So far I have been able to to bypass the "$" expansion with sed, by adding
a "\" in front of every occurrence.
The question mark "?" is possible to be bypassed by using "Ctrl-V". However
I could not find a way for to add that control sequence (like we can do
with "\n" or "\r" ) to a script line nor make clogin inject that before an
instance of "?".
The banner login and motd I could only make it work with clogin if I use
just one line for the text immediately followed by the delimiter character.
Any newlines will make the clogin script hang, which I suspect is because
expect is waiting for some shell return string or character.
Since most banners are several lines, right now I am having to trim the
banner statements out completely to avoid the problem. Anyone could give me
some direction as to how I could make clogin play ball both with Ctrl-V and
with banner text newlines?

use any char not in the banner. eg: =

Post by Ramon
Thanks,
Ramon

Post by Ramon
shouldbe, thanks.... that is exactly what I was looking for. Characters
"?" , "$" and "^C" all produce unwanted expansion. The "?" throws a WORD in
the example you cite, breaking the config script. The "$" in a secret or
password string breaks out the clogin process with because it tries to read
can't read "1": no such variable
while executing
"subst -nocommands [lindex $commands $i]"
(procedure "run_commands" line 26)
invoked from within
"run_commands $prompt $command"
("foreach" body line 186)
invoked from within
"foreach router [lrange $argv $i end] {
set router [string tolower $router]
# attempt at platform switching.
set platform ""
send_user ..."
(file "/usr/local/rancid/bin/clogin" line 740)
#banner login ^C
Enter TEXT message. End with the character '^'.
Error: TIMEOUT reached
I took the ASA out of the scope yesterday when I noticed what a mess that
would be. So right now I am only trying to make this work for Cisco routers
and switches (2911's and 2950's specifically). Router reload in completely
out of question, and configure replace has proved to be risky at times
(when the command is scripted you have to use force, and it has trimmed
correct vlans and left old ones in place).
I'm going to dig thru clogin code to see if I can come up with a
workaround for the parsing problems mentioned above, any help appreciated!
Ramon

Post by shouldbe q931

Post by Ramon
Hello all,
I am currently looking in to a way to implement rollback functionality
using rancid. What would it take to reverse bad changes and restore the
config of a cisco device back to a previous saved version?
I ask because just pushing the original config on top of the modified
version would not remove or flush out certain statements, possibly leaving
duplicates that would have to be removed manually.
My first idea was to create a negative file, by generating a "no
statement" for every line in the new config that does not match the old
config. After pushing the negative file and removing the changes I would
push the old config to restore any of the deleted statements.
Feedback on any foreseeable issues such as possible hierarchical
problems (interfaces, acls, bgp) would be very welcomed.
Thanks,
Ramon

A few problems that I thought of
If you're doing a "conf t" and then sending the updates, you need to be
able to parse and escape "special" characters, such as the ? in line three
below
1 ip ddns update method dyn.he.net
2 HTTP
3 add
<h>&myip=<a>
The thought of escaping characters lead me onto motd lines...
banner motd ^C
This is not the router you are looking for
^C
And finally, the order of NAT rules on an ASA can be critical, this would
mean either removing all of them (affecting all traffic) and then adding
all of them, or being able to parse all of the NAT rule lines to add the
sequence to the rollback, such as
no nat 14
nat (inside,outside) 14 source static i-server i-server.domain.comservice tcp-in-https tcp-in-https
I'm sure there are lots of other similar cases
If you can cope with a reload, then would go with either a tftp boot, or
erase startup, tftp copy to startup and then reload.
As well as rancid, I also like to archive to a FTP server, so when TFTP
isn't an option (such as a remote site router, I can simply go
erase start
And then reload
For a client that bought CSM, I manage their ASA firewalls with CSM,
after a particularly bad experience with the built in CSM rollback, I now
do the changes and deploy normally rather than a rollback deploy if changes
need to be "reverted".
Cheers

_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

[shouldbe q931]

Post by heasley
use any char not in the banner. eg: =

However _any_ character _could_ be in the banner, ASCII art in logon
banners
Loading Image...

As the banner is not important for routing (or switching) packets, I'd
probably go with removing it from the rollback, or adding a "hardcoded"
banner that said that the router config had been rolled back, possibly
including pertinent information such as the date and time that the rollback
had been done etc.

[Paul Magee]

+1

I'd also take this approach

-----Original Message-----
From: rancid-discuss-***@shrubbery.net [mailto:rancid-discuss-***@shrubbery.net] On Behalf Of Miles Lott
Sent: 12 February 2013 20:28
To: rancid-***@shrubbery.net
Subject: Re: [rancid] Rollback functionality and potential pitfalls

Seems like it might be better to just overwrite running or startup
configs using copy from tftp. Or maybe I am missing your point.

Post by Ramon
Hello all,
I am currently looking in to a way to implement rollback functionality
using rancid. What would it take to reverse bad changes and restore
the config of a cisco device back to a previous saved version?
I ask because just pushing the original config on top of the modified
version would not remove or flush out certain statements, possibly
leaving duplicates that would have to be removed manually.
My first idea was to create a negative file, by generating a "no
statement" for every line in the new config that does not match the
old config. After pushing the negative file and removing the changes I
would push the old config to restore any of the deleted statements.
Feedback on any foreseeable issues such as possible hierarchical
problems (interfaces, acls, bgp) would be very welcomed.
Thanks,
Ramon

--------------------------------------------------------------------------------
Privileged, Proprietary and/or Confidential Information may be contained in
this electronic message. If you are not the intended recipient, you should
not copy it, re-transmit it, use it or disclose its contents, but should
kindly return to the sender immediately and delete your copy from your system.
Gulf Interstate Engineering Company does not accept responsibility for
changes made to this electronic message or its attachments.

_______________________________________________
Rancid-discuss mailing list
Rancid-***@shrubbery.net
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss



--------------------------------------- -- ********************************************* Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and WHG Trading Limited (registered number 101439). Each of William Hill PLC, William Hill Organization Limited is registered in England and Wales and has its registered office at Greenside House, 50 Station Road, Wood Green, London N22 7TP. William Hill U.S. HoldCo, Inc. is 160 Greentree Drive, Suite 101, Dover 19904, Kent, Delaware, United States of America. Each of WHG (International) Limited and WHG Trading Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes. *******************************************