[rancid] How to extend Rancid. Basics.

[jm+]

Hi there,

I wanted to extend Rancid with a script to monitor some Checkpoint Gaia
firewalls.

What needs to be done there is login, issue a command to stop
pagination, and then say "show configuration". That's it.

Yeah I know these devices allow a scheduled backup that I could somehow
inject into SVN. Certainly. But that is not the question.

Now I find a myriad of scripts for other devices but which seem too
complicated to me, since I'm not the absolute Perl guru.

Clogin works fine for the login, expect for some commands for setting up
the terminal that it gives, which don't make sense on Checkpoint.

The question thus is: what is the absolute minimum required feature set
of the actual script to interact with the main rancid application i.e.
feed it the content so it is stored in SVN?

I don't want to use any advanced features. Is there a doc on all of this?
I'd love to provide "signatures" for certain devices but this somehow
stands in my way.

I also see that there seem to be several approchaches. There are
different xxrancid scripts in the bin directory. On the other hand, some
devices seem to call rancid directly with the -t parameter, which seems
to use libs from the lib directory, which looks like a more modular
approach. What's the difference, and what is the official/recommended
approach?

Thanks.

Marki

[Alex DEKKER]

Post by jm+
Hi there,
I wanted to extend Rancid with a script to monitor some Checkpoint
Gaia firewalls.

Obviously not wanting to hijack the thread or anything, I have a related
question: If one wanted to add a new device type, what is the best
existing device type to use as a starting point?

alexd

[Alan McKinnon]

Post by Alex DEKKER

Post by jm+
Hi there,
I wanted to extend Rancid with a script to monitor some Checkpoint
Gaia firewalls.

Obviously not wanting to hijack the thread or anything, I have a related
question: If one wanted to add a new device type, what is the best
existing device type to use as a starting point?

It's not that simple, there is no "master reference device type" so to
speak.

Here's how rancid works:

For each device, a process is fired off that will return a text file.
This file is checked into CVS. If it has changed, CVS tracks the change.
If it has not changed, CVS does nothing (the usual CVS behaviour). You
will notice that the only thing rancid wants back from this script is a
text file, which could in fact be anything. You could return a Mickey
Mouse cartoon screenplay and rancid will dutifully book it into CVS for
you, complete with a diff. That this file is of no real use to you is
irrelevant, rancid will still track it,

The oldest and still most used of the various parsers is the one for
Cisco IOS and there is a persistent view that this script and it's
associated clogin is somehow a canonical script or a master model.

Nothing could be further from the truth - it is nothing more than a
script the original author wrote that reliably gets the job done. If you
need to track a different device type that mostly resembles IOS[1], then
start with that one. Note that you don't have to, it's just a reasonable
way to get started and get a result. It's also the most generic script
of the lot because it has to deal with everything Cisco has done for
years - the others all tend to be much more device-specific.

I wrote such a parser for AudioCodes once, the IOS script was a decent
starting point. To do the same for my NetGear DSL modem at home, using
the same starting point would be totally pointless, it is just so different.

To help you find a starting point, you would need to say what device it
is, clearly define exactly how one logs into it, what commands must be
run and what the output looks like. This last is most important, the
bulk of getting rancid to behave properly is getting the regexes right
that process the output text.


So the question you need to answer is how similar to IOS is your device?


[1] By "resemble", I mean log in with telnet or ssh, possibly enable,
issue some terminal setup commands, issue a bunch of get config commands
then trawl through it all with heaps of magic regexes. The exact list of
commands is completely irrelevant, they are stored in a perl array in
the script itself and you should change the list to suit what your
device needs. The list need not share anything in common with IOS! It
does need to do telnet/ssh and give EOL similarly.

[JM]

Post by Alan McKinnon

Post by jm+
Hi there,
I wanted to extend Rancid with a script to monitor some Checkpoint
Gaia firewalls.

It's not that simple, there is no "master reference device type" so to
speak.

Ok, as I thought, you just give rancid some text to store.

For non-perl gurus, the modern modular method is complicated. In fact, the
regexes you are talking about are specific to that I believe. As are
variables like $found_end and $clean_run that seem to be important to talk
to the main process. (Or is it just the return codes?)

Nothing will however prevent me from using the "old" method, give it a
"script" and a "login" (in the types file).

"login" can be, well, expect the login prompt, send the username, etc. you
get it. The "script" can be a stupid expect script running "show
configuration" and capturing the output. Maybe with some wrapper in a
language one is fluent in around it to carry out additional checks.

Don't get me wrong. It's great that the "modern method" to use rancid is
more modular (also less code duplication like with copying scripts etc.). It
could however be interesting to have some abstract configuration file where
you just tell it what to do, what to expect as a return, what to treat as
error. Much like "expect" in itself, but simpler, because in that case you'd
need to be a TCL guru. I'm thinking more along the lines of an INI file or
something =D

[Alan McKinnon]

Post by JM

Post by Alan McKinnon

Post by jm+
Hi there,
I wanted to extend Rancid with a script to monitor some Checkpoint
Gaia firewalls.

It's not that simple, there is no "master reference device type" so to
speak.

Ok, as I thought, you just give rancid some text to store.
For non-perl gurus, the modern modular method is complicated. In fact, the
regexes you are talking about are specific to that I believe. As are
variables like $found_end and $clean_run that seem to be important to talk
to the main process. (Or is it just the return codes?)

The regexes are the bulk of the main loop inside the *rancid scripts,
and there are hundreds of them. Each one follows these general sort of
examples:

/some regular expression/ && next;
/some regular expression/ && ProcessHistory(...);

These are the magic bits of code that determine exactly what text will
make it into the output file that is eventually stored in CVS. They are
a monumental pain in the butt to maintain but there's really no other
way to do it. It's not rancid's fault - blame the kit vendors for not
agreeing to provide a standard way to access the info (which just
happens to be the second most important thing about it once the device
works as intended). So each line has to be examined individually and
treated as a separate entity.

$found_end and $clean_run are internal true/false flag variables. They
indicate if the script got the full text it was expecting (a chunk of
text corresponding to each command issues plus a proper EOF at the end),
and if no errors were found. These flags tell the script if it must exit
with success or failure.

Post by JM
Nothing will however prevent me from using the "old" method, give it a
"script" and a "login" (in the types file).

Yes, you can do this. It's probably the better route if you only need to
deal with one device type not supported by rancid as shipped. The code
will be clean and you know right away it's a local addition to the codebase

Post by JM
"login" can be, well, expect the login prompt, send the username, etc. you
get it. The "script" can be a stupid expect script running "show
configuration" and capturing the output. Maybe with some wrapper in a
language one is fluent in around it to carry out additional checks.

Yes, that's the way to do it. Use whatever tool you know and feels right

Post by JM
Don't get me wrong. It's great that the "modern method" to use rancid is
more modular (also less code duplication like with copying scripts etc.). It
could however be interesting to have some abstract configuration file where
you just tell it what to do, what to expect as a return, what to treat as
error. Much like "expect" in itself, but simpler, because in that case you'd
need to be a TCL guru. I'm thinking more along the lines of an INI file or
something =D

I'm not sure I understand this paragraph. What is it that you want to
configure?

[JM]

Post by Alan McKinnon

Post by JM
Don't get me wrong. It's great that the "modern method" to use rancid is
more modular (also less code duplication like with copying scripts etc.). It
could however be interesting to have some abstract configuration file where
you just tell it what to do, what to expect as a return, what to treat as
error. Much like "expect" in itself, but simpler, because in that case you'd
need to be a TCL guru. I'm thinking more along the lines of an INI file or
something =D

I'm not sure I understand this paragraph. What is it that you want to
configure?

As far as that section goes, I was just wondering why there has to be code
for what could be considered configuration/definition in a format everyone
is able to use.

Maybe we should take a step back and get a look at the big picture. What do
we want rancid to do? Login, issue commands, get some replies, filter what's
deemed useful, and then logout again.

What could a simple definition in an abstract langugage (INI file, JSON,
...) look like?


--------------------------------------------------

[router1]

## metadata

connectmethod=ssh
username=x
password=y
passphrase=z
# login prompts
userprompt=User Name:
passprompt=Password:
# command prompt (regex)
commprompt=/>\s*$/
# enable mode required/exists
# this could even be generalized to any other special modes
# or even issued as a simple cmd[] below
enable=1
enablecommand=enable
enableprompt=/#\s*$/

## now tell it what to do

# command after successful login (commprompt seen)
cmd[1]=show configuration
# how to find out when previous command has sent all output?
(default=commprompt)
end[1]=commprompt
# filters on output
filter[1][1]=s/password (.*)/x/
filter[1][2]=s/bla/blubb/

cmd[2]=show version
#end[2]=commprompt (default)

cmd[3]=show dir

# instead of using special "enable" statement above, just do it manually here
cmd[4]=enable
end[4]=/#\s*/
# then do more stuff

cmd[5]=exit
end[5]=Connection closed.

--------------------------------------------------

[Alan McKinnon]

Post by JM

Post by Alan McKinnon

Post by JM
Don't get me wrong. It's great that the "modern method" to use rancid is
more modular (also less code duplication like with copying scripts etc.). It
could however be interesting to have some abstract configuration file where
you just tell it what to do, what to expect as a return, what to treat as
error. Much like "expect" in itself, but simpler, because in that case you'd
need to be a TCL guru. I'm thinking more along the lines of an INI file or
something =D

I'm not sure I understand this paragraph. What is it that you want to
configure?

As far as that section goes, I was just wondering why there has to be code
for what could be considered configuration/definition in a format everyone
is able to use.
Maybe we should take a step back and get a look at the big picture. What do
we want rancid to do? Login, issue commands, get some replies, filter what's
deemed useful, and then logout again.

"filter what is deemed useful" is where it all breaks down. There are
not a few of these, there are HUNDREDS of them for each device. Again,
blame the vendors for forcing us to resort to the worst possible
solution - manually parse free-form text output to derive information.

And we haven't even touched yet on the bits where stuff needs to be
redacted. "show run" dumps plenty secrets to the output - encryption
keys, community strings, passwords, hashes, certificates, private keys
and who knows what else, all of which has to be manually examined and a
regex written.

And it's not just "write stuff to output", it's "write stuff to output
in such a way that it is consistently ordered to not continually upset
CVS with noise that isn't a real change". See the ACL handling for the
best example of why this is vital.

It very quickly explodes out of control if you attempt to do it any
other way than how it is done.

Post by JM
What could a simple definition in an abstract langugage (INI file, JSON,
...) look like?

At heart, it would be a lookup pair. I'm only concentrating on the
output text mangling as everything else is easy, .ini key-value pairs
will suffice.

You'd have a regex on the left (no other descriptive language comes
close to a regex) and an action on the right. The trouble is, that
action can only be a code snippet, and this opens a world of hurt.

I'm glad you've opened this topic as I've tried to find ways to solve
it, and like you I figure there must be an easier way. But every time a
good solution trips up on real life requirements...

If you have a viable and well-thought out idea, I'm all ears. I've
looked for years and haven't found one. But it won't be the first time I
overlook something :-)

Post by JM
--------------------------------------------------
[router1]
## metadata
connectmethod=ssh
username=x
password=y
passphrase=z
# login prompts
# command prompt (regex)
commprompt=/>\s*$/
# enable mode required/exists
# this could even be generalized to any other special modes
# or even issued as a simple cmd[] below
enable=1
enablecommand=enable
enableprompt=/#\s*$/
## now tell it what to do
# command after successful login (commprompt seen)
cmd[1]=show configuration
# how to find out when previous command has sent all output?
(default=commprompt)
end[1]=commprompt
# filters on output
filter[1][1]=s/password (.*)/x/
filter[1][2]=s/bla/blubb/
cmd[2]=show version
#end[2]=commprompt (default)
cmd[3]=show dir
# instead of using special "enable" statement above, just do it manually here
cmd[4]=enable
end[4]=/#\s*/
# then do more stuff
cmd[5]=exit
end[5]=Connection closed.
--------------------------------------------------
_______________________________________________
Rancid-discuss mailing list
http://www.shrubbery.net/mailman/listinfo/rancid-discuss

[Robert Drake]

Not that the capability exists now, but check out NETCONF/YANG for what
could be and then ask your vendors why they can't make something
standard. Some of the big players are involved with it but the problem
is you're inevitably going to have some big boy gear that supports
netconf and then 10 shit-tier vendors that never will.

Most ISP's are at least going to have some mission critical gear that
needs to be backed up that is going to be built by someone in the
shit-tier category. I've got some gear from people who build routing
and switching products and still don't have a "show config/show run" of
any sort. Their latest product doesn't have a "show ip interface brief"
equivalent so you need to either SNMP poll it for interface status, use
their dumb management console or use the web frontend.

I don't think there will be a good solution within our careers. It
would be nice if we could convince vendors that configuration isn't a
special thing that requires years of careful study in manuals then
specialized testing per platform. The problems are that certifications
make the vendors money and keep us locked in to their products, which
are generally all just thin software shells around the same set of chips
the other guy is using.

Post by Alan McKinnon
You'd have a regex on the left (no other descriptive language comes
close to a regex) and an action on the right. The trouble is, that
action can only be a code snippet, and this opens a world of hurt.
I'm glad you've opened this topic as I've tried to find ways to solve
it, and like you I figure there must be an easier way. But every time a
good solution trips up on real life requirements...
If you have a viable and well-thought out idea, I'm all ears. I've
looked for years and haven't found one. But it won't be the first time I
overlook something:-)

[Michael Newton]

I did something up for our Aruba switches a couple of months ago. I'm not a Perl guy but my PHP and regular expressions background was enough to figure it out: https://github.com/miken32/rancid-aruba<https://github.com/miken32/rancid-aruba/>/<https://github.com/miken32/rancid-aruba/>. As you suspect, the modular method is the way that modern RANCID works. If clogin works for your gear then you're one step ahead; you just need to write the Perl module. (Note that even though the Cisco login works, it may not react properly to failure conditions since error messages aren't necessarily the same.) Use an existing Perl module to figure out what's going on, and rebuild to your liking.

--
Michael Newton



On Sat, Aug 1, 2015 at 2:49 AM -0700, "jm+***@roth.lu" <jm+***@roth.lu<mailto:jm+***@roth.lu>> wrote:

Hi there,

I wanted to extend Rancid with a script to monitor some Checkpoint Gaia
firewalls.

What needs to be done there is login, issue a command to stop
pagination, and then say "show configuration". That's it.

Yeah I know these devices allow a scheduled backup that I could somehow
inject into SVN. Certainly. But that is not the question.

Now I find a myriad of scripts for other devices but which seem too
complicated to me, since I'm not the absolute Perl guru.

Clogin works fine for the login, expect for some commands for setting up
the terminal that it gives, which don't make sense on Checkpoint.

The question thus is: what is the absolute minimum required feature set
of the actual script to interact with the main rancid application i.e.
feed it the content so it is stored in SVN?

I don't want to use any advanced features. Is there a doc on all of this?
I'd love to provide "signatures" for certain devices but this somehow
stands in my way.

I also see that there seem to be several approchaches. There are
different xxrancid scripts in the bin directory. On the other hand, some
devices seem to call rancid directly with the -t parameter, which seems
to use libs from the lib directory, which looks like a more modular
approach. What's the difference, and what is the official/recommended
approach?

Thanks.

Marki

[heasley]

Post by jm+
The question thus is: what is the absolute minimum required feature set
of the actual script to interact with the main rancid application i.e.
feed it the content so it is stored in SVN?

These don't seem to have been answered.

Your script must retrieve and filter the output it wishes to collect and
deposit it in a file named <router>.new. control_rancid does the svn
parts.

Post by jm+
I also see that there seem to be several approchaches. There are
different xxrancid scripts in the bin directory. On the other hand, some
devices seem to call rancid directly with the -t parameter, which seems
to use libs from the lib directory, which looks like a more modular
approach. What's the difference, and what is the official/recommended
approach?

rancid with the -t is a wrapper for the devices handled by libraries. But,
the script definition in rancid.types.* can be any program (with args) that
achieves the aforementioned output via the arguments that
control_rancid/rancid-fe pass. This allows the old method of rancid, see
examples in rancid.types.base, and the new way, plus user-defined methods.
It also means that not every script need be transitioned to the library
method before this could be released.

that was the goal - along with more modularity and the ability for users to
changes commands and/or filtering without altering the rancid code - which is
thus easier to upgrade and easier for me and the mail list participants to
support users. Admittedly, more can be done to improve flexibility and
modularity.