Discussion:
[rancid] Rancid output "Binary files differ"
Árni Birgisson
2009-11-25 11:39:20 UTC
Permalink
Hi guys,

I'm wondering if anybody has seen this problem before.
This mail is going to be a bit long, starting with a simple description of the problem,
and then my diagnosis of it.

From time to time I'm getting a strange output from RANCID:

-----

Index: configs/router-x
===================================================================
retrieving revision 1.88
diff -U 4 -r1.88 router-x
Binary files /tmp/cvsQSyJ8S and router-x differ

-----

As I noticed that the revision number was quite high I looked around and it seems that this
does happen for more than one router - but happens very often to this one.
Most of the network equipment has had about 30-40 revisions since RANCID was installed
but this one has 88 revisions.

If I check a few versions of the config out of CVS I can see that indeed some of the files are weird.

-bash-4.0$ cvs diff -r1.86 -r1.89
--normal diff output showing config changes--
-bash-4.0$ cvs diff -r1.86 -r1.87
--normal diff output showing config changes--
-bash-4.0$ cvs diff -r1.86 -r1.88
cvs diff: Diffing pe
cvs diff: Diffing pe/configs
Index: pe/configs/router-x
===================================================================
RCS file: /var/rancid/CVS/pe/configs/router-x,v
retrieving revision 1.86
retrieving revision 1.88
diff -r1.86 -r1.88
Binary files /tmp/cvsCqpVfB and /tmp/cvsp09kv5 differ

Okay, looks like version 1.88 is the weird one.

Let's fetch 1.88 and 1.89
-bash-4.0$ cvs co -r1.88 pe/configs/router-x
U pe/configs/router-x
-bash-4.0$ cp pe/configs/router-x v1.88
-bash-4.0$ cvs co -r1.89 pe/configs/router-x
U pe/configs/router-x
-bash-4.0$ cp pe/configs/router-x v1.89

-bash-4.0$ ls -al v1.8*
-rw-r--r-x 1 rancid rancid 70556 2009-11-25 05:28 v1.88
-rw-r--r-x 1 rancid rancid 70556 2009-11-25 05:28 v1.89
-bash-4.0$ diff v1.88 v1.89
Binary files v1.88 and v1.89 differ
-bash-4.0$

--- same size of the files, but diff won't compare them because they are (or one of them is) a binary file.

-bash-4.0$ strings v1.88 | wc -l
1387
-bash-4.0$ strings v1.89 | wc -l
1542

Now when I have a look at the files it seems that the smaller file (v1.88 and "binary") starts with a lot of \0 bytes,
it would seem one for each character that is missing - and then continues exactly like the other one.
Or should I say the files are exactly the same - but it's as if the first (1542 - 1387) bytes of v1.88 are 0-byte instead
of the character that should be there.

Diff --text on them shows that v1.89 includes all the lines from v1.88 and more and there has been no real
change to the config.

I'm running rancid version 2.3.2, with a slight change to jrancid:
The change to jrancid is to make it backup a "show configuration | display set" version of the config as well as the
normal "show configuration" and is done by adding one function:

sub ShowConfigurationDisplaySet {
print STDERR " In ShowConfigurationDisplaySet: $_" if ($debug);

s/^[a-z]+@//;
ProcessHistory("","","","# $_");
while (<INPUT>) {
tr/\015//d;
last if (/^$prompt/);
next if (/^system (shutdown message from|going down )/i);
next if (/^\{(master|backup)}/);

/Unrecognized command/ && return;
/command is not valid/ && return;
/^\s+\^/ && return;
/syntax error/ && return;

ProcessHistory("","","","$_");
}
return(0);
}


and one item to @commandtable:
{'show configuration | display set' => 'ShowConfigurationDisplaySet'},

It could very well be that the errors are introduced because of my addition,
but I fail to see how.

For those that have read through all of this - thank you,
hopefully somebody can spot what's wrong!

- Arni















Arni Birgisson
Network Engineer
FATTOC Iceland
john heasley
2009-11-25 19:41:10 UTC
Permalink
Post by Árni Birgisson
Hi guys,
I'm wondering if anybody has seen this problem before.
This mail is going to be a bit long, starting with a simple description of the problem,
and then my diagnosis of it.
Árni Birgisson
2009-11-25 22:46:42 UTC
Permalink
Hi John, thanks for your reply.
not likely. do you have a fsys or memory problem on this host? broken
expect/tcl? some other unintentional change to rancid?
I don't believe there is any problem with the server itself, no errors in dmesg concerning
hard disks or filesystems (mirrored disks), and 6GB free memory of total 8GB.
Very little load on the server, and other processes running on it are working fine.

expect and tcl are newest versions from YUM,
-- snip --
Package 1:tcl-8.5.6-6.fc11.x86_64 already installed and latest version
Package expect-5.43.0-17.fc11.x86_64 already installed and latest version
-- snip --

And this behaviour is not constant, I have seen it from time to time with other routers, but
most of the time it's this one. Still not always on this one either - usually the backups come
through correct.

- Arni
Post by Árni Birgisson
Hi guys,
I'm wondering if anybody has seen this problem before.
This mail is going to be a bit long, starting with a simple description of the problem,
and then my diagnosis of it.
-----
Index: configs/router-x
===================================================================
retrieving revision 1.88
diff -U 4 -r1.88 router-x
Binary files /tmp/cvsQSyJ8S and router-x differ
-----
As I noticed that the revision number was quite high I looked around and it seems that this
does happen for more than one router - but happens very often to this one.
Most of the network equipment has had about 30-40 revisions since RANCID was installed
but this one has 88 revisions.
If I check a few versions of the config out of CVS I can see that indeed some of the files are weird.
-bash-4.0$ cvs diff -r1.86 -r1.89
--normal diff output showing config changes--
-bash-4.0$ cvs diff -r1.86 -r1.87
--normal diff output showing config changes--
-bash-4.0$ cvs diff -r1.86 -r1.88
cvs diff: Diffing pe
cvs diff: Diffing pe/configs
Index: pe/configs/router-x
===================================================================
RCS file: /var/rancid/CVS/pe/configs/router-x,v
retrieving revision 1.86
retrieving revision 1.88
diff -r1.86 -r1.88
Binary files /tmp/cvsCqpVfB and /tmp/cvsp09kv5 differ
Okay, looks like version 1.88 is the weird one.
Let's fetch 1.88 and 1.89
-bash-4.0$ cvs co -r1.88 pe/configs/router-x
U pe/configs/router-x
-bash-4.0$ cp pe/configs/router-x v1.88
-bash-4.0$ cvs co -r1.89 pe/configs/router-x
U pe/configs/router-x
-bash-4.0$ cp pe/configs/router-x v1.89
-bash-4.0$ ls -al v1.8*
-rw-r--r-x 1 rancid rancid 70556 2009-11-25 05:28 v1.88
-rw-r--r-x 1 rancid rancid 70556 2009-11-25 05:28 v1.89
-bash-4.0$ diff v1.88 v1.89
Binary files v1.88 and v1.89 differ
-bash-4.0$
--- same size of the files, but diff won't compare them because they are (or one of them is) a binary file.
-bash-4.0$ strings v1.88 | wc -l
1387
-bash-4.0$ strings v1.89 | wc -l
1542
Now when I have a look at the files it seems that the smaller file (v1.88 and "binary") starts with a lot of \0 bytes,
it would seem one for each character that is missing - and then continues exactly like the other one.
Or should I say the files are exactly the same - but it's as if the first (1542 - 1387) bytes of v1.88 are 0-byte instead
of the character that should be there.
Diff --text on them shows that v1.89 includes all the lines from v1.88 and more and there has been no real
change to the config.
The change to jrancid is to make it backup a "show configuration | display set" version of the config as well as the
sub ShowConfigurationDisplaySet {
print STDERR " In ShowConfigurationDisplaySet: $_" if ($debug);
ProcessHistory("","","","# $_");
while (<INPUT>) {
tr/\015//d;
last if (/^$prompt/);
next if (/^system (shutdown message from|going down )/i);
next if (/^\{(master|backup)}/);
/Unrecognized command/ && return;
/command is not valid/ && return;
/^\s+\^/ && return;
/syntax error/ && return;
ProcessHistory("","","","$_");
}
return(0);
}
{'show configuration | display set' => 'ShowConfigurationDisplaySet'},
It could very well be that the errors are introduced because of my addition,
but I fail to see how.
not likely. do you have a fsys or memory problem on this host? broken
expect/tcl? some other unintentional change to rancid?
Arni Birgisson
Network Engineer
FATTOC Iceland
Adalstraeti 6
101 Reykjavik
Iceland
john heasley
2009-11-27 21:53:57 UTC
Permalink
Post by Árni Birgisson
Hi John, thanks for your reply.
not likely. do you have a fsys or memory problem on this host? broken
expect/tcl? some other unintentional change to rancid?
I don't believe there is any problem with the server itself, no errors in dmesg concerning
hard disks or filesystems (mirrored disks), and 6GB free memory of total 8GB.
Very little load on the server, and other processes running on it are working fine.
expect and tcl are newest versions from YUM,
-- snip --
Package 1:tcl-8.5.6-6.fc11.x86_64 already installed and latest version
Package expect-5.43.0-17.fc11.x86_64 already installed and latest version
-- snip --
And this behaviour is not constant, I have seen it from time to time with other routers, but
most of the time it's this one. Still not always on this one either - usually the backups come
through correct.
that'd lead me to believe its timing-based and most likely related to
expect reading portions of the input buffer, but thats a WAG. could
be a number of things.

the easiest thing to try is get the lastest tcl and expect, apply the
expect patch from the rancid website and compile them yourself.
Post by Árni Birgisson
- Arni
Post by Árni Birgisson
Hi guys,
I'm wondering if anybody has seen this problem before.
This mail is going to be a bit long, starting with a simple description of the problem,
and then my diagnosis of it.
Loading...