INN FAQ overchan can't keep up.

INN FAQ Part 7

NOTE: The maintainers of the INN FAQ stopped publishing in December 1997.
An important update to this topic is provided by Mib Software in the Usenet RKT for Subscribers.

Subject: (7.21) overchan can't keep up.
> About once a month or so, I get the following warning messages: > > Jan 20 07:20:22 optima innd: overview!:31:proc:9193 cant flush count 14639 Operation would block > Jan 20 07:20:22 optima innd: overview! spooling 14639 bytes or > there's a file "overview!" in /usr/spool/news/out.going with stuff in it. > > Should I be doing anything more with this than ignoring it, and maybe > occasionally deleting it (it just grows)? This happens because innd is feeding info to overchan faster than overchan can process it. The overflow is sent to the file "overview!". This file can be deleted, as nnrpd will grab the missing data out of the articles "manually". The slow-down won't be noticed, much. You can "expireover -a" to "fill in the holes". To prevent this in the future, you need to make overchan run faster. This is easy to do. There are two things to do: 1. Increase the size of many of your kernel buffers. In particular, increase buffers relating to directory caches (the "namei" cache", to mention one). If you use SunOS, change "maxusers" to 200. Ignore the variable's name. This variable is used to calculate most of the other buffer sizes, etc. and 200 is good for a system that is as overworked as, say, a machine running netnews. Do this only if you have enough RAM. I can't exactly say what is 'enough' but for a machine with 48MB 200 seems to be too big (64 is ok here). The problem seems to be that the kernel then needs too much space and runs out of it. 2. (this is more important than #1) Move the .overview files out of the /var/spool/news hierarchy. For example, moving the overview files into /var/spool/news/over.view made things fast enough on one machine that the problem went away. To do this: change "_PATH_OVERVIEWDIR" in "config.data", recompile, and "make install". You will need to recompile any newsreaders that read via NFS or off the local disk. For really great performance, put the NOV files on a separate disk. (Not just a separate partition, a separate disk or spindle.) This one-liner will generate a shell script that will move your NOV files: awk '{ print $1 }' /usr/lib/news/active \| tr . / \| awk '{ print "mkdir -p /new/location/" $1 ; print "mv " $1 "/.overview /new/location/" $1 "/.overview" }' WHY THIS WORKS: Why does doing all this speed up overchan? overchan works by opening the proper ".overview" file, appending 1 line to it, then closing the file. If you have the ".overview" file in the same directory as 10000 articles then opening the ".overview" file will take a huge amount of time. The open() call literally searches though about 5000 (half of 10000) file names to find ".overview". If you move your ".overview" files so that each one is in it's own directory, (say, /usr/spool/news/over.view/{group}/{name}/.overview) then open() is searching through 3 files ( ".", "..", and ".overview") to find 1 file. ( O(N/2) where N=10000 vs. N=3... and you thought those first year CS classes would never be useful!) There isn't much you can do to make the "append" and "close" steps much faster, except maybe install a PrestoServe or similar write-cache, and that won't help very much. Profiling overchan (with PureSoft's Quantify product) found that the open() call was around 80% of the execution time of overchan. That was reduced to 40% when I moved the ".overview" files to their own directory. With the change, overview's profiling statistics are pretty flat. (which is good). IF YOU CAN'T DO THE ABOVE CHANGES: Run "expireover -a" to fix the problem temporarily. However, it will come back. DO NOT try feeding the "overview!" file to overchan manually. (1) overchan doesn't do any locking and you'll have two overchan's running at once. (2) overchan only appends to the ".overview" files. If you've gotten any articles since the "overview!" file was created (you will have) then you'll be appending told old entries that are out of order. Your ".overview" files must be in sorted order for the other utilities to work right. See also INN FAQ #4.19, INN FAQ #5.33, INN FAQ #6.30 ------------------------------ [Source: INN FAQ Part 7 Archive-name: usenet/software/inn-faq/part7] [Last Changed: $Date: 1997/09/16 01:25:57 $ $Revision: 2.25 $] [Copyright: 1997 Heiko Rupp, portions by Tom Limoncelli, Rich Salz, et al.]

Subject: (7.21) overchan can't keep up.

> About once a month or so, I get the following warning messages:
> 
> Jan 20 07:20:22 optima innd: overview!:31:proc:9193 cant flush count 14639 Operation would block
> Jan 20 07:20:22 optima innd: overview! spooling 14639 bytes

or

> there's a file "overview!" in /usr/spool/news/out.going with stuff in it.
>
> Should I be doing anything more with this than ignoring it, and maybe
> occasionally deleting it (it just grows)?

This happens because innd is feeding info to overchan faster than
overchan can process it. The overflow is sent to the file
"overview!". This file can be deleted, as nnrpd will grab the missing
data out of the articles "manually". The slow-down won't be noticed,
much. You can "expireover -a" to "fill in the holes".

To prevent this in the future, you need to make overchan run faster.
This is easy to do. There are two things to do:

1. Increase the size of many of your kernel buffers. In particular,
increase buffers relating to directory caches (the "namei" cache", to
mention one). If you use SunOS, change "maxusers" to 200. Ignore the
variable's name. This variable is used to calculate most of the other
buffer sizes, etc. and 200 is good for a system that is as overworked
as, say, a machine running netnews. Do this only if you have enough RAM.
I can't exactly say what is 'enough' but for a machine with 48MB 200 seems
to be too big (64 is ok here). The problem seems to be that the kernel then
needs too much space and runs out of it.

2. (this is more important than #1) Move the .overview files out of
the /var/spool/news hierarchy. For example, moving the overview files
into /var/spool/news/over.view made things fast enough on one machine
that the problem went away. To do this: change "_PATH_OVERVIEWDIR" in
"config.data", recompile, and "make install". You will need to
recompile any newsreaders that read via NFS or off the local disk.

For really great performance, put the NOV files on a separate disk.
(Not just a separate partition, a separate disk or spindle.)

This one-liner will generate a shell script that will move your NOV files:

	awk '{ print $1 }' /usr/lib/news/active | tr . / | awk '{ print "mkdir -p /new/location/" $1 ; print "mv " $1 "/.overview /new/location/" $1 "/.overview" }'

WHY THIS WORKS:

Why does doing all this speed up overchan? overchan works by opening
the proper ".overview" file, appending 1 line to it, then closing the
file. If you have the ".overview" file in the same directory as 10000
articles then opening the ".overview" file will take a huge amount of
time. The open() call literally searches though about 5000 (half of
10000) file names to find ".overview". If you move your ".overview"
files so that each one is in it's own directory, (say,
/usr/spool/news/over.view/{group}/{name}/.overview) then open() is
searching through 3 files ( ".", "..", and ".overview") to find 1
file. ( O(N/2) where N=10000 vs. N=3... and you thought those first
year CS classes would never be useful!)

There isn't much you can do to make the "append" and "close" steps much
faster, except maybe install a PrestoServe or similar write-cache, and
that won't help very much.

Profiling overchan (with PureSoft's Quantify product) found that the
open() call was around 80% of the execution time of overchan. That was
reduced to 40% when I moved the ".overview" files to their own
directory. With the change, overview's profiling statistics are pretty
flat. (which is good).

IF YOU CAN'T DO THE ABOVE CHANGES:
Run "expireover -a" to fix the problem temporarily. However, it will
come back.

DO NOT try feeding the "overview!" file to overchan manually.
(1) overchan doesn't do any locking and you'll have two overchan's
running at once.
(2) overchan only appends to the ".overview" files. If you've gotten
any articles since the "overview!" file was created (you will
have) then you'll be appending told old entries that are out of
order. Your ".overview" files must be in sorted order for the
other utilities to work right.

See also INN FAQ #4.19, INN FAQ #5.33, INN FAQ #6.30

------------------------------

[Source: INN FAQ Part 7 Archive-name: usenet/software/inn-faq/part7]
[Last Changed: $Date: 1997/09/16 01:25:57 $ $Revision: 2.25 $]
[Copyright: 1997 Heiko Rupp, portions by Tom Limoncelli, Rich Salz, et al.]