[tin-users] Openning huge group

[Urs Janßen]

In <mailman.1.1589102175.1535.tin-users at tin.org>, Dennis Preiser wrote:
> g.l.k has appr. 3,4 million article. The first time entring the group
> took here about 25 to 30 minutes. tin uses about 2 GB RAM. The cached
> overview file is about 1.2 GB in size.
> 
> | dennis at coredump:~% ps -o rss,command | grep '[g]mane'
> | 1926692 tin -g gmane
> | dennis at coredump:~% ls -hoS ~/tmp/.news-news.gmane.io
> | total 3466768
> | -rw-r--r--  1 dennis   1.2G 10 May 10:05 1097.1
> | [...]
> 
> The second time it took about 20 minutes. The slower the internet
> connection, the longer the first start takes.

Dennis looked into this a bit deeper, a lot of time was wasted in
reallocs due to too small hash-tables. With the current snapshot[1]
which also contains the surpression of the mime error if not running
in debug mode, initial caching the overviews of gmane.linux.kernel
from news.gmane.io "just" takes 9 min. with my slow dls-line.
Mesured with:

echo "gmane.linux.kernel:" > glk-gmane
export TIN_INDEX_NEWSDIR=/dev/shm
sed -i 's,^cache_overview_files=OFF,cache_overview_files=ON,' ~/.tin/tinrc
time tin -f glk-gmane -uvQg news.gmane.io

Afterwards entering gmane.linux.kernel is down to 1 min. (with
additional feedback).

Be aware that local caches do cost disk space and tin does not take
care to expire data for unvisted groups (and of course /dev/shm is not
the best place for them ,-);

Purging and refreshing the cache via a cronjob
(tools/tools/expiretover and tin -u) might be a good idea.

And as Dennis also mentioned -G is worth (negative number in the
thousends is usually what is most useful) to be considered.

HTH,
urs

[1]
<ftp://ftp.tin.org/pub/news/clients/tin/v2.4/snapshots/tin-2.4.5.tar.gz>
<ftp://ftp.tin.org/pub/news/clients/tin/v2.4/snapshots/tin-2.4.5.tar.bz2>
<ftp://ftp.tin.org/pub/news/clients/tin/v2.4/snapshots/tin-2.4.5.tar.xz>>