memory/zfs leaks? (system frozen)

[Jens Elkner]

Hi,

since some people locked up their desktop by just allocating too much mem,
they asked me, how to find out, how much memory they may allocate
without causing their system to swap/freeze. That's easy to answer I
though: freemem + arc_size - arc_size_min

But wrong: I used this TPOTT (The POor [man's] Tiny Tools] for this:
http://iws.cs.uni-magdeburg.de/~elkner/osol/ see memfault.txt etc.
(results are in its "res" sub directory).

What seems odd to me:
1) initialy (i.e.after boot) the mdb kernel pages take ~ 500-600 MB
2) a find in my scratch area:
a) let it grow to 2.5-3 GB - why?
b) causes finally an arc size of ~ 1.5 GB
3) a simple dd from /dev/zero to a zfs of ~ 4GB finally causes an
arc_size of ~ 4 GB; kernel size keeps about the same
4) now, when running the memtest, which just allocates memor in chunks of
128 MB 'til it got a certain amount of alloc errors
a) the kernel page size shrinks as expected, but not enough IMHO:
it never shrinks back to its original size (~ 600 MB) but keeps
hogging ~ 2GB even under high mem pressure.

Why isn't the kernel able to shrink to its initial size, when it
is under memory pressure? Is there a memory leak?

b) the arc size shrinks more or less slowly as well, but almost never
to its minimum size (arcstats:c_min). Actually when running the
test on a freshly booted machine, it goes down to ~ 1.2*c_min
on other "long running" machines (usually U40 with snv_b129, 4 GB)
it never goes below ~ 2*c_min.

So why doesn't ZFS obey its limits? Is this caused by another
memory leakage or perhaps wrong internal accounting?

c) Usually, when running the malloc memtest, the machine gets sooner
or later frozen (sometimes at the 1st run, sometimes at the 2nd or
3rd run, only if you are very lucky, the system will survive the test).
"Frozen" means: after a short swapping time (HDD acustics) total
silence: no screen updates, no response if logged in remotely
via ssh, etc.. The only thing one can do, to bring it back to life
is to push the power button!

Why doesn't the OS simply return a ENOMEM etc. if it can't handle
the request and stops the application from eating further memory?
Pretty odd! (BTW: not much difference between b129 and S11X).

Finally: is one able to explain, how the mdb stats align/correlate to
the kstats? E.g. on a 6 GB RAM system mdb says, the kernel uses 2.3 GB
(unswappable?), zfs data ~ 2.8 GB, ~ 0.7GB free + ~ 0.2 GB misc stuff -
so far ok (e.g. see res/mdbstats1dd.txt). On the other side kstat says
~ 0.7 GB free (ok), but arc size ~ 4 GB. Neither this nor the arc's
header/data/other sizes seem to correlate to mdbs kernel/zfs data stats
(e.g. see res/kstats1dd.txt). Well, when running the memtest, mdb shows
that zfs data size goes down to a negliable size (see
res/mdbstats2freeze.txt), but actually I can't deduce anything from
this as well - doesn't seem to be kstat related (see res/kstats2freeze.txt).

Last but not least back to the initial question: How can a user
determine the amount of memory it can use in its application without
causing the system to swap/freeze.
Right now my safe bet would be: PhysicalRAM - 2GB - 2*arc_size_min, i.e.
about half of the physical RAM (for 4..12GB WS).

But if I would tell that our people, they would immediately start
begging for linux ... :(

So any hints?

Regards,
jel.