Re: ghc releasing memory during compilation

Duncan Coutts wrote:
> As some are aware some Wash modules cause ghc to take a very large
> amount of memory. It also generates very large C and assembler files
> (the .raw_s file for one module is nearly 50Mb). So unsurprisingly it
> also makes gcc take a very large amount of memory.
>
> Unfortunately for people with weaker machines these happen at the same
> time. That is at the same time that gcc starts taking 300Mb, ghc is
> still taking up 400Mb. Even on machines with 1Gb of ram this pushes
> everything else out into swap.
>
> Note that unless constrained, ghc will take even more than 400Mb to
> build wash (I noticed it using over 750Mb). The Gentoo ebuild already
> limits ghc to 400Mb on 64 bit machines (and 200Mb on 32bit ones).
>
> What I was wondering is if ghc could do a major GC and free most of it's
> memory back to the system just before it calls gcc to compile the .hc
> code that ghc has generated. That way the memory spike of ghc and gcc
> would not coincide and our machines would not be brought to a crawl.
>
> Am I right in thinking that ghc's rts can free memory back to the system
> when the heap pressure reduces (at least if it's doing a compacting GC)?

No, not at the moment.  One thing we planned to do but never got around 
to is to use madvise() to improve swapping behaviour when memory is 
tight (see the thread that Bulat refferred to).

> In this case if it can do so, it should be quite dramatic. It'd ought to
> be able to go from 400Mb back down to just a few Mb or so.

Yes it ought to.

A related problem is that the block allocator is really stupid.  It 
makes no attempt to reduce fragmentation, and in fact freeing blocks is 
O(n) because the freelist is kept sorted in address order.  This isn't 
usually an issue, although it has been reported to be noticeable with 
very large residencies (500M+).  It's on my list to fix at some point.

I mention this because freeing all that memory might not be possible if 
it is highly fragmented.

Cheers,
        Simon