Re: Weird memory leak

G'day,

This might be a bit left-field, but are you working on some kind of tempfs 
or swapfs? I believe Mac OS X is built on a BSD kernel, which I suspect 
supports the notion of a swapfs filesystem for /tmp. I work under Solaris, 
and the basic principle is that instead of allocating swap space to work 
with, you allocate a temp filesystem which can be used for temporary files 
but can also be used for memory being swapped out by the operating system. 
If you're operating in such an area your data will be lost across a 
reboot, but that's fine for both swap and temporary files. As you fill up 
the temporary area your data is not necessarily written to disk, instead 
it's treated as swappable memory pages that may never be written to a 
physical media, particularly if your files are very short-lived.

Under Solaris if you write large files into /tmp you will see your 
physical memory available diminish until you erase the files. Why? Because 
it's really using memory. It's leaving them swapped in until it starts 
getting low on memory.

Another interpretation of the results is normal virtual filesystem 
caching. If you read large files from disk then a modern operating system 
will keep what it can in memory, just in case you want to go and read the 
file again. If you write to the file the change will be immediately 
reflected in memory and will be written back to disk at the first possible 
opportunity. In the mean-time your memory is consumed, but again if you 
started to run seriously short of memory the pages would simply be dropped 
from memory and the next attempt to access them would force them to be 
read from disk again. If you erase the file presumably the blocks would be 
flushed immediately from memory and thus memory would be freed.

Both of these possibilities come from the operating system. Alternatively 
you might just be unaware of a normal process which is chewing up memory 
:) Maybe you've accidently launched a fork-bomb which is causing many 
small processes to each consume a small amount of system resources :)

<shrug>

Benjamin.





"Steven Frank" <stevenf-d1CFi0FgTokAvxtiuMwx3w@xxxxxxxxxxxxxxxx>
21/11/02 05:44
Please respond to sqlite

 
        To:     sqlite-hHKSG33TihhbjbujkaE4pw@xxxxxxxxxxxxxxxx
        cc: 
        Subject:        [sqlite] Weird memory leak


Hi there, I'm new to SQLite, and I'm using it on a Mac OS X project. 
It's really wonderful, and fast (once I figured out indexing).

The current problem I'm having is a memory leak -- however, it's a
little unusual. 

My program creates a single database with a few tables and gradually
adds about 14,000 rows to one of the tables.  The rows are added in
transactions of 100 rows at a time.

As this is going on, available system memory gradually drops to zero,
until the system starts paging.  However, the memory loss is not
attributable to any running process.  Looking at top, no particular
process's memory usage is increasing, but available system memory
still eventually drops to zero.

If I kill my SQLite program, most of the memory is reclaimed, but some
is not returned to the system until I delete SQLite's database file. 
This leads me to believe the OS is mapping the file somehow.  (The
database file is about 200 MB in size if the process runs to
completion.)

I've confirmed there aren't any other significant memory leaks in the
program -- if I simply comment out the line that performs the database
inserts, the behavior goes away.  (And, as I mentioned, the memory
usage of my app does not increase as it's running.)



To unsubscribe from this group, send an email to:
sqlite-unsubscribe-VgYJa0VH1e9BDgjK7y7TUQ@xxxxxxxxxxxxxxxx

 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/