Re: Linux locks up when handling large data sets

On Fri, May 02, 2008 at 10:36:36PM +0200, squeak414@xxxxxxx wrote:
> Hi, I've been testing Squeak's ability to handle large amounts of data. This
> snippet:
> 
> testArrayFilling
>       | startTime endTime iArray jArray kArray |
>       iArray := Array ofSize: 100.
>       1
>               to: 100
>               do: [:i |
>                       jArray := Array ofSize: 1000.
>                       1
>                               to: 1000
>                               do: [:j |
>                                       startTime := Time millisecondClockValue.
>                                       kArray := Array ofSize: 1000.
>                                       1
>                                               to: 1000
>                                               do: [:k | kArray at: k put: 
> Object new].
>                                       jArray at: j put: kArray.
>                                       endTime := Time millisecondClockValue.
>                                       Transcript cr; show: i asString , ',' , 
> j asString , ' ' , (endTime -
> startTime) asFloat asString.
>                                       startTime := Time 
> millisecondClockValue].
>                       iArray at: i put: jArray].
>       Transcript cr; show: 'Finished'
> 
> creates about 5 million objects , then the image freezes. When I run on the 
> same
> machine under Windows, it happily continues until the short of memory warning 
> (
> about 70 Million objects in my case). The VM is 3.7 in both cases, the image
> Damien's 3.9 development image.

You are probably just growing your image to the point where the operating
system starts swapping. The image will appear to be unresponsive, but if
you interrupt it with <alt>period, it will eventually wake up and return
control to you.

The Unix VM automatically allocates memory from the operating system
as the Squeak object memory grows. This has the nice property of making
the image just as big as it needs to be without you worrying about it,
but it also means that if you write code that allocates far more memory
that is physically available, the operating system is going to start
thrashing to the point where Squeak becomes unusable. It will not really
be frozen though; if you are patient enough you can still save the image,
install more memory on your computer, and restart it ;-)

johnps11@xxxxxxxxxxx gave so good tips on how to control VM memory
allocation for the Unix VM.

I have also found that large images that I make on a Linux box
may refuse to load on Windows, with the Windows VM apparently
unable to allocate enough memory (even though plenty of memory
is available). I'm afraid that I never bothered to figure out
why (possibly error or ignorance on my part) but I mention it
so you won't be surprised if you try to show off your large
data sets on someone else's Windows laptop.

Dave