PROBLEM: x86 / x86_64: sys_brk should fail but does not

This is a bug report for the brk system call on x86[Linux kernel 
2.2,2.4; glibc 2.1.3,2.2.4,2.3.2]/x86_64[Linux kerel 2.4.21; glibc 
2.3.2] platforms.

sbrk is reporting success when it should be failing.  A certain sequence 
of calls to sbrk with some very large arguments causes this; see below.

I've run the C program below on several x86 Linux boxes with both 2.4 
and 2.2 kernels, some with glibc 2.2.  The last call to sbrk, which 
should fail, succeeds on all of them.

The output it yields on x86, or on x86_64 in 32-bit mode, is something 
like this:

sbrk(1536) is 134519100
errno=0
sbrk(32212) is 134520636
errno=0
sbrk(4160356352) is 4294967295
errno=12
sbrk(2147483648) is 4294967295
errno=12
sbrk(65536) is 134552848
errno=0
sbrk(31232) is 134618384
errno=0
sbrk(4160225280) is 4294967295
errno=12
sbrk(4160356352) is 134649616
errno=0

On x86_64, with a 64-bit compile (SuSE Linux 9.0 (x86-64) - Kernel 
2.4.21-193-default), the output is this, which seems odd because certain 
sbrk calls seem to be interpreted as requesting to give back memory to 
the system (i.e., the current break location is not monotonically 
increasing), when it seems they are all demanding more from the system:

sbrk(1536) is 5245984
errno=0
sbrk(32212) is 5247520
errno=0
sbrk(4160356352) is 5279732
errno=0
sbrk(2147483648) is 4165636084
errno=0
sbrk(65536) is 2018152436
errno=0
sbrk(31232) is 2018217972
errno=0
sbrk(4160225280) is 2018249204
errno=0
sbrk(4160356352) is 1883507188
errno=0


There's nothing magic about the number 4160356352 -- if you insert 
sbrk(4160356352) earlier in the sequence, it fails as it should.

I appreciate it,
David Chandler



#include <stdio.h>
#include <unistd.h>
#include <errno.h>

main(int argc) {
    errno = 0;
    printf("sbrk(1536) is %u\n", sbrk(1536));
    printf("errno=%d\n", errno); errno = 0;
    printf("sbrk(32212) is %u\n", sbrk(32212));
    printf("errno=%d\n", errno); errno = 0;
    printf("sbrk(4160356352) is %u\n", sbrk(4160356352UL));
    printf("errno=%d\n", errno); errno = 0;
    printf("sbrk(2147483648) is %u\n", sbrk(2147483648UL));
    printf("errno=%d\n", errno); errno = 0;
    printf("sbrk(65536) is %u\n", sbrk(65536));
    printf("errno=%d\n", errno); errno = 0;
    printf("sbrk(31232) is %u\n", sbrk(31232));
    printf("errno=%d\n", errno); errno = 0;
    printf("sbrk(4160225280) is %u\n", sbrk(4160225280UL));
    printf("errno=%d\n", errno); errno = 0;
    printf("sbrk(4160356352) is %u\n", sbrk(4160356352UL));
    printf("errno=%d\n", errno);
    if (0 == errno)
        printf("\n\nThat seems weird -- the last one succeeded?\n");
    else
        printf("\n\nWhat system are you using?  No bug there\n");
    return 0;
}