Re: mysqld crash due to innoDB problems?

Rusty,

I found and fixed now the following bug:

" mem0pool.c:
    Fix a memory corruption bug: in 32-bit computers, every 4 billionth
malloc outside
innodb_additional_mem_pool_size was mistreated when freeing the memory; this
could
corrupt the InnoDB additional mem pool and could have caused crashes
anywhere, also
inside MySQL, or even database corruption! the bug exists also in 3.23 and
4.1;
workaround: configure innodb_additional_mem_pool_size big enough
"

How big had you set innodb_additional_mem_pool_size?

You can use

SHOW INNODB STATUS\G

to look at the memory usage in the additional mem pool. Note that the real
usage is maybe 2 - 4 times bigger than what it reports. If you are using
BLOBs, that will temporarily increase memory usage, and can explain why the
crashes were associated with the blob_heap of a table handle.

----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 84966343; in additional pool allocated 1402624

Regards,

Heikki

----- Original Message ----- 
From: "Heikki Tuuri" <Heikki.Tuuri@xxxxxxxxxx>
To: <bugs@xxxxxxxxxxxxxxx>; <rusty.shanklin@xxxxxxxxxxxxxxxxxxxxx>
Sent: Monday, May 17, 2004 11:19 PM
Subject: Re: mysqld crash due to innoDB problems?


> Rusty,
>
> please send the WHOLE mysqld .err log to me. Please also send also the
> output of
>
> SHOW CREATE TABLE Inbound;
>
> I am starting to suspect there might be an InnoDB memory corruption bug
> associated with the blob_heap of the table handle. You are the second to
> report a crash in the same place.
>
> Though it is hard to see what might corrupt exactly 4, or maybe 16, bytes
at
> the start of an InnoDB additional mem pool block. It does not look like an
> ordinary memory buffer overrun. Anyway, it is easy to add diagnostic code
to
> 4.0.2x, so that we can study this in detail.
>
> Best regards,
>
> Heikki Tuuri
> Innobase Oy
> Foreign keys, transactions, and row level locking for MySQL
> InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up
MyISAM
> tables
> http://www.innodb.com/order.php
>
> Order MySQL technical support from https://order.mysql.com/
>
> 0x80705d0 handle_segfault + 420
> 0x8297878 pthread_sighandler + 184
> 0x822837c mem_area_free + 548
> 0x82271d1 mem_heap_block_free + 385
> 0x81322fd row_sel_store_mysql_rec + 65
> 0x8137047 row_search_for_mysql + 11491
> 0x80cbe07 general_fetch__11ha_innobasePcUiUi + 359
> 0x80cbeb3 index_next__11ha_innobasePc + 35
> 0x809dd75 join_read_next__FP14st_read_record + 33
> 0x809d109 sub_select__FP4JOINP13st_join_tableb + 337
> 0x809cdc0 do_select__FP4JOINPt4List1Z4ItemP8st_tableP9Procedure + 412
> 0x8095402
> mysql_select__FP3THDP13st_table_listRt4List1Z4ItemP4ItemP8st_orderT4T3
> T4UlP13select_result + 6994
> 0x8093876 handle_select__FP3THDP6st_lexP13select_result + 102
> 0x807afd0 mysql_execute_command__Fv + 1004
> 0x807e9ba mysql_parse__FP3THDPcUi + 146
> 0x807a0c3 dispatch_command__F19enum_server_commandP3THDPcUi + 1435
> 0x8079b1d do_command__FP3THD + 157
> 0x8079359 handle_one_connection + 641
> 0x829502c pthread_start_thread + 220
> 0x82be82a thread_start + 4
>
> void
> row_sel_store_mysql_rec(
> /*====================*/
>         byte*           mysql_rec,      /* out: row in the MySQL format */
>         row_prebuilt_t* prebuilt,       /* in: prebuilt struct */
>         rec_t*          rec)            /* in: Innobase record in the
index
>                                         which was described in prebuilt's
>                                         template */
> {
>         mysql_row_templ_t*      templ;
>         mem_heap_t*             extern_field_heap       = NULL;
>         byte*                   data;
>         ulint                   len;
>         byte*                   blob_buf;
>         ulint                   i;
>
>         ut_ad(prebuilt->mysql_template);
>
>         if (prebuilt->blob_heap != NULL) {
>                 mem_heap_free(prebuilt->blob_heap); <--- crash here
>                 prebuilt->blob_heap = NULL;
>         }
>
> .................
>
> List:MySQL Bugs« Previous Message
> From:Rusty W. ShanklinDate:May 17 2004 5:11pm
> Subject:mysqld crash due to innoDB problems?
>
>
>
> DESCRIPTION:
>   Mysqld appears to crash every few days or so.  The .err file shows
> that it may be an innodb problem.  (See below for detail)
>
> How-To-Repeat
>   I am not able to repeat the problem.  The query that runs when the
> mysqld server dies, is automatically re-run when the server comes back
> up.  The query always appears to run successfully after the restart.
>
> Fix:
>    The automatic restart of mysqld solves the problem, however the
> server going down is now my problem.
>
> Submitter-ID: RustyShanklin
> Organization: Pikeville Medical Center
> MySQL support: NONE
> Synopsis: mysqld crash - Possible InnoDB corruption?
> Severity: serious
> Severity: medium
> Category: mysql
> Class: sw-bug
> Release:       mysql-4.0.18-standard (Official MySQL-standard binary)
> C compiler:    2.95.3
> C++ compiler:  2.95.3
> Environment:
> System: Linux mish02.pikevillehospital.org 2.4.18-14smp
> #1 SMP Wed Sep 4 12:34:47 EDT 2002 i686 i686 i386 GNU/Linux
>  (4 processor system) 2GB Ram.
> Architecture: i686
> Some paths:  /usr/bin/perl /usr/bin/make /usr/bin/gmake /usr/bin/gcc
> /usr/bin/cc
> GCC: Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2/specs
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> --infodir=/u
> sr/share/info --enable-shared --enable-threads=posix --disable-checking
> --host=i
> 386-redhat-linux --with-system-zlib --enable-__cxa_atexit
> Thread model: posix
> gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7)
> Compilation info: CC='gcc'  CFLAGS='-O2 -mcpu=pentiumpro'  CXX='gcc'
> CXXFLAGS='
> -O2 -mcpu=pentiumpro -felide-constructors'  LDFLAGS=''  ASFLAGS=''
> LIBC:
> lrwxrwxrwx    1 root     root           14 Mar  4  2003 /lib/libc.so.6
> -> libc-2
> .2.93.so
> -rwxr-xr-x    1 root     root      1235468 Sep  5  2002
> /lib/libc-2.2.93.so
> -rw-r--r--    1 root     root      2233342 Mar 10  2003 /usr/lib/libc.a
> -rw-r--r--    1 root     root          178 Mar 10  2003 /usr/lib/libc.so
> Configure command: ./configure '--prefix=/usr/local/mysql'
> '--localstatedir=/usr
> /local/mysql/data' '--libexecdir=/usr/local/mysql/bin'
> '--with-comment=Official
> MySQL-standard binary' '--with-extra-charsets=complex'
> '--with-server-suffix=-st
> andard' '--enable-thread-safe-client' '--enable-local-infile'
> '--enable-assemble
> r' '--disable-shared' '--with-client-ldflags=-all-static'
> '--with-mysqld-ldflags
> =-all-static' '--with-embedded-server' '--with-innodb' 'CFLAGS=-O2
> -mcpu=pentium
> pro' 'CXXFLAGS=-O2 -mcpu=pentiumpro -felide-constructors' 'CXX=gcc'
>
> ----------mysql.err file:
> InnoDB: Error: Mem area size is 0. Possibly a memory overrun of the
> InnoDB: previous allocated area!
> InnoDB: Apparent memory corruption: mem dump  len 500; hex
> 4e43303b5350433a4c414
> 23b3133343a3a3a3a3a3a3a4e3a3a22223a3a3a31333439383b5354414d5045523b44414
> e49454c3
> b483b3b3b4d443a3a22223a22223a3a3a3230303430353135323230343a3b20203a3a4f7
> 26465726
> 5643a3a313b22223b22223b3230303430353136303230323b22223b333b3b3b3a3a3a222
> 23a3a3a3
> a3a3a0d4447313a39393a49393a3a3a3a574f524b494e473a3a3a3a3a3a3a3a0d5a43413
> a22223a2
> 2223a3b3b3b0d3506001e05353738303708535441525f57454213323030342d30352d313
> 52032323
> a30343a3133016e03413038fc09064d53483a3b7e5c263a000000000000000000000000a
> 90100000
> 0000000b010a808a0010000f63d775f9268a520773073656c2e6300a2080000000000000
> 00000000
> 00000000000000000000000900100000000000000000000900100004000000000000000f
> 00e41084
> d53483a3b7e5c263a48424f433a413a534f46544c41423a413a323030343035313530373
> 33830363
> a3a4f524d3b4f30313b30313a373239303430343a503a322e313a373239303430343a3a0
> d5049443
> a3a3a3031303030353b3b3b413a3a5455524e45523b4b4154484c45454e3a3a313933303
> 03631333
> a463a3a3a3a3a3a3a3a3a3a303431333130333336373b3b3b413a3430302d33382d38393
> 1380d4f5
> 2433a4e413a31353630; asc
> NC0;SPC:LAB;134:::::::N::"":::13498;STAMPER;DANIEL;H;;;
> MD::"":"":::200405152204:;
> ::Ordered::1;"";"";200405160202;"";3;;;:::""::::::.D
> G1:99:I9::::WORKING::::::::.ZCA:"":"":;;;.5....57807.STAR_WEB.2004-05-15
> 22:04:1
> 3.n.A08...MSH:;~\&:.............................=w_.h.
> w0sel.c..................
> .......................@.........A.MSH:;~\&:HBOC:A:SOFTLAB:A:20040515073
> 806::ORM
> ;O01;01:7290404:P:2.1:7290404::.PID:::010005;;;A::TURNER;KATHLEEN::19300
> 613:F:::
> :::::::0413103367;;;A:400-38-8918.ORC:NA:1560;
> InnoDB: Scanning backward trying to find previous allocated mem blocks
> Freed mem block at - 52160, file w0sel.c, line 2210
> Freed mem block at - 53016, file w0sel.c, line 2210
> Freed mem block at - 54584, file w0sel.c, line 2210
> Freed mem block at - 117720, file w0sel.c, line 2210
> Mem block at - 117912, file 0pcur.c, line 28
> Mem block at - 118128, file w0upd.c, line 289
> Mem block at - 118232, file m0rec.c, line 443
> Freed mem block at - 124928, file w0sel.c, line 2210
> Mem block at - 125216, file w0sel.c, line 2273
> Mem block at - 126704, file w0sel.c, line 2210
> InnoDB: Scanning forward trying to find next allocated mem blocks
> Freed mem block at + 16, file w0sel.c, line 2210
> Mem block at + 2448, file nodb.cc, line 1920
> Freed mem block at + 2736, file w0sel.c, line 2210
> Mem block at + 3832, file m0rec.c, line 443
> Mem block at + 3936, file w0upd.c, line 289
> Mem block at + 4152, file 0pcur.c, line 28
> Freed mem block at + 4344, file w0sel.c, line 2210
> Mem block at + 15560, file w0sel.c, line 2210
> Mem block at + 17520, file w0sel.c, line 2210
> Freed mem block at + 17912, file w0sel.c, line 2210
> 040517  9:32:12  InnoDB: Assertion failure in thread 110620 in file
> mem0pool.c l
> ine 493
> InnoDB: Failing assertion: 0
> InnoDB: We intentionally generate a memory trap.
> InnoDB: Send a detailed bug report to mysql@xxxxxxxxxxxxxxx
> mysqld got signal 11;
> This could be because you hit a bug. It is also possible that this
> binary
> or one of the libraries it was linked against is corrupt, improperly
> built,
> or misconfigured. This error can also be caused by malfunctioning
> hardware.
> We will try our best to scrape up some info that will hopefully help
> diagnose
> the problem, but since we have already crashed, something is definitely
> wrong
> and this may fail.
> key_buffer_size=268435456
> read_buffer_size=1044480
> max_used_connections=70
> max_connections=100
> threads_connected=38
> It is possible that mysqld could use up to
> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
> = 466543
>  K
> bytes of memory
> Hope that's ok; if not, decrease some variables in the equation.
>
> thd=0x50e28b08
> Attempting backtrace. You can use the following information to find out
> where mysqld died. If you see no messages after this, something went
> terribly wrong...
> Cannot determine thread, fp=0xbfc3e728, backtrace may not be correct.
> Stack range sanity check OK, backtrace follows:
> 0x80705d0
> 0x8297878
> 0x822837c
> 0x82271d1
> 0x81322fd
> 0x8137047
> 0x80cbe07
> 0x80cbeb3
> 0x809dd75
> 0x809d109
> 0x809cdc0
> 0x8095402
> 0x8093876
> 0x807afd0
> 0x807e9ba
> 0x807a0c3
> 0x8079b1d
> 0x8079359
> 0x829502c
> 0x82be82a
> New value of fp=(nil) failed sanity check, terminating stack trace!
> Please read http://www.mysql.com/doc/en/Using_stack_trace.html
> and
> follow instru
> ctions on how to resolve the stack trace. Resolved
> stack trace is much more helpful in diagnosing the problem, so please do
> resolve it
> Trying to get some variables.
> Some pointers may be invalid and cause the dump to abort...
> thd->query at 0x8a7b610 = select * from Inbound where status = 'n' order
> by id
> thd->thread_id=18105
> thd->thread_id=18105
> The manual page at http://www.mysql.com/doc/en/Crashing.html
> contains
> information that should help you find out what is causing the crash.
>
> Number of processes running now: 0
> 040517 09:32:12  mysqld restarted
> 040517  9:32:13  InnoDB: Database was not shut down normally.
> InnoDB: Starting recovery from log files...
> InnoDB: Starting log scan based on checkpoint at
> InnoDB: log sequence number 1 2535462740
> InnoDB: Doing recovery: scanned up to log sequence number 1 2535485165
> 040517  9:32:13  InnoDB: Starting an apply batch of log records to the
> database.
> ..
> InnoDB: Progress in percents: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
> 20 21 22
>  23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
> 47 48 4
> 9 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
> 73 74 75
> 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
> InnoDB: Apply batch completed
> InnoDB: Last MySQL binlog file position 0 460531001, file name
> ./mish02-bin.046
> 040517  9:32:14  InnoDB: Flushing modified pages from the buffer pool...
> 040517  9:32:14  InnoDB: Started
> /usr/local/mysql-4.0.18/bin/mysqld: ready for connections.
> Version: '4.0.18-standard-log'  socket: '/var/lib/mysql/mysql.sock'
> port: 3306
>
> Rusty W. Shanklin, Director-IT
> Pikeville Medical Center
> (606) 218 - 4940
>


-- 
MySQL Bugs Mailing List
For list archives: http://lists.mysql.com/bugs
To unsubscribe:    http://lists.mysql.com/bugs?unsub=gcdmb-bugs@xxxxxxxxxxx