List:General Discussion« Previous MessageNext Message »
From:Clif Smith Date:November 17 2004 8:27pm
Subject:Re: Server Won't Start "Next record offset is nonsensical"
View as plain text  
 >Cliff,

>
> your OS or hardware has probably corrupted the ibdata file.

Ouch!

>> Next record offset is nonsensical 28769 in record at offset 7022
>
>
> Before writing an index page to the file, InnoDB checks that offsets 
> are sensible (< 16 kB).
>
>> InnoDB: rec address 407b1b6e, first buffer frame 401c0000
>> InnoDB: buffer pool high end 409c0000, buf fix count 1
>> 041116 17:17:10  InnoDB: Page dump in ascii and hex (16384 bytes):
>> len 16384; hex
>>
>> <large snip>
>>
>> ";InnoDB: End of page dump
>> 74 041116 17:17:10  InnoDB: Page checksum 3244520732,
>> prior-to-4.0.14-form checksum 1495873249
>> InnoDB: stored checksum 4145305205, prior-to-4.0.14-form stored 
>> checksum 0
>
> Hmm... the 'old checksum' at offsets 16 kB - 8 ... 16 kB - 4 is zero. 
> That is probably file corruption.
>
>> InnoDB: Resetting space id's in the doublewrite buffer
>
> "
>        if (mach_read_from_4(doublewrite + 
> TRX_SYS_DOUBLEWRITE_SPACE_ID_STORED)
>            != TRX_SYS_DOUBLEWRITE_SPACE_ID_STORED_N) {
>
>                /* We are upgrading from a version < 4.1.x to a version 
> where
>                multiple tablespaces are supported. We must reset the 
> space id
>                field in the pages in the doublewrite buffer because 
> starting
>                from this version the space id is stored to
>                FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID. */
>
>                trx_doublewrite_must_reset_space_ids = TRUE;
>
>                fprintf(stderr,
> "InnoDB: Resetting space id's in the doublewrite buffer\n");
>        } else {
>                trx_sys_multiple_tablespace_format = TRUE;
>        }
> "
>
> The printout looks like you tried a downgrade and upgrade of MySQL to 
> resolve the crash? Is that true? Which 4.1.x version you are running?
>
Yes, initially I had thought that there was an issue with the
installation itself, so I upgraded to v4.1.7-0.

> Please send the FULL .err log to me heikki.tuuri@stripped for
> more detailed analysis. Do not cut anything off.

Unfortunately the mailing list stripped your domain off.  Please send an 
email to spam at cjs226.com and I'll send you the err log.

>
> Your lsn is only 300 MB. High-end users of InnoDB on a stable Linux 
> computer can easily reach 300 GB without any corruption.
>
> Best regards,
>
> Heikki Tuuri
> Innobase Oy
> Foreign keys, transactions, and row level locking for MySQL
> InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up 
> MyISAM tables
> http://www.innodb.com/order.php
>
> Order MySQL technical support from https://order.mysql.com/
>
>
> ----- Original Message ----- From: "Clif Smith" <spam@stripped>
> Newsgroups: mailing.database.myodbc
> Sent: Wednesday, November 17, 2004 4:53 AM
> Subject: Server Won't Start "Next record offset is nonsensical"
>
>
>> Everything was fine...I haven't installed anything lately, etc.  I've
>> got a Fedora FC1 system running MySQL v4.  I noticed my db exports
>> failing this morning.  The db wasn't running and now won't startup.  I'm
>> googling but...  Here's what's in the log:
>>
>> 41116 17:17:09  mysqld started
>> 041116 17:17:09 [Warning] Asked for 196608 thread stack, but got 126976
>> 041116 17:17:09  InnoDB: Database was not shut down normally!
>> InnoDB: Starting crash recovery.
>> InnoDB: Reading tablespace information from the .ibd files...
>> InnoDB: Restoring possible half-written data pages from the doublewrite
>> InnoDB: buffer...
>> InnoDB: Resetting space id's in the doublewrite buffer
>> 041116 17:17:10  InnoDB: Starting log scan based on checkpoint at
>> InnoDB: log sequence number 0 296311265.
>> 041116 17:17:10  InnoDB: Starting an apply batch of log records to the
>> database...
>> InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
>> 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
>> 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 InnoDB:
>> Next record offset is nonsensical 28769 in record at offset 7022
>>
>> InnoDB: rec address 407b1b6e, first buffer frame 401c0000
>> InnoDB: buffer pool high end 409c0000, buf fix count 1
>> 041116 17:17:10  InnoDB: Page dump in ascii and hex (16384 bytes):
>> len 16384; hex
>>
>> <large snip>
>>
>> ";InnoDB: End of page dump
>> 74 041116 17:17:10  InnoDB: Page checksum 3244520732,
>> prior-to-4.0.14-form checksum 1495873249
>> InnoDB: stored checksum 4145305205, prior-to-4.0.14-form stored 
>> checksum 0
>> InnoDB: Page lsn 0 296329762, low 4 bytes of lsn at page end 296329762
>> InnoDB: Page number (if stored to page already) 6570,
>> InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
>> InnoDB: Page may be an index page where index id is 0 38
>> 041116 17:17:10InnoDB: Assertion failure in thread 12292 in file
>> ../include/page0page.ic line 494
>> InnoDB: Failing assertion: 0
>> InnoDB: We intentionally generate a memory trap.
>> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
>> InnoDB: If you get repeated assertion failures or crashes, even
>> InnoDB: immediately after the mysqld startup, there may be
>> InnoDB: corruption in the InnoDB tablespace. Please refer to
>> InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html
>> InnoDB: about forcing recovery.
>> 75 mysqld got signal 11;
>> This could be because you hit a bug. It is also possible that this 
>> binary
>> or one of the libraries it was linked against is corrupt, improperly 
>> built,
>> or misconfigured. This error can also be caused by malfunctioning 
>> hardware.
>> We will try our best to scrape up some info that will hopefully help
>> diagnose
>> the problem, but since we have already crashed, something is definitely
>> wrong
>> and this may fail.
>>
>> key_buffer_size=0
>> read_buffer_size=131072
>> max_used_connections=0
>> max_connections=100
>> threads_connected=0
>> It is possible that mysqld could use up to
>> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
>> = 217599 K
>> bytes of memory
>> Hope that's ok; if not, decrease some variables in the equation.
>>
>> 76 thd=(nil)
>> Attempting backtrace. You can use the following information to find out
>> where mysqld died. If you see no messages after this, something went
>> terribly wrong...
>> Cannot determine thread, fp=0xbff3ecb8, backtrace may not be correct.
>> Stack range sanity check OK, backtrace follows:
>> 0x808d737
>> 0x82e17a8
>> 0x825f3dd
>> 0x825ee95
>> 0x820d264
>> 0x820e2c5
>> 0x81f2751
>> 0x8231a83
>> 0x813ed39
>> 0x82def5c
>> 0x83088da
>> New value of fp=(nil) failed sanity check, terminating stack trace!
>> Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html
>> and
>> follow instructions on how to resolve the stack trace. Resolved
>> stack trace is much more helpful in diagnosing the problem, so please do
>> resolve it
>> The manual page at http://www.mysql.com/doc/en/Crashing.html
>> contains
>> information that should help you find out what is causing the crash.
>> 77 78 041116 17:17:10  mysqld ended
>> <end of log>
>>
>> I read the above and came up with the following:
>> 0x808d737 handle_segfault + 423
>> 0x82e17a8 pthread_sighandler + 184
>> 0x825f3dd page_cur_insert_rec_low + 1261
>> 0x825ee95 page_cur_parse_insert_rec + 3749
>> 0x820d264 recv_parse_or_apply_log_rec_body + 68
>> 0x820e2c5 recv_recover_page + 2933
>> 0x81f2751 buf_page_io_complete + 593
>> 0x8231a83 fil_aio_wait + 899
>> 0x813ed39 io_handler_thread + 25
>> 0x82def5c pthread_start_thread + 220
>> 0x83088da thread_start + 4
>>
>> But I'm just sad sys admin reading greek at this point...  Any ideas?
>>
>> -- 
>> MySQL General Mailing List
>> For list archives: http://lists.mysql.com/mysql
>> To unsubscribe: http://lists.mysql.com/mysql?unsub=1
>>
Thread
Server Won't Start "Next record offset is nonsensical"Clif Smith17 Nov
  • Re: Server Won't Start "Next record offset is nonsensical"Gleb Paharenko17 Nov
Re: Server Won't Start "Next record offset is nonsensical"Heikki Tuuri17 Nov
Re: Server Won't Start "Next record offset is nonsensical"Clif Smith17 Nov
Re: Server Won't Start "Next record offset is nonsensical"Heikki Tuuri18 Nov
Re: Server Won't Start "Next record offset is nonsensical"Heikki Tuuri18 Nov