List:General Discussion« Previous MessageNext Message »
From:bert hubert Date:December 28 2001 7:17am
Subject:Re: [BUG] 'ENOTCONN' replication dies every once in a while
View as plain text  
On Sat, Dec 22, 2001 at 02:00:00PM +0100, bert hubert wrote:
> I can't easily replicate this as it happens only once every few weeks. We
> are very happy with MySQL otherwise! It is wonderful as a nameserver
> database backend.

One additional datapoint is that we are using a self-compiled version (I
don't really know why), not the binaries from the website. I've been doing
some further digging, but I don't really see a cause for this.

We're running Debian stable, so our glibc is pretty dated. Might that have
something to do with it?

Thanks for your attention!

FYI, the original post:

> We have a server in the US that replicates from a master in The Netherlands.
> Connectivity is usually very good. However, sometimes we see this error:
> 
> 011115 10:19:20  mysqld started
> /opt/mysql/libexec/mysqld: ready for connections
> 
> 011115 10:19:20  Slave: connected to master 'repl@stripped:3306',
> replication started in log 'brk-bin.001' at position 73
> 
> (three weeks of succesful replication pass!)
> 
> 011207 10:29:31  Error reading packet from server: Lost connection to MySQL
> server during query (read_errno 107,server_errno=2013)
> 
> 011207 10:29:31  Slave: Failed reading log event, reconnecting to retry, log
> 'brk-bin.001' position 229330
> 
> (30.0 seconds pass)
> 
> 011207 10:30:01  Slave thread: error re-connecting to master: Lost
> connection to MySQL server during query, last_errno=2013, retry in 60 sec
> 
> (60.0 seconds pass)
> 
> 011207 10:31:01  Slave thread killed during or after a reconnect done to
> recover from failed read
> 
> 011207 10:31:01  Slave thread exiting, replication stopped in log
> 'brk-bin.001' at position 229330
> 
> 011213  6:14:02  /opt/mysql/libexec/mysqld: Normal shutdown
> 011213  6:14:02  /opt/mysql/libexec/mysqld: Shutdown Complete
> 011213 06:14:02  mysqld ended
> 
> And some time later, the same happens:
> 
> 011213 06:14:07  mysqld started
> /opt/mysql/libexec/mysqld: ready for connections
> 
> 011213  6:14:07  Slave: connected to master 'repl@stripped:3306',
> replication started in log 'brk-bin.001' at position 229330
> 
> (5 days pass)
> 
> 011218 21:50:34  Error reading packet from server: Lost connection to MySQL
> server during query (read_errno 107,server_errno=2013)
> 
> 011218 21:50:34  Slave: Failed reading log event, reconnecting to retry, log
> 'brk-bin.001' position 342556
> 
> (30 seconds pass)
> 
> 011218 21:51:04  Slave thread: error re-connecting to master: Lost
> connection to MySQL server during query, last_errno=2013, retry in 60 sec
> 
> (60.0 seconds pass)
> 
> 011218 21:52:04  Slave thread killed during or after a reconnect done to
> recover from failed read
> 
> 011218 21:52:04  Slave thread exiting, replication stopped in log
> 'brk-bin.001' at position 342556
> 
> Now, read_errno=107 is 'ENOTCONN', which indicates trying to read from a
> socket that isn't connected to anything yet. master_retry_count is set to 0.
> From my readying, it appears as if mc_mysql_reconnect() returns zero on the
> second retry, causing the loop to terminate.
> 
> So the timeline is this:
> 
> 1) Failed reading log event because of ENOTCONN.
> 2) Immediately try to reconnect, which fails after 30 seconds with the same
>    MySQL error, 'Lost connection to MySQL server during query'
> 3) Wait 60 seconds
> 4) Again try to reconnect, which fails immediately. By this time, the slave
>    is killed, and the loop terminates.
> 
> Now, I can't figure out who killed the slave! I think that is the crux of
> the matter.
> 
> Version details: Linux 2.2.18 on the slave, 2.4.12-ac3 on the master. Both
> running:
> /opt/mysql/bin/mysql  Ver 11.15 Distrib 3.23.43, for pc-linux-gnu (i686)
> 
> Help! :-)
> 
> Extended-status of when it is running well (on the slave):
> 
> +--------------------------+--------+
> | Variable_name            | Value  |
> +--------------------------+--------+
> | Aborted_clients          | 0      |
> | Aborted_connects         | 0      |
> | Bytes_received           | 108186 |
> | Bytes_sent               | 209295 |
> | Connections              | 24     |
> | Created_tmp_disk_tables  | 0      |
> | Created_tmp_tables       | 0      |
> | Created_tmp_files        | 0      |
> | Delayed_insert_threads   | 0      |
> | Delayed_writes           | 0      |
> | Delayed_errors           | 0      |
> | Flush_commands           | 1      |
> | Handler_delete           | 0      |
> | Handler_read_first       | 1      |
> | Handler_read_key         | 863    |
> | Handler_read_next        | 1799   |
> | Handler_read_prev        | 0      |
> | Handler_read_rnd         | 0      |
> | Handler_read_rnd_next    | 119410 |
> | Handler_update           | 0      |
> | Handler_write            | 0      |
> | Key_blocks_used          | 33     |
> | Key_read_requests        | 10432  |
> | Key_reads                | 33     |
> | Key_write_requests       | 0      |
> | Key_writes               | 0      |
> | Max_used_connections     | 12     |
> | Not_flushed_key_blocks   | 0      |
> | Not_flushed_delayed_rows | 0      |
> | Open_tables              | 6      |
> | Open_files               | 13     |
> | Open_streams             | 0      |
> | Opened_tables            | 12     |
> | Questions                | 946    |
> | Select_full_join         | 0      |
> | Select_full_range_join   | 0      |
> | Select_range             | 0      |
> | Select_range_check       | 0      |
> | Select_scan              | 54     |
> | Slave_running            | ON     |
> | Slave_open_temp_tables   | 0      |
> | Slow_launch_threads      | 0      |
> | Slow_queries             | 0      |
> | Sort_merge_passes        | 0      |
> | Sort_range               | 0      |
> | Sort_rows                | 0      |
> | Sort_scan                | 0      |
> | Table_locks_immediate    | 923    |
> | Table_locks_waited       | 0      |
> | Threads_cached           | 0      |
> | Threads_created          | 22     |
> | Threads_connected        | 13     |
> | Threads_running          | 1      |
> | Uptime                   | 70098  |
> +--------------------------+--------+
> 
> Variables from same:
> 
>
> +---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
> | Variable_name                   | Value                                            
>                                                                                           
>              |
>
> +---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
> | back_log                        | 50                                               
>                                                                                           
>              |
> | basedir                         | /opt/mysql/                                      
>                                                                                           
>              |
> | binlog_cache_size               | 32768                                            
>                                                                                           
>              |
> | character_set                   | latin1                                           
>                                                                                           
>              |
> | character_sets                  | latin1 dec8 dos german1 hp8 koi8_ru latin2 swe7
> usa7 cp1251 danish hebrew win1251 estonia hungarian koi8_ukr win1251ukr greek win1250
> croat cp1257 latin5 |
> | concurrent_insert               | ON                                               
>                                                                                           
>              |
> | connect_timeout                 | 5                                                
>                                                                                           
>              |
> | datadir                         | /opt/databases/xxx/data/                         
>                                                                                           
>       |
> | delay_key_write                 | ON                                               
>                                                                                           
>              |
> | delayed_insert_limit            | 100                                              
>                                                                                           
>              |
> | delayed_insert_timeout          | 300                                              
>                                                                                           
>              |
> | delayed_queue_size              | 1000                                             
>                                                                                           
>              |
> | flush                           | OFF                                              
>                                                                                           
>              |
> | flush_time                      | 0                                                
>                                                                                           
>              |
> | have_bdb                        | NO                                               
>                                                                                           
>              |
> | have_gemini                     | NO                                               
>                                                                                           
>              |
> | have_innodb                     | NO                                               
>                                                                                           
>              |
> | have_isam                       | YES                                              
>                                                                                           
>              |
> | have_raid                       | NO                                               
>                                                                                           
>              |
> | have_openssl                    | NO                                               
>                                                                                           
>              |
> | init_file                       |                                                  
>                                                                                           
>              |
> | interactive_timeout             | 28800                                            
>                                                                                           
>              |
> | join_buffer_size                | 131072                                           
>                                                                                           
>              |
> | key_buffer_size                 | 8388600                                          
>                                                                                           
>              |
> | language                        | /opt/mysql/share/mysql/english/                  
>                                                                                           
>              |
> | large_files_support             | ON                                               
>                                                                                           
>              |
> | locked_in_memory                | OFF                                              
>                                                                                           
>              |
> | log                             | OFF                                              
>                                                                                           
>              |
> | log_update                      | OFF                                              
>                                                                                           
>              |
> | log_bin                         | OFF                                              
>                                                                                           
>              |
> | log_slave_updates               | OFF                                              
>                                                                                           
>              |
> | log_long_queries                | OFF                                              
>                                                                                           
>              |
> | long_query_time                 | 10                                               
>                                                                                           
>              |
> | low_priority_updates            | OFF                                              
>                                                                                           
>              |
> | lower_case_table_names          | 0                                                
>                                                                                           
>              |
> | max_allowed_packet              | 1048576                                          
>                                                                                           
>              |
> | max_binlog_cache_size           | 4294967295                                       
>                                                                                           
>              |
> | max_binlog_size                 | 1073741824                                       
>                                                                                           
>              |
> | max_connections                 | 100                                              
>                                                                                           
>              |
> | max_connect_errors              | 10                                               
>                                                                                           
>              |
> | max_delayed_threads             | 20                                               
>                                                                                           
>              |
> | max_heap_table_size             | 16777216                                         
>                                                                                           
>              |
> | max_join_size                   | 4294967295                                       
>                                                                                           
>              |
> | max_sort_length                 | 1024                                             
>                                                                                           
>              |
> | max_user_connections            | 0                                                
>                                                                                           
>              |
> | max_tmp_tables                  | 32                                               
>                                                                                           
>              |
> | max_write_lock_count            | 4294967295                                       
>                                                                                           
>              |
> | myisam_max_extra_sort_file_size | 256                                              
>                                                                                           
>              |
> | myisam_max_sort_file_size       | 2047                                             
>                                                                                           
>              |
> | myisam_recover_options          | 0                                                
>                                                                                           
>              |
> | myisam_sort_buffer_size         | 8388608                                          
>                                                                                           
>              |
> | net_buffer_length               | 16384                                            
>                                                                                           
>              |
> | net_read_timeout                | 30                                               
>                                                                                           
>              |
> | net_retry_count                 | 10                                               
>                                                                                           
>              |
> | net_write_timeout               | 60                                               
>                                                                                           
>              |
> | open_files_limit                | 0                                                
>                                                                                           
>              |
> | pid_file                        | /opt/databases/xxx/var/mysql.pid                 
>                                                                                           
>       |
> | port                            | 3306                                             
>                                                                                           
>              |
> | protocol_version                | 10                                               
>                                                                                           
>              |
> | record_buffer                   | 131072                                           
>                                                                                           
>              |
> | record_rnd_buffer               | 131072                                           
>                                                                                           
>              |
> | query_buffer_size               | 0                                                
>                                                                                           
>              |
> | safe_show_database              | OFF                                              
>                                                                                           
>              |
> | server_id                       | 336                                              
>                                                                                           
>              |
> | slave_net_timeout               | 3600                                             
>                                                                                           
>              |
> | skip_locking                    | ON                                               
>                                                                                           
>              |
> | skip_networking                 | OFF                                              
>                                                                                           
>              |
> | skip_show_database              | OFF                                              
>                                                                                           
>              |
> | slow_launch_time                | 2                                                
>                                                                                           
>              |
> | socket                          | /opt/databases/xxx/var/mysql.sock                
>                                                                                           
>       |
> | sort_buffer                     | 2097144                                          
>                                                                                           
>              |
> | sql_mode                        | 0                                                
>                                                                                           
>              |
> | table_cache                     | 64                                               
>                                                                                           
>              |
> | table_type                      | MYISAM                                           
>                                                                                           
>              |
> | thread_cache_size               | 0                                                
>                                                                                           
>              |
> | thread_stack                    | 65536                                            
>                                                                                           
>              |
> | transaction_isolation           | READ-COMMITTED                                   
>                                                                                           
>              |
> | timezone                        | EST                                              
>                                                                                           
>              |
> | tmp_table_size                  | 33554432                                         
>                                                                                           
>              |
> | tmpdir                          | /tmp/                                            
>                                                                                           
>              |
> | version                         | 3.23.43                                          
>                                                                                           
>              |
> | wait_timeout                    | 28800                                            
>                                                                                           
>              |
>
> +---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
> 
> Thanks for your attention.
> 
> Regards,
> 
> bert hubert
> 
> -- 
> http://www.PowerDNS.com          Versatile DNS Software & Services
> http://www.tk                              the dot in .tk
> Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
> Linux Advanced Routing & Traffic Control: http://ds9a.nl/lartc
> 
> ---------------------------------------------------------------------
> Before posting, please check:
>    http://www.mysql.com/manual.php   (the manual)
>    http://lists.mysql.com/           (the list archive)
> 
> To request this thread, e-mail <mysql-thread94644@stripped>
> To unsubscribe, e-mail
> <mysql-unsubscribe-ahu=outpost.powerdns.com@stripped>
> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
> 
> 

-- 
http://www.PowerDNS.com          Versatile DNS Software & Services
http://www.tk                              the dot in .tk
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
Linux Advanced Routing & Traffic Control: http://ds9a.nl/lartc
Thread
[BUG] 'ENOTCONN' replication dies every once in a whilebert hubert22 Dec
  • Re: [BUG] 'ENOTCONN' replication dies every once in a whilebert hubert28 Dec