List:General Discussion« Previous MessageNext Message »
From:Lawrence Robertson Date:September 21 2009 7:23pm
Subject:100% CPU load problem
View as plain text  
Hi. 

We have some MySql servers in a circular replication, and one of the servers is having
some performance issues for some weeks. I tried with mysqlcheck and nothing is wrong. I'm
logging the slow queries and there's nothing wrong. 

The complete database in this server is like 1GB when it's GZipped. And almost 10GB in the
/var/lib/mysql directories, without bin-logs or ibdata files. It's a CentOS 4.4 Linux and
a MySql 5.0.27 server.


Here is my STATUS, SHOW STATUS, SHOW VARIABLES and TOP output. Hope it's useful. 

mysql> status
--------------
mysql  Ver 14.12 Distrib 5.0.27, for Win32 (ia32)

Connection id:          30
Current database:
Current user:           root@stripped
SSL:                    Not in use
Using delimiter:        ;
Server version:         5.0.26-standard-log
Protocol version:       10
Connection:             192.168.0.19 via TCP/IP
Server characterset:    latin1
Db     characterset:    latin1
Client characterset:    latin1
Conn.  characterset:    latin1
TCP port:               6419
Uptime:                 17 min 46 sec

Threads: 17  Questions: 314  Slow queries: 0  Opens: 37  Flush tables: 1  Open t
ables: 31  Queries per second avg: 0.295
--------------


mysql> show status;
+-----------------------------------+-----------+
| Variable_name                     | Value     |
+-----------------------------------+-----------+
| Aborted_clients                   | 2         |
| Aborted_connects                  | 0         |
| Binlog_cache_disk_use             | 0         |
| Binlog_cache_use                  | 0         |
| Bytes_received                    | 782       |
| Bytes_sent                        | 14389     |
| Com_admin_commands                | 0         |
| Com_alter_db                      | 0         |
| Com_alter_table                   | 0         |
| Com_analyze                       | 0         |
| Com_backup_table                  | 0         |
| Com_begin                         | 0         |
| Com_change_db                     | 0         |
| Com_change_master                 | 0         |
| Com_check                         | 0         |
| Com_checksum                      | 0         |
| Com_commit                        | 0         |
| Com_create_db                     | 0         |
| Com_create_function               | 0         |
| Com_create_index                  | 0         |
| Com_create_table                  | 0         |
| Com_dealloc_sql                   | 0         |
| Com_delete                        | 0         |
| Com_delete_multi                  | 0         |
| Com_do                            | 0         |
| Com_drop_db                       | 0         |
| Com_drop_function                 | 0         |
| Com_drop_index                    | 0         |
| Com_drop_table                    | 0         |
| Com_drop_user                     | 0         |
| Com_execute_sql                   | 0         |
| Com_flush                         | 0         |
| Com_grant                         | 0         |
| Com_ha_close                      | 0         |
| Com_ha_open                       | 0         |
| Com_ha_read                       | 0         |
| Com_help                          | 0         |
| Com_insert                        | 0         |
| Com_insert_select                 | 0         |
| Com_kill                          | 0         |
| Com_load                          | 0         |
| Com_load_master_data              | 0         |
| Com_load_master_table             | 0         |
| Com_lock_tables                   | 0         |
| Com_optimize                      | 0         |
| Com_preload_keys                  | 0         |
| Com_prepare_sql                   | 0         |
| Com_purge                         | 0         |
| Com_purge_before_date             | 0         |
| Com_rename_table                  | 0         |
| Com_repair                        | 0         |
| Com_replace                       | 0         |
| Com_replace_select                | 0         |
| Com_reset                         | 0         |
| Com_restore_table                 | 0         |
| Com_revoke                        | 0         |
| Com_revoke_all                    | 0         |
| Com_rollback                      | 0         |
| Com_savepoint                     | 0         |
| Com_select                        | 8         |
| Com_set_option                    | 0         |
| Com_show_binlog_events            | 0         |
| Com_show_binlogs                  | 0         |
| Com_show_charsets                 | 0         |
| Com_show_collations               | 0         |
| Com_show_column_types             | 0         |
| Com_show_create_db                | 0         |
| Com_show_create_table             | 0         |
| Com_show_databases                | 0         |
| Com_show_errors                   | 0         |
| Com_show_fields                   | 0         |
| Com_show_grants                   | 0         |
| Com_show_innodb_status            | 0         |
| Com_show_keys                     | 0         |
| Com_show_logs                     | 0         |
| Com_show_master_status            | 0         |
| Com_show_ndb_status               | 0         |
| Com_show_new_master               | 0         |
| Com_show_open_tables              | 0         |
| Com_show_privileges               | 0         |
| Com_show_processlist              | 0         |
| Com_show_slave_hosts              | 0         |
| Com_show_slave_status             | 0         |
| Com_show_status                   | 7         |
| Com_show_storage_engines          | 0         |
| Com_show_tables                   | 0         |
| Com_show_triggers                 | 0         |
| Com_show_variables                | 0         |
| Com_show_warnings                 | 0         |
| Com_slave_start                   | 0         |
| Com_slave_stop                    | 0         |
| Com_stmt_close                    | 0         |
| Com_stmt_execute                  | 0         |
| Com_stmt_fetch                    | 0         |
| Com_stmt_prepare                  | 0         |
| Com_stmt_reset                    | 0         |
| Com_stmt_send_long_data           | 0         |
| Com_truncate                      | 0         |
| Com_unlock_tables                 | 0         |
| Com_update                        | 0         |
| Com_update_multi                  | 0         |
| Com_xa_commit                     | 0         |
| Com_xa_end                        | 0         |
| Com_xa_prepare                    | 0         |
| Com_xa_recover                    | 0         |
| Com_xa_rollback                   | 0         |
| Com_xa_start                      | 0         |
| Compression                       | OFF       |
| Connections                       | 47        |
| Created_tmp_disk_tables           | 0         |
| Created_tmp_files                 | 5         |
| Created_tmp_tables                | 3         |
| Delayed_errors                    | 0         |
| Delayed_insert_threads            | 0         |
| Delayed_writes                    | 0         |
| Flush_commands                    | 1         |
| Handler_commit                    | 0         |
| Handler_delete                    | 0         |
| Handler_discover                  | 0         |
| Handler_prepare                   | 0         |
| Handler_read_first                | 0         |
| Handler_read_key                  | 0         |
| Handler_read_next                 | 0         |
| Handler_read_prev                 | 0         |
| Handler_read_rnd                  | 0         |
| Handler_read_rnd_next             | 492       |
| Handler_rollback                  | 0         |
| Handler_savepoint                 | 0         |
| Handler_savepoint_rollback        | 0         |
| Handler_update                    | 0         |
| Handler_write                     | 620       |
| Innodb_buffer_pool_pages_data     | 161       |
| Innodb_buffer_pool_pages_dirty    | 0         |
| Innodb_buffer_pool_pages_flushed  | 18        |
| Innodb_buffer_pool_pages_free     | 48991     |
| Innodb_buffer_pool_pages_latched  | 2         |
| Innodb_buffer_pool_pages_misc     | 0         |
| Innodb_buffer_pool_pages_total    | 49152     |
| Innodb_buffer_pool_read_ahead_rnd | 1         |
| Innodb_buffer_pool_read_ahead_seq | 0         |
| Innodb_buffer_pool_read_requests  | 325529426 |
| Innodb_buffer_pool_reads          | 102       |
| Innodb_buffer_pool_wait_free      | 0         |
| Innodb_buffer_pool_write_requests | 22        |
| Innodb_data_fsyncs                | 29        |
| Innodb_data_pending_fsyncs        | 0         |
| Innodb_data_pending_reads         | 0         |
| Innodb_data_pending_writes        | 0         |
| Innodb_data_read                  | 4820992   |
| Innodb_data_reads                 | 133       |
| Innodb_data_writes                | 37        |
| Innodb_data_written               | 598016    |
| Innodb_dblwr_pages_written        | 18        |
| Innodb_dblwr_writes               | 5         |
| Innodb_log_waits                  | 0         |
| Innodb_log_write_requests         | 2         |
| Innodb_log_writes                 | 7         |
| Innodb_os_log_fsyncs              | 14        |
| Innodb_os_log_pending_fsyncs      | 0         |
| Innodb_os_log_pending_writes      | 0         |
| Innodb_os_log_written             | 4608      |
| Innodb_page_size                  | 16384     |
| Innodb_pages_created              | 0         |
| Innodb_pages_read                 | 161       |
| Innodb_pages_written              | 18        |
| Innodb_row_lock_current_waits     | 0         |
| Innodb_row_lock_time              | 0         |
| Innodb_row_lock_time_avg          | 0         |
| Innodb_row_lock_time_max          | 0         |
| Innodb_row_lock_waits             | 0         |
| Innodb_rows_deleted               | 0         |
| Innodb_rows_inserted              | 0         |
| Innodb_rows_read                  | 2795      |
| Innodb_rows_updated               | 0         |
| Key_blocks_not_flushed            | 0         |
| Key_blocks_unused                 | 646466    |
| Key_blocks_used                   | 252       |
| Key_read_requests                 | 1912      |
| Key_reads                         | 252       |
| Key_write_requests                | 155       |
| Key_writes                        | 123       |
| Last_query_cost                   | 0.000000  |
| Max_used_connections              | 19        |
| Not_flushed_delayed_rows          | 0         |
| Open_files                        | 68        |
| Open_streams                      | 0         |
| Open_tables                       | 31        |
| Opened_tables                     | 0         |
| Qcache_free_blocks                | 1         |
| Qcache_free_memory                | 67032416  |
| Qcache_hits                       | 52        |
| Qcache_inserts                    | 36        |
| Qcache_lowmem_prunes              | 0         |
| Qcache_not_cached                 | 31        |
| Qcache_queries_in_cache           | 36        |
| Qcache_total_blocks               | 79        |
| Questions                         | 324       |
| Rpl_status                        | NULL      |
| Select_full_join                  | 0         |
| Select_full_range_join            | 0         |
| Select_range                      | 0         |
| Select_range_check                | 0         |
| Select_scan                       | 3         |
| Slave_open_temp_tables            | 0         |
| Slave_retried_transactions        | 0         |
| Slave_running                     | ON        |
| Slow_launch_threads               | 0         |
| Slow_queries                      | 0         |
| Sort_merge_passes                 | 0         |
| Sort_range                        | 0         |
| Sort_rows                         | 0         |
| Sort_scan                         | 0         |
| Ssl_accept_renegotiates           | 0         |
| Ssl_accepts                       | 0         |
| Ssl_callback_cache_hits           | 0         |
| Ssl_cipher                        |           |
| Ssl_cipher_list                   |           |
| Ssl_client_connects               | 0         |
| Ssl_connect_renegotiates          | 0         |
| Ssl_ctx_verify_depth              | 0         |
| Ssl_ctx_verify_mode               | 0         |
| Ssl_default_timeout               | 0         |
| Ssl_finished_accepts              | 0         |
| Ssl_finished_connects             | 0         |
| Ssl_session_cache_hits            | 0         |
| Ssl_session_cache_misses          | 0         |
| Ssl_session_cache_mode            | NONE      |
| Ssl_session_cache_overflows       | 0         |
| Ssl_session_cache_size            | 0         |
| Ssl_session_cache_timeouts        | 0         |
| Ssl_sessions_reused               | 0         |
| Ssl_used_session_cache_entries    | 0         |
| Ssl_verify_depth                  | 0         |
| Ssl_verify_mode                   | 0         |
| Ssl_version                       |           |
| Table_locks_immediate             | 193       |
| Table_locks_waited                | 0         |
| Tc_log_max_pages_used             | 0         |
| Tc_log_page_size                  | 0         |
| Tc_log_page_waits                 | 0         |
| Threads_cached                    | 2         |
| Threads_connected                 | 17        |
| Threads_created                   | 19        |
| Threads_running                   | 10        |
| Uptime                            | 1109      |
+-----------------------------------+-----------+
245 rows in set (0.00 sec)

mysql> show variables;
+---------------------------------+------------------------------------------+
| Variable_name                   | Value                                    |
+---------------------------------+------------------------------------------+
| auto_increment_increment        | 1                                        |
| auto_increment_offset           | 1                                        |
| automatic_sp_privileges         | ON                                       |
| back_log                        | 50                                       |
| basedir                         | /                                        |
| binlog_cache_size               | 32768                                    |
| bulk_insert_buffer_size         | 8388608                                  |
| character_set_client            | latin1                                   |
| character_set_connection        | latin1                                   |
| character_set_database          | latin1                                   |
| character_set_filesystem        | binary                                   |
| character_set_results           | latin1                                   |
| character_set_server            | latin1                                   |
| character_set_system            | utf8                                     |
| character_sets_dir              | /usr/share/mysql/charsets/               |
| collation_connection            | latin1_swedish_ci                        |
| collation_database              | latin1_swedish_ci                        |
| collation_server                | latin1_swedish_ci                        |
| completion_type                 | 0                                        |
| concurrent_insert               | 1                                        |
| connect_timeout                 | 5                                        |
| datadir                         | /var/lib/mysql/                          |
| date_format                     | %Y-%m-%d                                 |
| datetime_format                 | %Y-%m-%d %H:%i:%s                        |
| default_week_format             | 0                                        |
| delay_key_write                 | ON                                       |
| delayed_insert_limit            | 100                                      |
| delayed_insert_timeout          | 300                                      |
| delayed_queue_size              | 1000                                     |
| div_precision_increment         | 4                                        |
| engine_condition_pushdown       | OFF                                      |
| expire_logs_days                | 0                                        |
| flush                           | OFF                                      |
| flush_time                      | 0                                        |
| ft_boolean_syntax               | + -><()~*:""&|                           |
| ft_max_word_len                 | 84                                       |
| ft_min_word_len                 | 4                                        |
| ft_query_expansion_limit        | 20                                       |
| ft_stopword_file                | (built-in)                               |
| group_concat_max_len            | 1024                                     |
| have_archive                    | YES                                      |
| have_bdb                        | NO                                       |
| have_blackhole_engine           | NO                                       |
| have_compress                   | YES                                      |
| have_crypt                      | YES                                      |
| have_csv                        | NO                                       |
| have_dynamic_loading            | YES                                      |
| have_example_engine             | NO                                       |
| have_federated_engine           | NO                                       |
| have_geometry                   | YES                                      |
| have_innodb                     | YES                                      |
| have_isam                       | NO                                       |
| have_merge_engine               | YES                                      |
| have_ndbcluster                 | NO                                       |
| have_openssl                    | DISABLED                                 |
| have_query_cache                | YES                                      |
| have_raid                       | NO                                       |
| have_rtree_keys                 | YES                                      |
| have_symlink                    | YES                                      |
| init_connect                    |                                          |
| init_file                       |                                          |
| init_slave                      |                                          |
| innodb_additional_mem_pool_size | 41943040                                 |
| innodb_autoextend_increment     | 8                                        |
| innodb_buffer_pool_awe_mem_mb   | 0                                        |
| innodb_buffer_pool_size         | 805306368                                |
| innodb_checksums                | ON                                       |
| innodb_commit_concurrency       | 0                                        |
| innodb_concurrency_tickets      | 500                                      |
| innodb_data_file_path           | ibdata1:512M;ibdata2:10M:autoextend      |
| innodb_data_home_dir            | /var/lib/mysql/                          |
| innodb_doublewrite              | ON                                       |
| innodb_fast_shutdown            | 1                                        |
| innodb_file_io_threads          | 4                                        |
| innodb_file_per_table           | ON                                       |
| innodb_flush_log_at_trx_commit  | 1                                        |
| innodb_flush_method             |                                          |
| innodb_force_recovery           | 0                                        |
| innodb_lock_wait_timeout        | 50                                       |
| innodb_locks_unsafe_for_binlog  | OFF                                      |
| innodb_log_arch_dir             | /var/lib/mysql/                          |
| innodb_log_archive              | OFF                                      |
| innodb_log_buffer_size          | 33554432                                 |
| innodb_log_file_size            | 536870912                                |
| innodb_log_files_in_group       | 2                                        |
| innodb_log_group_home_dir       | /var/lib/mysql/                          |
| innodb_max_dirty_pages_pct      | 90                                       |
| innodb_max_purge_lag            | 0                                        |
| innodb_mirrored_log_groups      | 1                                        |
| innodb_open_files               | 300                                      |
| innodb_support_xa               | ON                                       |
| innodb_sync_spin_loops          | 20                                       |
| innodb_table_locks              | ON                                       |
| innodb_thread_concurrency       | 8                                        |
| innodb_thread_sleep_delay       | 10000                                    |
| interactive_timeout             | 28800                                    |
| join_buffer_size                | 131072                                   |
| key_buffer_size                 | 805306368                                |
| key_cache_age_threshold         | 300                                      |
| key_cache_block_size            | 1024                                     |
| key_cache_division_limit        | 100                                      |
| language                        | /usr/share/mysql/english/                |
| large_files_support             | ON                                       |
| large_page_size                 | 0                                        |
| large_pages                     | OFF                                      |
| lc_time_names                   | en_US                                    |
| license                         | GPL                                      |
| local_infile                    | ON                                       |
| locked_in_memory                | OFF                                      |
| log                             | OFF                                      |
| log_bin                         | ON                                       |
| log_bin_trust_function_creators | OFF                                      |
| log_error                       |                                          |
| log_queries_not_using_indexes   | OFF                                      |
| log_slave_updates               | ON                                       |
| log_slow_queries                | ON                                       |
| log_warnings                    | 1                                        |
| long_query_time                 | 1                                        |
| low_priority_updates            | OFF                                      |
| lower_case_file_system          | OFF                                      |
| lower_case_table_names          | 0                                        |
| max_allowed_packet              | 1047552                                  |
| max_binlog_cache_size           | 18446744073709551615                     |
| max_binlog_size                 | 52428800                                 |
| max_connect_errors              | 10                                       |
| max_connections                 | 200                                      |
| max_delayed_threads             | 20                                       |
| max_error_count                 | 64                                       |
| max_heap_table_size             | 16777216                                 |
| max_insert_delayed_threads      | 20                                       |
| max_join_size                   | 18446744073709551615                     |
| max_length_for_sort_data        | 1024                                     |
| max_prepared_stmt_count         | 16382                                    |
| max_relay_log_size              | 0                                        |
| max_seeks_for_key               | 18446744073709551615                     |
| max_sort_length                 | 1024                                     |
| max_sp_recursion_depth          | 0                                        |
| max_tmp_tables                  | 32                                       |
| max_user_connections            | 0                                        |
| max_write_lock_count            | 18446744073709551615                     |
| multi_range_count               | 256                                      |
| myisam_data_pointer_size        | 6                                        |
| myisam_max_sort_file_size       | 9223372036854775807                      |
| myisam_recover_options          | OFF                                      |
| myisam_repair_threads           | 1                                        |
| myisam_sort_buffer_size         | 134217728                                |
| myisam_stats_method             | nulls_unequal                            |
| net_buffer_length               | 16384                                    |
| net_read_timeout                | 30                                       |
| net_retry_count                 | 10                                       |
| net_write_timeout               | 60                                       |
| new                             | OFF                                      |
| old_passwords                   | ON                                       |
| open_files_limit                | 1234                                     |
| optimizer_prune_level           | 1                                        |
| optimizer_search_depth          | 62                                       |
| pid_file                        | /var/lib/mysql/server6419.pid            |
| port                            | 6419                                     |
| preload_buffer_size             | 32768                                    |
| prepared_stmt_count             | 0                                        |
| protocol_version                | 10                                       |
| query_alloc_block_size          | 8192                                     |
| query_cache_limit               | 1048576                                  |
| query_cache_min_res_unit        | 4096                                     |
| query_cache_size                | 67108864                                 |
| query_cache_type                | ON                                       |
| query_cache_wlock_invalidate    | OFF                                      |
| query_prealloc_size             | 8192                                     |
| range_alloc_block_size          | 2048                                     |
| read_buffer_size                | 4190208                                  |
| read_only                       | OFF                                      |
| read_rnd_buffer_size            | 16773120                                 |
| relay_log_purge                 | ON                                       |
| relay_log_space_limit           | 0                                        |
| rpl_recovery_rank               | 0                                        |
| secure_auth                     | OFF                                      |
| server_id                       | 19                                       |
| skip_external_locking           | ON                                       |
| skip_networking                 | OFF                                      |
| skip_show_database              | OFF                                      |
| slave_compressed_protocol       | OFF                                      |
| slave_load_tmpdir               | /tmp/                                    |
| slave_net_timeout               | 3600                                     |
| slave_skip_errors               | OFF                                      |
| slave_transaction_retries       | 10                                       |
| slow_launch_time                | 2                                        |
| socket                          | /var/lib/mysql/mysql.sock                |
| sort_buffer_size                | 4194296                                  |
| sql_big_selects                 | ON                                       |
| sql_mode                        |                                          |
| sql_notes                       | ON                                       |
| sql_warnings                    | OFF                                      |
| ssl_ca                          |                                          |
| ssl_capath                      |                                          |
| ssl_cert                        |                                          |
| ssl_cipher                      |                                          |
| ssl_key                         |                                          |
| storage_engine                  | MyISAM                                   |
| sync_binlog                     | 0                                        |
| sync_frm                        | ON                                       |
| system_time_zone                | CDT                                      |
| table_cache                     | 512                                      |
| table_lock_wait_timeout         | 50                                       |
| table_type                      | MyISAM                                   |
| thread_cache_size               | 8                                        |
| thread_stack                    | 262144                                   |
| time_format                     | %H:%i:%s                                 |
| time_zone                       | SYSTEM                                   |
| timed_mutexes                   | OFF                                      |
| tmp_table_size                  | 33554432                                 |
| tmpdir                          | /tmp/                                    |
| transaction_alloc_block_size    | 8192                                     |
| transaction_prealloc_size       | 4096                                     |
| tx_isolation                    | REPEATABLE-READ                          |
| updatable_views_with_limit      | YES                                      |
| version                         | 5.0.26-standard-log                      |
| version_comment                 | MySQL Community Edition - Standard (GPL) |
| version_compile_machine         | x86_64                                   |
| version_compile_os              | unknown-linux-gnu                        |
| wait_timeout                    | 28800                                    |
+---------------------------------+------------------------------------------+
220 rows in set (0.00 sec)

[root@server6419 ~]# top
top - 14:06:21 up  1:58,  3 users,  load average: 0.93, 0.97, 0.89
Tasks: 108 total,   2 running, 106 sleeping,   0 stopped,   0 zombie
Cpu(s): 49.9% us,  0.2% sy,  0.0% ni, 49.9% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   2048128k total,  1356920k used,   691208k free,    41012k buffers
Swap:  2031608k total,        0k used,  2031608k free,   844688k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4812 mysql     16   0 1844m 246m 3900 S  100 12.3  23:44.19 mysqld
 3214 root      16   0 18572 5832 1264 S    0  0.3   0:01.53 hald
    1 root      16   0  4756  548  456 S    0  0.0   0:00.51 init
    2 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
    3 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0
    4 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1
    5 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1


Thanks a lot!
Thread
100% CPU load problemLawrence Robertson21 Sep
  • Re: 100% CPU load problemKyong Kim21 Sep
    • Re: 100% CPU load problemLawrence Robertson22 Sep
    • Re: 100% CPU load problemLawrence Robertson22 Sep