bk commit into 5.1 tree (tnurnberg:1.2671) BUG#35272

[Tatjana A Nuernberg]

Below is the list of changes that have just been committed into a local
5.1 repository of tnurnberg.  When tnurnberg does a push these changes
will be propagated to the main repository and, within 24 hours after the
push, to the public repository.
For information on how to access the public repository
see http://dev.mysql.com/doc/mysql/en/installing-source-tree.html

ChangeSet@stripped, 2008-03-14 08:26:31+01:00, tnurnberg@stripped +3 -0
  Bug#35272: @@global.key_buffer_size = 4294967295 let the server crash
  
  When failing to get requested amount of memory for keybuffer,
  we signalled an error to the console and while trying to send
  OK to the client. Hilarity ensued.
  
  We are actually checking the return value of the function that
  detects the error condition now, and propagate it upwards.

  mysql-test/r/key_cache.result@stripped, 2008-03-14 08:26:27+01:00, tnurnberg@stripped +4 -0
    show that Out Of Memory on key_buffer_size no longer
    bombs with an assert(). show for 32- and 64-bit.

  mysql-test/t/key_cache.test@stripped, 2008-03-14 08:26:28+01:00, tnurnberg@stripped +25 -0
    show that Out Of Memory on key_buffer_size no longer
    bombs with an assert(). show for 32- and 64-bit.

  mysys/mf_keycache.c@stripped, 2008-03-14 08:26:28+01:00, tnurnberg@stripped +1 -1
    Actually check whether malloc worked.

diff -Nrup a/mysql-test/r/key_cache.result b/mysql-test/r/key_cache.result
--- a/mysql-test/r/key_cache.result	2007-11-30 14:52:49 +01:00
+++ b/mysql-test/r/key_cache.result	2008-03-14 08:26:27 +01:00
@@ -368,3 +368,7 @@ Variable_name	Value
 key_cache_block_size	1536
 SET GLOBAL key_cache_block_size= @bug28478_key_cache_block_size;
 DROP TABLE t1;
+SET @@global.key_buffer_size = 4294967295;
+SET @@global.key_buffer_size = 9223372036854775807;
+Got one of the listed errors
+SET @@global.key_buffer_size=@save_key_buffer;
diff -Nrup a/mysql-test/t/key_cache.test b/mysql-test/t/key_cache.test
--- a/mysql-test/t/key_cache.test	2007-05-31 20:04:51 +02:00
+++ b/mysql-test/t/key_cache.test	2008-03-14 08:26:28 +01:00
@@ -247,3 +247,28 @@ SET GLOBAL key_cache_block_size= @bug284
 DROP TABLE t1;
 
 # End of 4.1 tests
+
+
+
+#
+# Bug#35272: @@global.key_buffer_size = 4294967295 let the server crash
+#
+
+# Wee try to force Out Of Memory here. key_buffer_size is ULL, so
+# on a 32 bit machine, 4GB is the most we can ask for before the
+# server complains about value/variable mismatch. At the off chance
+# of one of our 64-bit machines actually offering us 4GB, we also
+# accept "no error" (in addition to the expected "out of memory")
+# here:
+--error 0,5
+SET @@global.key_buffer_size = 4294967295;
+
+# now, for a 64-bit value.
+# on 32-bit, we get "out of range", on 64-bit, "out of memory."
+--error 1210,5
+SET @@global.key_buffer_size = 9223372036854775807;
+
+# restore normal value, just in case we got the 4GB or something.
+SET @@global.key_buffer_size=@save_key_buffer;
+
+# End of 5.0 tests
diff -Nrup a/mysys/mf_keycache.c b/mysys/mf_keycache.c
--- a/mysys/mf_keycache.c	2007-10-18 12:32:38 +02:00
+++ b/mysys/mf_keycache.c	2008-03-14 08:26:28 +01:00
@@ -442,7 +442,7 @@ int init_key_cache(KEY_CACHE *keycache, 
         my_large_free(keycache->block_mem, MYF(0));
         keycache->block_mem= 0;
       }
-      if (blocks < 8)
+      if (blocks < 8 || keycache->block_mem == 0)
       {
         my_errno= ENOMEM;
         goto err;