List:Commits« Previous MessageNext Message »
From:Guilhem Bichot Date:September 20 2010 8:12am
Subject:bzr push into mysql-next-mr-bugfixing branch (guilhem:3203 to 3204)
View as plain text  
 3204 Guilhem Bichot	2010-09-20
      a Python script to validate that the traces found in a result file are JSON-compliant
      (currently they are not because " and newlines in queries are not escaped)

    added:
      WL4800_validate_json.py
    modified:
      WL4800_Doxyfile
      WL4800_extract_plans.py
      WL4800_select_to_explain_test.py
      sql/opt_trace.cc
      sql/opt_trace.h
      unittest/gunit/opt_trace-t.cc
 3203 Guilhem Bichot	2010-09-18
      1) implementing the main remaining pieces of WL#5257:
      1.1) @@optimizer_trace_offset, @@optimizer_trace_limit.
      @@optimizer_trace_offset is a signed variable, so I had to add support for
      this in MySQL (so far all integer system variables were unsigned, it was impossible
      to set a system variable to a negative value, it would be rounded to 0).
      1.2) using stored routines implies that one statement has sub-statements; so far
      all of the statement's and sub-statements' traces were concatenated in one
      single row of INFORMATION_SCHEMA.OPTIMIZER_TRACE. This prevented
      @@optimizer_trace_offset/limit to function. Fixed by making each statement and
      sub-statement generate a new row in INFORMATION_SCHEMA.OPTIMIZER_TRACE.
      1.3) @@optimizer_trace_max_mem_size. If allocating memory for extending a trace
      fails because of optimizer_trace_max_mem_size, report missing bytes in
      INFORMATION_SCHEMA.OPTIMIZER_TRACE.MISSING_BYTES_BEYOND_MAX_MEM_SIZE.
      If OS' malloc()/realloc() fail, report failure in
      INFORMATION_SCHEMA.OPTIMIZER_TRACE.OS_MALLOC_ERROR.
      1.4) mtr tests of the new variables
      2) test of optimizer_trace vs {query with a stored function, or a stored procedure,
      or a trigger}.
      3) test of optimizer_trace vs {a view, or an INFORMATION_SCHEMA table}.
      4) test that first and second EXECUTE of a prepared statement give the same trace.
     @ include/my_getopt.h
        utility function
     @ include/mysql/plugin.h
        comments
     @ include/mysql/plugin_audit.h.pp
        layout of space changed in .h, so changes in .h.pp
     @ include/mysql/plugin_ftparser.h.pp
        layout of space changed in .h, so changes in .h.pp
     @ mysql-test/include/have_optimizer_trace.inc
        optimizer_trace tests must not fail if binary doesn't support it
     @ mysql-test/include/optimizer_trace.inc
        new tests
     @ mysql-test/lib/My/SafeProcess/safe_process.cc
        temporary fix (should not be pushed to main trees);
        without it, some valgrind "at shutdown" errors (memory leaks)
        are missed because kill-9 prevents valgrind from finding/printing them
     @ mysql-test/r/bug42620.result
        update result due to new columns in I_S.OPTIMIZER_TRACE
     @ mysql-test/r/mysqld--help-notwin.result
        update result due to new variables
     @ mysql-test/r/optimizer_trace_ps_prot.result
        result is identical to no_prot, with the exception of 3 subquery
        transformations as explained at the start of optimizer_trace.inc
     @ mysql-test/suite/sys_vars/t/optimizer_trace_limit_basic.test
        basic test of reading/writing the optimizer_trace_limit variable
     @ mysql-test/suite/sys_vars/t/optimizer_trace_max_mem_size_basic.test
        basic test of reading/writing the optimizer_trace_max_mem_size variable
     @ mysql-test/suite/sys_vars/t/optimizer_trace_offset_basic.test
        basic test of reading/writing the optimizer_trace_offset variable
     @ mysql-test/suite/sys_vars/t/optimizer_trace_offset_max-master.opt
        this should cap "SET optimizer-trace-offset" at values between -50 and 50
     @ mysql-test/suite/sys_vars/t/optimizer_trace_offset_max.test
        test that --maximum-optimizer-trace-offset caps values as intended
     @ mysys/my_getopt.c
        reducing code size by using a utility function, max_of_int_range()
     @ sql/CMakeLists.txt
        to-be-unit-tested object files should be linked into sqlgunitlib
     @ sql/item_func.cc
        support for SHOW_SIGNED_LONG. The "unsigned_flag" is what makes
        -1 be displaid as -1 and not a huge positive number in "SELECT @@var"
     @ sql/item_subselect.cc
        unused variable
     @ sql/mysqld.cc
        unimportant change (get_tail() is now safe to call even
        if trace is not started)
     @ sql/opt_range.cc
        more disabling of range optimizer's tracing. Fixing compiler warnings.
        Removing unneeded todo comments.
     @ sql/opt_trace.cc
        see comments in opt_trace.h
     @ sql/set_var.cc
        see explanation in commit comment of sql/sys_vars.h
     @ sql/sql_class.h
        new variables optimizer_trace_offset, optimizer_trace_limit,
        optimizer_trace_max_mem_size.
     @ sql/sql_plugin.h
        support for SHOWing _signed_ long integer variables (in SHOW VARIABLES,
        in SELECT @@thevar): request this by using SHOW_SIGNED_LONG.
     @ sql/sql_select.cc
        removing unneeded todo comments
     @ sql/sql_show.cc
        the first change is not needed today (there are no SHOW_SIGNED_LONG variables
        in SHOW STATUS) but good for the future.
        The second change is what makes -1 be displaid as -1 and not a huge positive number
        in "SHOW VARIABLES"
     @ sql/sys_vars.cc
        new variables optimizer_trace_offset, optimizer_trace_limit,
        optimizer_trace_max_mem_size.
     @ sql/sys_vars.h
        Support for _signed_ integer system variables. This makes do_check()
        more complicated, as code cannot be the same as for unsigned.
        The third parameter of throw_bounds_warning(), named 'fixed', was
        here 'var->save_result.ulonglong_value != uv' which tests whether
        there was rounding due to the allowed range defined with VALID_RANGE()
        (see sys_vars.cc); that rounding is done by getopt_ull_limit_value().
        But there is another 'fixing', which is that a negative value is rounded
        to 0; this is done earlier with
          if (var->value->unsigned_flag)
            ...
          else
            uv= (ulonglong) (v < 0 ? 0 : v);
        The passed value of 'fixed' didn't reflect this rounding, so
        throw_bounds_warning() had to explicitely detect it with the
        part after || :
          if (fixed || (!is_unsigned && v < 0))
        By changing the parameter to be
          var->save_result.ulonglong_value != (ulonglong)v
        which detects whether any change happened between input value
        (v) and final value (var->save_result.ulonglong_value) we can
        just test 'fixed' in throw_bounds_warning().
     @ unittest/gunit/CMakeLists.txt
        build opt_trace unit test
     @ unittest/gunit/Makefile.am
        build opt_trace unit test
     @ unittest/gunit/opt_trace-t.cc
        sql/ unit tests live should be put in unittest/gunit nowadays

    added:
      mysql-test/include/have_optimizer_trace.inc
      mysql-test/suite/sys_vars/r/optimizer_trace_limit_basic.result
      mysql-test/suite/sys_vars/r/optimizer_trace_max_mem_size_basic.result
      mysql-test/suite/sys_vars/r/optimizer_trace_offset_basic.result
      mysql-test/suite/sys_vars/r/optimizer_trace_offset_max.result
      mysql-test/suite/sys_vars/t/optimizer_trace_limit_basic.test
      mysql-test/suite/sys_vars/t/optimizer_trace_max_mem_size_basic.test
      mysql-test/suite/sys_vars/t/optimizer_trace_offset_basic.test
      mysql-test/suite/sys_vars/t/optimizer_trace_offset_max-master.opt
      mysql-test/suite/sys_vars/t/optimizer_trace_offset_max.test
    renamed:
      sql/opt_trace-t.cc => unittest/gunit/opt_trace-t.cc
    modified:
      WL4800_TODO.txt
      include/my_getopt.h
      include/mysql/plugin.h
      include/mysql/plugin_audit.h.pp
      include/mysql/plugin_ftparser.h.pp
      mysql-test/include/optimizer_trace.inc
      mysql-test/lib/My/SafeProcess/safe_process.cc
      mysql-test/r/bug42620.result
      mysql-test/r/mysqld--help-notwin.result
      mysql-test/r/optimizer_trace_no_prot.result
      mysql-test/r/optimizer_trace_ps_prot.result
      mysql-test/suite/sys_vars/t/optimizer_trace_basic.test
      mysql-test/t/bug42620.test
      mysql-test/t/optimizer_trace_bugs.test
      mysys/my_getopt.c
      sql/CMakeLists.txt
      sql/item_func.cc
      sql/item_subselect.cc
      sql/mysqld.cc
      sql/opt_range.cc
      sql/opt_trace.cc
      sql/opt_trace.h
      sql/set_var.cc
      sql/sql_class.h
      sql/sql_plugin.h
      sql/sql_select.cc
      sql/sql_show.cc
      sql/sys_vars.cc
      sql/sys_vars.h
      unittest/gunit/CMakeLists.txt
      unittest/gunit/Makefile.am
      unittest/gunit/opt_trace-t.cc
=== modified file 'WL4800_Doxyfile'
--- a/WL4800_Doxyfile	2010-09-05 16:08:07 +0000
+++ b/WL4800_Doxyfile	2010-09-20 08:12:18 +0000
@@ -1,5 +1,20 @@
 # Doxyfile 1.6.1
 
+# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation; version 2 of the License.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software
+#  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA */
+
 # This file describes the settings to be used by the documentation system
 # doxygen (www.doxygen.org) for a project
 #

=== modified file 'WL4800_extract_plans.py'
--- a/WL4800_extract_plans.py	2010-05-24 18:38:10 +0000
+++ b/WL4800_extract_plans.py	2010-09-20 08:12:18 +0000
@@ -1,5 +1,20 @@
 #! /usr/bin/python
 
+# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation; version 2 of the License.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software
+#  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA */
+
 import sys
 
 usage = """

=== modified file 'WL4800_select_to_explain_test.py'
--- a/WL4800_select_to_explain_test.py	2010-09-04 17:40:51 +0000
+++ b/WL4800_select_to_explain_test.py	2010-09-20 08:12:18 +0000
@@ -1,5 +1,20 @@
 #! /usr/bin/python
 
+# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation; version 2 of the License.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software
+#  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA */
+
 import sys, re
 
 usage = """

=== added file 'WL4800_validate_json.py'
--- a/WL4800_validate_json.py	1970-01-01 00:00:00 +0000
+++ b/WL4800_validate_json.py	2010-09-20 08:12:18 +0000
@@ -0,0 +1,65 @@
+#! /usr/bin/python
+
+# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation; version 2 of the License.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software
+#  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA */
+
+import sys
+
+usage = """
+Usage:
+  %s <a_file>
+
+It will verify that all optimizer traces of a_file (usually a_file
+is a .result or .reject file which contains
+SELECT * FROM OPTIMIZER_TRACE; ) are JSON-compliant
+""" % sys.argv[0]
+
+if len(sys.argv) != 2:
+    print usage
+    sys.exit(1)
+
+import json, re
+
+def check(trace, first_trace_line):
+    s = "".join(trace)
+    try:
+        json.loads(s)
+    except:
+        print "error at line", first_trace_line
+        print sys.exc_info()
+        print s
+    else:
+        print "ok at line", first_trace_line
+    print
+
+all = open(sys.argv[1]).readlines()
+
+first_trace_line = trace_line = 0
+trace = None
+for l in all:
+    trace_line += 1
+    if l == "{\n":
+        assert first_trace_line == 0
+        trace = []
+        first_trace_line = trace_line
+    if l.startswith("}\t") or l.startswith("}\n"): # end of current trace
+        assert first_trace_line != 0
+        trace.append("}") # eliminate any following columns of table (OS_MALLOC_ERROR etc)
+        check(trace, first_trace_line)
+        first_trace_line = 0
+    if first_trace_line != 0:
+        # eliminate /* */ (not valid JSON)
+        no_comment = re.sub("/\*.*\*/", "", l)
+        trace.append(no_comment)

=== modified file 'sql/opt_trace.cc'
--- a/sql/opt_trace.cc	2010-09-18 16:25:43 +0000
+++ b/sql/opt_trace.cc	2010-09-20 08:12:18 +0000
@@ -1,4 +1,4 @@
-/* Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2009, 2010, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by

=== modified file 'sql/opt_trace.h'
--- a/sql/opt_trace.h	2010-09-18 16:25:43 +0000
+++ b/sql/opt_trace.h	2010-09-20 08:12:18 +0000
@@ -1,4 +1,4 @@
-/* Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2009, 2010, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by

=== modified file 'unittest/gunit/opt_trace-t.cc'
--- a/unittest/gunit/opt_trace-t.cc	2010-09-18 16:25:43 +0000
+++ b/unittest/gunit/opt_trace-t.cc	2010-09-20 08:12:18 +0000
@@ -1,4 +1,4 @@
-/* Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by


Attachment: [text/bzr-bundle] bzr/guilhem@mysql.com-20100920081218-3r0d16cbtwuup7hp.bundle
Thread
bzr push into mysql-next-mr-bugfixing branch (guilhem:3203 to 3204) Guilhem Bichot20 Sep