List:Commits« Previous MessageNext Message »
From:paul Date:March 9 2006 10:25pm
Subject:svn commit - mysqldoc@docsrva: r1544 - in trunk: . mysqltest
View as plain text  
Author: paul
Date: 2006-03-09 22:25:11 +0100 (Thu, 09 Mar 2006)
New Revision: 1544

Log:
 r3817@kite-hub:  paul | 2006-03-09 15:23:32 -0600
 Initial import of Wiki material into mysqltest manual.
 Some markup added, more to come.


Modified:
   trunk/
   trunk/mysqltest/mysqltest.xml


Property changes on: trunk
___________________________________________________________________
Name: svk:merge
   - b5ec3a16-e900-0410-9ad2-d183a3acac99:/mysqldoc-local/mysqldoc/trunk:8509
bf112a9c-6c03-0410-a055-ad865cd57414:/mysqldoc-local/mysqldoc/trunk:3795
   + b5ec3a16-e900-0410-9ad2-d183a3acac99:/mysqldoc-local/mysqldoc/trunk:8509
bf112a9c-6c03-0410-a055-ad865cd57414:/mysqldoc-local/mysqldoc/trunk:3817

Modified: trunk/mysqltest/mysqltest.xml
===================================================================
--- trunk/mysqltest/mysqltest.xml	2006-03-09 18:20:46 UTC (rev 1543)
+++ trunk/mysqltest/mysqltest.xml	2006-03-09 21:25:11 UTC (rev 1544)
@@ -1530,7 +1530,7 @@
 
         <refsynopsisdiv>
           <cmdsynopsis>
-            <command>mysql-test-run
[<replaceable>options</replaceable>]
[<replaceable>test_case</replaceable>] ...</command>
+            <command>mysql-test-run
[<replaceable>options</replaceable>]
[<replaceable>test_name</replaceable>] ...</command>
           </cmdsynopsis>
         </refsynopsisdiv>
 
@@ -1561,15 +1561,15 @@
           </para>
 
 <programlisting>
-shell&gt; <userinput>mysql-test-run
[<replaceable>options</replaceable>]
[<replaceable>test_case</replaceable>] ...</userinput>
+shell&gt; <userinput>mysql-test-run
[<replaceable>options</replaceable>]
[<replaceable>test_name</replaceable>] ...</userinput>
 </programlisting>
 
           <para>
-            For each <replaceable>test_case</replaceable> argument,
+            For each <replaceable>test_name</replaceable> argument,
             <command>mysql-test-run</command> runs [Q: the test found in
             the
-           
<filename>t/<replaceable>test_case</replaceable>.test</filename>
-            file?] With no <replaceable>test_case</replaceable>
+           
<filename>t/<replaceable>test_name</replaceable>.test</filename>
+            file?] With no <replaceable>test_name</replaceable>
             arguments, <command>mysql-test-run</command> [Q: does what?
             Runs all <filename>.test</filename> files in the
             <filename>t</filename> subdirectory?]
@@ -5238,4 +5238,1018 @@
 
   </chapter>
 
+  <chapter id="mysqltest-wiki">
+
+    <title><command>mysqltest</command> Stuff from the
Wiki</title>
+
+    <section id="test-framework">
+
+      <title>The <command>mysqltest</command> Test
Framework</title>
+
+      <para>
+        The MySQL test framework is used primarily to run the MySQL
+        server regression suite. The test cases are for most part SQL
+        commands, mixed by some MySQL Test language constructs to
+        control how the test is to be run and how the result is to be
+        verified.
+      </para>
+
+      <para>
+        The application that run the test suite is named
+        <command>mysql-test-run</command> and is located in the
+        <filename>mysql-test</filename> directory of both source and
+        binary MySQL server distributions.
+      </para>
+
+      <para>
+        The test cases are located in the
+        <filename>mysql-test/t</filename> directory. Each test file is
+        usually refered to as one test, but usually consist of a
+        sequence of related tests. An unexpected failure of a single
+        statement in the test file, will make this test fail.
+      </para>
+
+      <para>
+        The <command>mysql-test-run</command> application will start
+        MySQL servers, restart it if a specific test case needs
+        different start arguments, and present the test result. You may
+        hear referenses to the mysqltest application, this is the
+        application that is called by <command>mysql-test-run</command>
+        to run a single test case, intepret the test language
+        constructs, and send SQL commands to the server.
+      </para>
+
+      <para>
+        There is two ways a test can fail. The test engine check the
+        result codes from executing each SQL command. If the failure is
+        unexpected, the test case will fail.
+      </para>
+
+      <para>
+        The other way it can fail, is by producing the incorrect
+        trace/protocol output. When running the test, it writes out the
+        result from SHOW and other commands to a file. This file is then
+        compared to the expected trace found in the
+        <filename>mysql-test/r</filename> directory. If it is different,
+        the test case will fail and the difference will be presented for
+        evaluation.
+      </para>
+
+      <para>
+        This way of comparing the result trace, puts some restrictions
+        on how a test case can be written. You can't have a result that
+        varies from run to run, like the use the current time. If the
+        data that varies is unimportant for the evaluation there are
+        ways to instruct the test engine to replace those fields in the
+        output with fixed data.
+      </para>
+
+      <para>
+        Because the test cases are written mostly in SQL in a text file,
+        there is no direct support to do test cases that are written in
+        C, Java or other languages. This is not the scope of this test
+        framework. But the test framework support executing your own
+        scripts, initiating with your own data and in a test case
+        execute any program so it may be used for some other testing
+        than just SQL statements.
+      </para>
+
+    </section>
+
+    <section id="mysqltest-tutorial">
+
+      <title><command>mysqltest</command> Tutorial</title>
+
+      <section>
+
+        <title>Writing a test case</title>
+
+        <para>
+          You write test cases in any text editor that use LF (linefeed)
+          as the end-of-line character. File names are in lower case
+          ASCII, with no spaces.
+        </para>
+
+        <para>
+          We are adding support for multiple <quote>suites.</quote>
+          Until then, all test cases are to be located in
+          <filename>mysql-test/t</filename>.
+        </para>
+
+        <para>
+          One test file can be a collection of tests that you think
+          belongs together. But this also means that if one of the tests
+          fail, the whole test case will fail. While this makes it
+          tempting to write each small test into a single file, this
+          will be to inefficient and make the test runs unbearably slow.
+          So make the test files not too big, not too small.
+        </para>
+
+        <para>
+          To make us talk about the same thing, let us define
+        </para>
+
+        <itemizedlist>
+
+          <listitem>
+            <para>
+              One <quote>test file</quote> is one <quote>test
+              case.</quote>
+            </para>
+          </listitem>
+
+          <listitem>
+            <para>
+              One <quote>test case</quote> might contain a <quote>test
+              sequence</quote> (that is, a number of individual tests
+              that are grouped together in the same test file).
+            </para>
+          </listitem>
+
+        </itemizedlist>
+
+        <para>
+          Also, a test case (i.e. a test file) is to be self contained.
+          This means you are not allowed to create and/or fill a table
+          in one test case and depend on this in a later test case. If
+          you have some common initiation to be done, you put this into
+          a file in the <filename>mysq-test/include</filename>
+          directory, and in each test case that needs it, you do a
+          <literal>--source include/xxxxx.inc</literal>. We take this by
+          example. Lets say you add a test case called
+          <literal>foo</literal>. You then create a file
+          <filename>mysql-test/t/foo.test</filename>.
+        </para>
+
+        <para>
+          A comment in a test case can be started with the '#'
+          character, or the "--" characters. Note that the first word
+          after the "--" might be a command to the test engine itself,
+          More about that later.
+        </para>
+
+        <para>
+          It is safest to use the '#' character for comments, not to
+          accidently run a mysq-test directive/command. For example
+          writing "-- End or test 43" will give an error message because
+          "end" is something that <command>mysqltest</command> thinks is
+          a directive/command.
+        </para>
+
+        <para>
+          Note that because all our test cases will be published to all
+          Internet, be careful that your comments inside the test cases
+          contain no confidential information.
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>Naming rules for tables, views etc</title>
+
+        <para>
+          The test cases might be used to run against a production
+          server (we will generally not do that, but our customers
+          might). To reduce the risk that by running the test suite,
+          important tables are altered or destroyed, you are to name
+          your tables, views and other objects like
+        </para>
+
+<programlisting>
+table names : t1, t2, t3.....
+view names  : v1, v2, v3.....
+</programlisting>
+
+        <para>
+          Look in the other test cases for examples of this. Of course
+          you can name field names and other objects inside tables as
+          you wish. If you don't have a special reason to, you are to
+          use the default database already created for you 'test'.
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>Cleaning up from an old run</title>
+
+        <para>
+          For efficiency our test engine will not start with a clean new
+          database for running each test case. So a test case generally
+          starts with a "cleaning up section". Let us assume we are to
+          use two tables in our new test case <literal>foo</literal>.
+          You then begin your test file with
+        </para>
+
+<programlisting>
+#
+# Test of XXXXX
+#
+--disable_warnings
+drop table if exists t1,t2;
+--enable_warnings
+</programlisting>
+
+        <para>
+          The "disable_warnings" command instructs the test engine not
+          to log any warnings until "enable_warnings" or the test case
+          is ended. For some unknown reason MySQL choose to write a
+          warning if the table 't1' or 't2' exists. Surrounding this
+          part with the disable/enable warnings makes the test case
+          output the same if the tables exists or not before the test is
+          started. Now we are free to put in any SQL commands that
+          create, use and drop the tables 't1' and 't2'. The rule is
+          that you are to have a corresponding drop of each table you
+          create in the test.
+        </para>
+
+        <para>
+          So lets put in some SQL code into this test case
+        </para>
+
+<programlisting>
+create table t1 (
+  Period smallint(4) unsigned zerofill default '0000' not null,
+  Varor_period smallint(4) unsigned default '0' not null
+);
+
+insert into t1 values (9410,9412);
+
+select period from t1;
+select * from t1;
+select t1.* from t1;
+
+drop table t1;
+</programlisting>
+
+      </section>
+
+      <section>
+
+        <title>Determining whether a test fails</title>
+
+        <para>
+          Note that the test code we just wrote contains no checks of
+          the result. The test will report a failure for one of two
+          reasons
+        </para>
+
+        <itemizedlist>
+
+          <listitem>
+            <para>
+              A SQL statement failed with an error
+            </para>
+          </listitem>
+
+          <listitem>
+            <para>
+              The result is not what was expected
+            </para>
+          </listitem>
+
+        </itemizedlist>
+
+        <para>
+          So we are to specify the expected result somehow. You do this
+          by running the test with the flag "--record", like
+        </para>
+
+<programlisting>
+% cd mysql-test
+% ./mysql-test-run --record foo
+</programlisting>
+
+        <para>
+          [Q: Is the <quote>protocol</quote> file the result file or the
+          reject file? I have seen the term applied to both.]
+        </para>
+
+        <para>
+          This will create a file
+          <filename>mysql-test/r/foo.result</filename>, also called the
+          <quote>protocol file,</quote> with the content
+        </para>
+
+<programlisting>
+drop table if exists t1,t2;
+create table t1 (
+Period smallint(4) unsigned zerofill default '0000' not null,
+Varor_period smallint(4) unsigned default '0' not null
+);
+insert into t1 values (9410,9412);
+select period from t1;
+period
+9410
+select * from t1;
+Period  Varor_period
+9410    9412
+select t1.* from t1;
+Period  Varor_period
+9410    9412
+drop table t1;
+</programlisting>
+
+        <para>
+          If we look at this file, it contains the commands in the
+          <filename>foo.test</filename> file together with the output
+          from the select commands. The output is the table headings and
+          fields separated by the TAB character. So now you are to
+          inspect this file, decide if the result is the expected one,
+          and let it be part of your test case.
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>Running test cases</title>
+
+        <para>
+          You will normally either run the test case from within a
+          source tree, or on an installed server. In source tree you
+          will be located in the <filename>mysql-test</filename>
+          directory. The installed server also have a
+          <filename>mysql-test</filename> directory. The program running
+          the test suite, <command>mysql-test-run</command>, will figure
+          out if you are in a source tree, or in an installed directory
+          tree.
+        </para>
+
+        <para>
+          You will not run against a started server. Instead the
+          <command>mysql-test-run</command> program will start the
+          server or servers needed on ports that doesn't conflicts with
+          a production server, if you happen to have one on the same
+          machine.
+        </para>
+
+        <para>
+          You can run our newly created test case <literal>foo</literal>
+          as part of the whole suite, or run it by itself like
+        </para>
+
+<programlisting>
+% ./mysql-test-run foo
+</programlisting>
+
+        <para>
+          If you have a family of test cases, prefixed with something
+          common, you can use
+        </para>
+
+<programlisting>
+% ./mysql-test-run --do-tests=mytest
+</programlisting>
+
+      </section>
+
+      <section>
+
+        <title>Mutilple users on the same machine, or shared
directories</title>
+
+        <para>
+          If you are multiple users that run test on the same machine,
+          you need to specify the ports to the
+          <command>mysql-test-run</command> program, so that your run
+          doesn't conflict with others. You add unique port arguments to
+          <command>mysql-test-run</command>, such as
+          <option>--no-manager --master_port=3911
+          --slave_port=3927</option>.
+        </para>
+
+        <para>
+          Also note that only one person can run the
+          <command>mysql-test-run</command> program in the same
+          <filename>mysql-test</filename> directory on a shared drive,
+          at the same time. The <filename>mysql-test/var</filename>
+          directory created can't be shared between mutiple users.
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>How to get more information to check the result
against</title>
+
+        <para>
+          The test engine by default only output things from "select",
+          "show" and other SQL commands you expect to produce output,
+          i.e. commands that create a data set. But it can be instructed
+          to be more or less verbose, including other data as well.
+        </para>
+
+        <para>
+          Lets say we are a bit more interested how many rows are
+          effected by an operation, we add "--enable_info" to the test
+          case before we create the table. Now, after we rerun the test
+          with "--record", we get some more information in our result
+          file
+        </para>
+
+<programlisting>
+drop table if exists t1,t2;
+create table t1 (
+Period smallint(4) unsigned zerofill default '0000' not null,
+Varor_period smallint(4) unsigned default '0' not null
+);
+affected rows: 0
+insert into t1 values (9410,9412);
+affected rows: 1
+select period from t1;
+period
+9410
+affected rows: 1
+select * from t1;
+Period  Varor_period
+9410    9412
+affected rows: 1
+select t1.* from t1;
+Period  Varor_period
+9410    9412
+affected rows: 1
+drop table t1;
+</programlisting>
+
+        <para>
+          Options can in general be enabled and disabled for different
+          parts of the test file. Lets say we are interested in the
+          internals of the database as well. We can then enable the
+          output of "meta data" using "--enable_metadata". The output
+          now begins to be a bit verbose, but as said earlier, this can
+          be enabled and disabled just for a part of the test where it
+          interests you to know more.
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>How to get less information</title>
+
+        <para>
+          You might be initializing a table, where we don't really
+          expect a failure. You can then temporarily disable the
+          recording of the command lines executed with
+          "--disable_query_log" and then enable the recording again with
+          "--enable_query_log". You can disable the recording of the
+          output from executing commands using "--disable_result_log"
+          and enable again with "--enable_result_log".
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>What if the output varies?</title>
+
+        <para>
+          In general you are to avoid having tests that give different
+          output for each run, or for differences in how the mysql
+          binaries are created, operating system etc. The first example
+          that comes to mind is the use of the current date and time.
+          The test engine have no way of verifying that it is correct.
+        </para>
+
+        <para>
+          Now, you might have a test that have a column that will vary,
+          but you are not really interested in what is in that column
+          anyway. The test engine let you specify that you want to
+          replace whatever is in that column with a string. This makes
+          the output from our run the same each time.
+        </para>
+
+        <para>
+          Just to illustrate the functionality, we add after our first
+          insert, the row
+        </para>
+
+<programlisting>
+insert into t1 values (date_format(now(), '%s'),9999);
+</programlisting>
+
+        <para>
+          We now record and run the test case
+        </para>
+
+<programlisting>
+% ./mysql-test-run --record foo
+% ./mysql-test-run foo
+</programlisting>
+
+        <para>
+          Now we will, most likely, get a failure and the test engine
+          will display the difference between what we got and the
+          expected result, like
+        </para>
+
+<programlisting>
+Below are the diffs between actual and expected results:
+-------------------------------------------------------
+*** r/foo.result        Thu Jan 20 18:38:37 2005
+--- r/foo.reject        Thu Jan 20 18:39:00 2005
+***************
+*** 11,25 ****
+  select period from t1;
+  period
+  9410
+! 0037
+  affected rows: 2
+  select * from t1;
+  Period        Varor_period
+  9410  9412
+! 0037  9999
+  affected rows: 2
+  select t1.* from t1;
+  Period        Varor_period
+  9410  9412
+! 0037  9999
+  affected rows: 2
+--- 11,25 ----
+  select period from t1;
+  period
+  9410
+! 0059
+  affected rows: 2
+  select * from t1;
+  Period        Varor_period
+  9410  9412
+! 0059  9999
+  affected rows: 2
+  select t1.* from t1;
+  Period        Varor_period
+  9410  9412
+! 0059  9999
+  affected rows: 2
+-------------------------------------------------------
+</programlisting>
+
+        <para>
+          If we are not really interested in the first column, one way
+          to solve this by using the directive/command
+          "--replace_column". The duration of the effect of this
+          directive/command, is the next SQL statement, so we need one
+          before each select call, like
+        </para>
+
+<programlisting>
+--replace_column 1 SECONDS
+select period from t1;
+--replace_column 1 SECONDS
+select * from t1;
+--replace_column 1 SECONDS
+select t1.* from t1;
+</programlisting>
+
+        <para>
+          Note that "SECONDS" could be any string, it is just to display
+          something. If we now record again and run the test, we will
+          succeed each time. The result file now looks like
+        </para>
+
+<programlisting>
+drop table if exists t1,t2;
+create table t1 (
+Period smallint(4) unsigned zerofill default '0000' not null,
+Varor_period smallint(4) unsigned default '0' not null
+);
+affected rows: 0
+insert into t1 values (9410,9412);
+affected rows: 1
+insert into t1 values (date_format(now(), '%s'),9999);
+affected rows: 1
+select period from t1;
+period
+SECONDS
+SECONDS
+affected rows: 2
+select * from t1;
+Period  Varor_period
+SECONDS 9412
+SECONDS 9999
+affected rows: 2
+select t1.* from t1;
+Period  Varor_period
+SECONDS 9412
+SECONDS 9999
+affected rows: 2
+</programlisting>
+
+        <para>
+          If the result may vary, but just between some know values, you
+          can use "--replace_result" to match the different result, and
+          replace them with a common string, making the result file the
+          same even if the result varies some.
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>Expected failures</title>
+
+        <para>
+          A good test suite not only test when things go right, but also
+          that we get the correct error message, when it goes wrong. The
+          test engine let you specify "expected failures". Lets say we
+          try to create 't1' again without dropping it in between, like
+        </para>
+
+<programlisting>
+--disable_warnings
+drop table if exists t1,t2;
+--enable_warnings
+
+create table t1 (
+  Period smallint(4) unsigned zerofill default '0000' not null,
+  Varor_period smallint(4) unsigned default '0' not null
+);
+
+insert into t1 values (9410,9412);
+
+select period from t1;
+select * from t1;
+select t1.* from t1;
+
+create table t1 (something smallint(4));
+</programlisting>
+
+        <para>
+          This gives a failure and an error
+        </para>
+
+<programlisting>
+At line 19: query 'create table t1 (something smallint(4))'
+    failed: 1050: Table 't1' already exists
+</programlisting>
+
+        <para>
+          We put in a "--error 1050" just before the second "create
+          table", to mark that this is an expected failure, and that we
+          expected it to return the MySQL error code 1050. If we look at
+          the end of the result file we now have
+        </para>
+
+<programlisting>
+create table t1 (something smallint(4));
+ERROR 42S01: Table 't1' already exists
+</programlisting>
+
+        <para>
+          So we still can match on the expected error, but it will not
+          terminate the test case run because of that error. The "42S01"
+          is the SQLSTATE code. Alternatively you might match on that
+          error code, by using the 'S' prefix, like "--error S42S01".
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>Setting server options</title>
+
+        <para>
+          Many server options can be set from within the test case, like
+        </para>
+
+<programlisting>
+set sql_warnings=1;
+set sql_mode=no_auto_value_on_zero;
+</programlisting>
+
+        <para>
+          But sometimes you need to restart the server to use some
+          options in the form of extra command line arguments. The test
+          engine will restart the server just before running a test
+          case, if there exists a file
+         
<filename>mysql-test/t/<replaceable>test_name</replaceable>-master.opt</filename>
+          with the options you want to use. After the test case is
+          completed, the server will be restarted again, with the
+          default arguments.
+        </para>
+
+      </section>
+
+    </section>
+
+    <section id="mysqltest-test-cases">
+
+      <title>More on Writing Test Cases</title>
+
+      <section>
+
+        <title>Running Test Case Overview</title>
+
+        <para>
+          This is a brief description of how to write test cases and the
+          language used to write them. Normally you run the test suite,
+          when developing.
+        </para>
+
+        <para>
+          For example:
+        </para>
+
+<programlisting>
+% cd mysql-test
+% mysql-test-run
+</programlisting>
+
+        <para>
+          Or:
+        </para>
+
+<programlisting>
+% mysql-test-run --local --force
+</programlisting>
+
+        <para>
+          Use --force if you want mysql-test to continue on failure of a
+          test case
+        </para>
+
+        <para>
+          For a single test case or group or test cases(the .test
+          extension exluded)
+        </para>
+
+<programlisting>
+% mysql-test-run --local
--do-test=<replaceable>test_name_prefix</replaceable>
+</programlisting>
+
+        <para>
+          This can be a single test case or a many tests with the same
+          prefix.
+        </para>
+
+        <para>
+          Examples:
+        </para>
+
+<programlisting>
+% mysql-test-run --local --do-test=rpl_abcd
+</programlisting>
+
+        <para>
+          This will run the rpl_abcd.test
+        </para>
+
+<programlisting>
+% mysql-test-run --local --do-test=rpl
+</programlisting>
+
+        <para>
+          This will run all the tests that start with rpl.
+        </para>
+
+        <para>
+          Please view the README file in the mysql-test directory for
+          more details about how to invoke the test script
+          <command>mysql-test-run</command>. The
+          <command>mysql-test-run</command> is setting up the
+          environment for calling the program mysqltest and start a
+          server etc. mysqltest is the program that will read the test
+          case language text and start managers, create connections,
+          send queries to a server etc.
+        </para>
+
+        <para>
+          The test case language is a mix of commands that the
+          <command>mysqltest</command> program understand and SQL. What
+          it doesn't understand is assumed to be SQL queries and are
+          sent to the database server. This makes the test case language
+          familiar to those that know how to write SQL and powerful
+          enough to add the control needed to write test cases.
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>Writing Test Cases Basics</title>
+
+        <para>
+          The base principle is that the test script compares the output
+          from running the tests with the expected result. This is just
+          a diff between the output and a file the test writer provide.
+          With this simplistic way of comparing you can't handle
+          variations in the output from running the script but the test
+          language provide ways to suppress unwanted output making this
+          usable. The steps to write a new test case are
+        </para>
+
+        <orderedlist>
+
+          <listitem>
+            <para>
+              Change directories to the test directory
+             
<filename>mysql-<replaceable>version</replaceable>/mysql-test</filename>
+            </para>
+
+<programlisting>
+% cd mysql-<replaceable>version</replaceable>/mysql-test
+</programlisting>
+          </listitem>
+
+          <listitem>
+            <para>
+              Create the test in a file
+              "t/<replaceable>test_name</replaceable>.test"
+            </para>
+          </listitem>
+
+          <listitem>
+            <para>
+              Create an empty result file
+            </para>
+
+<programlisting>
+% touch r/<replaceable>test_name</replaceable>.result
+</programlisting>
+          </listitem>
+
+          <listitem>
+            <para>
+              Run the test
+            </para>
+
+<programlisting>
+% ./mysql-test-run <replaceable>test_name</replaceable>
+</programlisting>
+          </listitem>
+
+          <listitem>
+            <para>
+              The failing test will create a file
+              "r/<replaceable>test_name</replaceable>.reject". Examine
+              this file and if it looks ok you copy its content to the
+              result file
+            </para>
+
+<programlisting>
+% cp r/<replaceable>test_name</replaceable>.reject
r/<replaceable>test_name</replaceable>.result
+</programlisting>
+          </listitem>
+
+          <listitem>
+            <para>
+              Run the test again, this time it should succeed
+            </para>
+
+<programlisting>
+% ./mysql-test-run <replaceable>test_name</replaceable>
+</programlisting>
+          </listitem>
+
+        </orderedlist>
+
+        <para>
+          The environment can also be set up so the
+          <command>mysqltest</command> program can be called directly.
+          For more information about options to the
+          <command>mysqltest</command> program run
+        </para>
+
+<programlisting>
+% mysqltest --help
+</programlisting>
+
+        <para>
+          There are also some notes about <command>mysqltest</command>
+          in the README in this directory.
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>Sample Test Case</title>
+
+        <para>
+          Small sample test case
+        </para>
+
+<programlisting>
+--disable_warnings
+DROP TABLE IF EXISTS t1;
+--enable_warnings
+SET SQL_WARNINGS=1;
+
+CREATE TABLE t1 (a INT);
+INSERT INTO t1 VALUES (1);
+INSERT INTO t1 VALUES ("hej");
+</programlisting>
+
+        <para>
+          The first rows try to cleanup from possibly earlier runs of
+          the script by dropping the table "t1". We disable storing
+          warnings (--disable_warnings) in the output during this
+          because we are not interested to know if the table "t1" was
+          there or not. Then we enable storing warnings again
+          (--enable_warnings) and enable verbose warnings in MySQL using
+          the SET SQL_WARNINGS=1;" line.
+        </para>
+
+        <para>
+          We create the table "t1" and try some operations. The creation
+          of the table and first insertion will not create any warnings
+          but the last line will. We collect the output from running
+          this test and add this to the result file
+        </para>
+
+<programlisting>
+DROP TABLE IF EXISTS t1;
+SET SQL_WARNINGS=1;
+CREATE TABLE t1 (a INT);
+INSERT INTO t1 VALUES (1);
+INSERT INTO t1 VALUES ("hej");
+Warnings:
+Warning 1265    Data truncated for column 'a' at row 1
+</programlisting>
+
+        <para>
+          Note that we see the commands as well in the result. This can
+          be disabled with the "--disable_query_log" test language
+          command. There are several options how to control the amount
+          of output from running the tests. If there was an failure this
+          will be reported to the screen. You can see the actual output
+          from the last unsuccessful run of the testcase in the test
+          case reject file
+          "r/<replaceable>test_name</replaceable>.reject".
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>Test Environment Overview</title>
+
+        <para>
+          There are two ways the same test environment should handle.
+          One is to talk to an installed running server. The second case
+          is to be able to start a server from the binaries in the
+          source tree, i.e. an uninstalled MySQL server. Because of this
+          in general you don't start a server and create a connection
+          from the test case, this is done by the
+          <command>mysql-test-run</command> script before your test case
+          is run.
+        </para>
+
+        <para>
+          FIXME add the most important options to use calling
+          <command>mysql-test-run</command>
+        </para>
+
+      </section>
+
+      <section>
+
+        <title>Language Overview</title>
+
+        <para>
+          FIXME make descriptions shorter by introducing a syntax to
+          inform that an argument could be a test language variable. Why
+          not all?
+        </para>
+
+        <para>
+          FIXME some environment variables are passed from the
+          environment to test language variables, list them. Why not
+          all??
+        </para>
+
+        <para>
+          The following is a description of the language used in
+          .test-files.
+        </para>
+
+        <para>
+          The <command>mysqltest</command> program is a small
+          interpreter that interpret a special test language. Lines that
+          it doesn't recognize are assumed to be SQL queries that can be
+          sent to the MySQL server. In effect this makes the test files
+          look like SQL with some small extensions.
+        </para>
+
+        <para>
+          Lines starting with a "#" in the test files are comments.
+        </para>
+
+        <para>
+          Commands to the test engine itself are often put into
+          comments, like --cmd. This sometimes leads to conflicts with
+          comments that start with the same word as a command.
+        </para>
+
+        <para>
+          FIXME what is @filename ?!
+        </para>
+
+      </section>
+
+    </section>
+
+  </chapter>
+
 </book>

Thread
svn commit - mysqldoc@docsrva: r1544 - in trunk: . mysqltestpaul9 Mar