Author: paul
Date: 2006-03-14 22:55:52 +0100 (Tue, 14 Mar 2006)
New Revision: 1582
Log:
r3907@kite-hub: paul | 2006-03-14 15:55:30 -0600
General revisions.
Modified:
trunk/
trunk/mysqltest/command-reference.xml
trunk/mysqltest/mysqltest.xml
trunk/mysqltest/tutorial.xml
Property changes on: trunk
___________________________________________________________________
Name: svk:merge
- b5ec3a16-e900-0410-9ad2-d183a3acac99:/mysqldoc-local/mysqldoc/trunk:8644
bf112a9c-6c03-0410-a055-ad865cd57414:/mysqldoc-local/mysqldoc/trunk:3899
+ b5ec3a16-e900-0410-9ad2-d183a3acac99:/mysqldoc-local/mysqldoc/trunk:8644
bf112a9c-6c03-0410-a055-ad865cd57414:/mysqldoc-local/mysqldoc/trunk:3907
Modified: trunk/mysqltest/command-reference.xml
===================================================================
--- trunk/mysqltest/command-reference.xml 2006-03-14 21:49:31 UTC (rev 1581)
+++ trunk/mysqltest/command-reference.xml 2006-03-14 21:55:52 UTC (rev 1582)
@@ -792,12 +792,14 @@
<para>
End an <literal>if</literal> or <literal>while</literal>
block. If there is no such block open,
- <command>mysqltest</command> exits with an error.
+ <command>mysqltest</command> exits with an error. See
+ <xref linkend="mysqltest-flow-control"/>, for information on
+ flow-control constructs.
</para>
<para>
- [<command>mysqltest</command> appears to consider
- <literal>}</literal> and <literal>end</literal> the same]
+ [Q: <command>mysqltest</command> appears to consider
+ <literal>}</literal> and <literal>end</literal> the same?]
</para>
</listitem>
@@ -851,24 +853,28 @@
</para>
<para>
- [Wiki has some stuff about non-obvious effects for specifying
- error 0 in combination with non-zero error codes]
+ [TODO: Wiki has some stuff about non-obvious effects for
+ specifying error 0 in combination with non-zero error codes.
+ Need to add that here.]
</para>
<para>
- [Q: 0 and S00000 are equivalent for specifying <quote>no
- error</quote>?]
+ [Q: <literal>error 0</literal> is the same as saying <quote>no
+ error expected, the statement must succeed</quote>?]
</para>
<para>
- [Q: --error 0 is the same as saying <quote>no error expected,
- the statement must succeed</quote>?]
+ [Q: 0 and S00000 are equivalent for specifying <quote>no
+ error</quote>?]
</para>
<para>
- [Q: Can you use <literal>error</literal> to specify shell
- status values for testing the value of shell commands
- (<literal>exec</literal>, <literal>system</literal>)?]
+ [Q: It appears that you use <literal>error</literal> to
+ specify shell status values for testing the value of shell
+ commands executed via the <literal>exec</literal> command.
+ However, this does not seem to aply to the
+ <literal>system</literal>, for which the command status seems
+ to be ignored?]
</para>
</listitem>
@@ -878,16 +884,17 @@
</para>
<para>
- Evaluate the statement and send it to the server as a
- statement to be executed. References to variables within the
- text are replaced with the corresponding values. Use
+ Evaluate the statement and send it to the server to be
+ executed. References to variables within the text are replaced
+ with the corresponding values. Use
‘<literal>\$</literal>’ to specify a literal
‘<literal>$</literal>’ character.
</para>
<para>
- [Q: What is the advantage of using <literal>eval</literal>
- over not using <literal>eval</literal>? Variable reference
+ [Q: What is the advantage of using <literal>eval
+ <replaceable>statement</replaceable></literal> versus just
+ <replaceable>statement</replaceable>? Variable reference
expansion?]
</para>
</listitem>
@@ -898,7 +905,7 @@
</para>
<para>
- [Unknown]
+ [TODO: Unknown]
</para>
</listitem>
@@ -938,6 +945,12 @@
<para>
Terminate the test case.
</para>
+
+ <para>
+ [Q: Is this considered a <quote>normal termination,</quote>
+ such that using <literal>exit</literal> does not result in
+ evaluation of the test case as having failed?]
+ </para>
</listitem>
<listitem>
@@ -963,10 +976,6 @@
<xref linkend="mysqltest-flow-control"/>, for information on
flow-control constructs.
</para>
-
- <para>
- [Q: There is no provision for <literal>else</literal>?]
- </para>
</listitem>
<listitem>
@@ -994,9 +1003,9 @@
<para>
If the <literal>let</literal> command is specified as a normal
command (that is, not beginning with
- ‘<literal>--</literal>’), the value contains
- everything up to the command delimiter, and thus can span
- multiple lines.
+ ‘<literal>--</literal>’),
+ <replaceable>value</replaceable> includes everything up to the
+ command delimiter, and thus can span multiple lines.
</para>
<para>
@@ -1005,10 +1014,9 @@
</para>
<para>
- [Q: The Wiki says there can be no space between the var name
- and the <literal>=</literal> sign. Is that really true? It
- also seems that whitespace following the <literal>=</literal>
- sign is discarded.]
+ [Q: It appears that there can be no space between the variable
+ name and the <literal>=</literal> operator before MySQL
+ 4.1.15? (Otherwise, the statement might assign no value.)]
</para>
</listitem>
@@ -1062,7 +1070,8 @@
<listitem>
<para>
- <literal>query_vertical<replaceable>statement</replaceable></literal>
+ <literal>query_vertical
+ <replaceable>statement</replaceable></literal>
</para>
<para>
@@ -1080,7 +1089,7 @@
<replaceable>num</replaceable> can have a fractional part.
Unlike the <literal>sleep</literal> command,
<literal>real_sleep</literal> is not affected, by the
- <option>--sleep</option> command option.
+ <option>--sleep</option> command-line option.
</para>
</listitem>
@@ -1093,6 +1102,11 @@
Receive the result of the statement most recently sent with
the <literal>send</literal> command.
</para>
+
+ <para>
+ [BUG: If no statement has yet been sent, you'll end up waiting
+ a very long time for the result.]
+ </para>
</listitem>
<listitem>
@@ -1104,8 +1118,8 @@
</para>
<para>
- Replace strings in the result. The value in
- <replaceable>col_num</replaceable> is replaced by the
+ Replace strings in the output from the next statement. The
+ value in <replaceable>col_num</replaceable> is replaced by the
corresponding <replaceable>value</replaceable>. There can be
more than one
<replaceable>col_num</replaceable>/<replaceable>value</replaceable>
@@ -1113,10 +1127,21 @@
</para>
<para>
- [Q: This command applies only to the next command? Is that
- true for all the replace_xxx commands? Can values be quoted or
- specified using variables?]
+ [Q: Can values be quoted or specified using variables?]
</para>
+
+ <para>
+ [Q: If mixed
+ <literal>replace_<replaceable>xxx</replaceable></literal>
+ commands are given, only the final one applies? ]
+ </para>
+
+ <para>
+ [Q: <literal>replace_column</literal> does not appears to
+ apply to the output from <literal>exec</literal>, whereas
+ <literal>replace_regex</literal> and
+ <literal>replace_result</literal> do?]
+ </para>
</listitem>
<listitem>
@@ -1127,13 +1152,13 @@
</para>
<para>
- Find strings that match the pattern
- <replaceable>pattern</replaceable> and replace them with
- <replaceable>replacement</replaceable>. Each instance of a
- string in the line that matches the pattern is replaced.
- Matching is case sensitive by default. Specify the optional
- <literal>i</literal> modifier to cause matching to be case
- insensitive.
+ In the output from the next statement, find strings that match
+ the pattern <replaceable>pattern</replaceable> and replace
+ them with <replaceable>replacement</replaceable>. Each
+ instance of a string in the line that matches the pattern is
+ replaced. Matching is case sensitive by default. Specify the
+ optional <literal>i</literal> modifier to cause matching to be
+ case insensitive.
</para>
<para>
@@ -1167,15 +1192,11 @@
</programlisting>
<para>
- [Better to use patterns with metacharacters for this example]
+ If a given pattern is not found, no error occurs and the input
+ is unchanged.
</para>
<para>
- If a pattern is not found, no error occurs and the input is
- unchanged.
- </para>
-
- <para>
The <literal>replace_regex</literal> command was added in
MySQL 5.1.6.
</para>
@@ -1202,14 +1223,19 @@
</para>
<para>
- [Q: This applies both to the line of column names at the
- beginning of statement output, and to the data lines that
- follow? (Is that true for other replace_xxx commands?)]
+ [Q: Should these inconsistencies exist?
+ <literal>replace_column</literal> applies only to the data
+ rows, and not to the row of column headers.
+ <literal>replace_regex</literal> applies both to the column
+ header row and to the data rows.
+ <literal>replace_result</literal> applies to the column header
+ row, the data rows, <emphasis>and</emphasis> to the SQL
+ statement that generates the query output!]
</para>
<para>
- [Q: The behavior differs from
- <literal>replace_regex</literal>. For example
+ [Q: The behavior for <literal>replace_result</literal> also
+ differs from <literal>replace_regex</literal> this way:
<literal>replace_result a b b c</literal> does not change a to
c. Apparently, result replacement for a column stops as soon
as a match is found?]
@@ -1223,17 +1249,28 @@
</para>
<para>
- Compare the result [Q: from the current test?] with the
- contents of the named file. If the content does not match or
- there is some other error, the test aborts with an
- "abort_not_supported_test" error message.
+ Compare the result with the contents of the named file. If the
+ content does not match or there is some other error, the test
+ aborts with an "abort_not_supported_test" error message.
</para>
+
+ <para>
+ [Q: Does this command specify a file to be used for comparison
+ when the test case completes? Or does the comparison take
+ place immediately, using test case result output generated so
+ far? Is the effect of this command modified if the
+ <option>--record</option> command-line option is given?]
+ </para>
</listitem>
<listitem>
<para>
<literal>require_manager</literal>
</para>
+
+ <para>
+ [TODO: Unknown]
+ </para>
</listitem>
<listitem>
@@ -1242,26 +1279,28 @@
</para>
<para>
- Compare the result [from the current test?] with the contents
- of the named file. If the content does not match or there is
- some other error, write the result to
+ Compare the result with the contents of the named file. If the
+ content does not match or there is some other error, write the
+ result to
<filename>r/<replaceable>file_name</replaceable>.reject</filename>.
</para>
<para>
- [Q: So this assumes that there is an <filename>r</filename>
- directory under the current directory?
+ [Q: Does this command specify a file to be used for comparison
+ when the test case completes? Or does the comparison take
+ place immediately, using test case result output generated so
+ far? Is the effect of this command modified if the
+ <option>--record</option> command-line option is given?]
</para>
<para>
- [Ask these questions, but base them on the Wiki text, not my
- rewrite in the previous para.] [Q: So, does that mean
- <replaceable>file_name</replaceable> is given without the
- <filename>.result</filename> extension? Is the "current" test
- the most recent command executed within the test case, or the
- result of the entire test case up to the
- <literal>result</literal> command? Is "write the result"
- actually "write the diff of the expected and actual results"?]
+ [Q: What is the expected format of
+ <replaceable>file_name</replaceable>? Is it the pathname to
+ the <literal>.result</literal> file, so that the name of the
+ <filename>.reject</filename> file can be determined just be
+ changing the extension? (Otherwise, doesn't this command
+ assume that there is an <filename>r</filename> directory under
+ the current directory?)]
</para>
</listitem>
@@ -1269,6 +1308,10 @@
<para>
<literal>rpl_probe</literal>
</para>
+
+ <para>
+ [TODO: Unknown]
+ </para>
</listitem>
<listitem>
@@ -1294,6 +1337,12 @@
The result must be received with the <literal>reap</literal>
command.
</para>
+
+ <para>
+ [Q: It appears that <replaceable>statement</replaceable> is
+ optional? What is the effect of the command if the statement
+ is omitted?]
+ </para>
</listitem>
<listitem>
@@ -1350,8 +1399,8 @@
<para>
Sleep <replaceable>num</replaceable> seconds.
<replaceable>num</replaceable> can have a fractional part. If
- the <option>--sleep</option> command option was given, the
- option value overrides the value given in the
+ the <option>--sleep</option> command-line option was given,
+ the option value overrides the value given in the
<literal>sleep</literal> command. For example, if
<command>mysqltest</command> is started with
<option>--sleep=10</option>, the command <literal>sleep
@@ -1388,7 +1437,7 @@
</para>
<para>
- [Q: Max recursion level?]
+ [Q: Max nesting level?]
</para>
</listitem>
@@ -1506,8 +1555,8 @@
<para>
Poll the current connection, which is assumed to be a
- connection to a slave replication server, by using the
- <literal>SHOW STATUS LIKE 'Slave_running'</literal> statement
+ connection to a slave replication server, by executing
+ <literal>SHOW STATUS LIKE 'Slave_running'</literal> statements
until the result is <literal>OFF</literal>.
</para>
</listitem>
@@ -1550,35 +1599,74 @@
</para>
<para>
- Variables:
+ You can define variables and refer to their values. You can also
+ refer to environment variables, and there is a built-in variable
+ for that contains the result of the most recent SQL statement.
</para>
<para>
- <literal>$mysql_errno</literal> is a built-in variable that
- contains the numeric error returned by the most recent statement
- sent to the server, or 0 if the command executes successfully.
- <literal>$mysql_errno</literal> has a value of −1 if no
- command has yet been sent.
+ To define a variable, use the <literal>let</literal> command.
+ Examples:
</para>
+<programlisting>
+let $a = 14;
+let $a = this is a string;
+--let $a = 14
+--let $a = this is a string
+</programlisting>
+
<para>
- [Q: Is $mysql_errno set only by SQL statements, or is it also set
- by exec, system, mysqltest internal commands?]
+ If a variable has a numeric value, you can increment or decrement
+ the value:
</para>
+<programlisting>
+inc $a;
+dec $a;
+--inc $a
+--dec $a
+</programlisting>
+
<para>
- You can refer to environment variables. [Q: How is a variable
- determined to be such? If it's not recognized as a built-in
- variable or a variable defined in the script?]
+ <literal>inc</literal> and <literal>dec</literal> are commonly
+ used in <literal>while</literal> loops to modify the value of a
+ counter variable that controls loop execution.
</para>
<para>
- [Q: Variable names appear not to be case sensitive. This leads to
- an ambiguity: If you refer to a variable named
- <literal>$path</literal>, is that a <command>mysqltest</command>
- variable, or the <literal>$PATH</literal> environment variable?]
+ References to variables can occur in the <literal>echo</literal>,
+ <literal>eval</literal>, <literal>exec</literal>, and
+ <literal>system</literal> commands. Variable references are
+ replaced by their values.
</para>
+ <para>
+ You can refer to environment variables. For example, this command
+ displays the value of the <literal>$PATH</literal> variable from
+ the environment:
+ </para>
+
+<programlisting>
+--echo $PATH
+</programlisting>
+
+ <para>
+ <literal>$mysql_errno</literal> is a built-in variable that
+ contains the numeric error returned by the most recent SQL
+ statement sent to the server, or 0 if the command executed
+ successfully. <literal>$mysql_errno</literal> has a value of
+ −1 if no statement has yet been sent.
+ </para>
+
+ <para>
+ [Q: Variable names appear not to be case sensitive. Is that
+ correct? If so, that leads to an ambiguity: If you refer to a
+ variable named <literal>$path</literal>, is that a
+ <command>mysqltest</command> variable, or the
+ <literal>$PATH</literal> environment variable?]
+ </para>
+
</section>
<section id="mysqltest-flow-control">
@@ -1586,7 +1674,8 @@
<title><command>mysqltest</command> Flow Control Constructs</title>
<para>
- [Cover <literal>exit</literal> here?]
+ [Q: There is no provision for <literal>else</literal> with
+ <literal>if</literal>?]
</para>
<para>
@@ -1614,31 +1703,43 @@
</para>
<para>
+ For a <literal>while</literal> loop, make sure that the loop
+ includes some exit condition that eventually occurs. This can be
+ done by writing <replaceable>expr</replaceable> so that it goes
+ false at some point.
+ </para>
+
+ <para>
[Q: The opening <literal>{</literal> must be on the next line
following <literal>if</literal> or <literal>while</literal>?]
</para>
<para>
- No spaces allowed between parens and the expression within the
- parens? What is the allowable syntax for <literal>expr</literal>?]
+ [Q: No spaces allowed between parens and the expression within the
+ parens? What is the allowable syntax for
+ <replaceable>expr</replaceable>?]
</para>
<para>
- [Q: I'm confused by this in several respects. Is a block written
- using <literal>{</literal> and <literal>}</literal>, or does it
- end with <literal>end</literal>? Also, the comments in the code
- indicate that <replaceable>expr</replaceable> is true if greater
- than zero and that adding <literal>!</literal> makes
- <replaceable>expr</replaceable> true if it's zero. But those are
- not complementary. What if <replaceable>expr</replaceable> is less
- than zero?]
+ [Q: Is a block written using <literal>{</literal> and
+ <literal>}</literal>, or does it end with <literal>end</literal>?
+ Or is either syntax allowable?]
</para>
<para>
- Hmm: A block can begin with <literal>{</literal> and end with
- <literal>end</literal>.
+ [Hmm: A block can begin with <literal>{</literal> and end with
+ <literal>end</literal>.]
</para>
+ <para>
+ [Q: The comments in the source code indicate that
+ <replaceable>expr</replaceable> is true if greater than zero and
+ that adding <literal>!</literal> makes
+ <replaceable>expr</replaceable> true if it's zero. But those are
+ not complementary conditions. What if
+ <replaceable>expr</replaceable> is less than zero?]
+ </para>
+
</section>
</chapter>
Modified: trunk/mysqltest/mysqltest.xml
===================================================================
--- trunk/mysqltest/mysqltest.xml 2006-03-14 21:49:31 UTC (rev 1581)
+++ trunk/mysqltest/mysqltest.xml 2006-03-14 21:55:52 UTC (rev 1582)
@@ -3795,7 +3795,7 @@
</para>
<para>
- The <command>mysqltest</command> program is a small interpreters
+ The <command>mysqltest</command> program is a small interpreter
for a special test language. Lines that it doesn't recognize as
its own commands are assumed to be SQL statements to be sent to
the MySQL server. In effect, this makes the test files look like
Modified: trunk/mysqltest/tutorial.xml
===================================================================
--- trunk/mysqltest/tutorial.xml 2006-03-14 21:49:31 UTC (rev 1581)
+++ trunk/mysqltest/tutorial.xml 2006-03-14 21:55:52 UTC (rev 1582)
@@ -7,147 +7,72 @@
]>
<chapter id="tutorial">
- <title>MySQL Test Framework Tutorial</title>
+ <title>Tutorial</title>
<para>
[This chapter is based on the information that was in the Wiki]
</para>
<para>
- [TODO: Add tutorial introductory text here]
+ This chapter provides a tutorial on running and writing test cases
+ for the MySQL test framework. It gives a brief description of how to
+ write test cases and the language used to write them. Normally, you
+ run the test suite during the development process to ensure that
+ your changes do not cause existing test cases to break. You can also
+ write new test cases or add tests to existing cases. This happens
+ when you fix a bug (so that the bug cannot reappear later without
+ being detected) or when you add new capabilities to the server or
+ other MySQL programs.
</para>
- <section id="tutorial-test-case-basics">
+ <section id="tutorial-running-test-cases">
- <title>Test Cases Basics</title>
+ <title>Running Test Cases</title>
<para>
- The base principle is that the test script compares the output
- from running the tests with the expected result. This is just a
- diff between the output and a file that the test writer provides.
- With this simplistic way of comparing, you can't handle variations
- in the output from running the script. However, the test language
- provides ways to suppress unwanted output and making the result
- usable. Use the following procedure to write a new test case:
+ Normally, you run the test suite either from within a source tree
+ (after MySQL has been built), or on a host where the MySQL server
+ distribution has been installed. A source tree has a
+ <filename>mysql-test</filename> directory, and an installed server
+ distribution also should have a <filename>mysql-test</filename>
+ directory. To run tests, you should be located in the
+ <filename>mysql-test</filename> directory. The program that runs
+ the test suite, <command>mysql-test-run</command>, will figure out
+ whether you are in a source tree or an installed directory tree.
</para>
<para>
- [Q: This procedure seems to ignore the existence of the
- <option>--record</option> option. Perhaps it was written before
- that option was available?]
+ To run the test suite, change location into your
+ <filename>mysql-test</filename> directory and invoke the
+ <command>mysql-test-run</command> script:
</para>
- <orderedlist>
-
- <listitem>
- <para>
- Change directories to the test directory
- <filename>mysql-<replaceable>version</replaceable>/mysql-test</filename>:
- </para>
-
<programlisting>
-shell> <userinput>cd mysql-<replaceable>version</replaceable>/mysql-test</userinput>
+shell> <userinput>cd mysql-test</userinput>
+shell> <userinput>./mysql-test-run</userinput>
</programlisting>
- </listitem>
- <listitem>
- <para>
- Create the test in a file
- <filename>t/<replaceable>test_name</replaceable>.test</filename>:
- </para>
- </listitem>
-
- <listitem>
- <para>
- Create an empty result file:
- </para>
-
-<programlisting>
-shell> <userinput>touch r/<replaceable>test_name</replaceable>.result</userinput>
-</programlisting>
- </listitem>
-
- <listitem>
- <para>
- Run the test:
- </para>
-
-<programlisting>
-shell> <userinput>./mysql-test-run <replaceable>test_name</replaceable></userinput>
-</programlisting>
- </listitem>
-
- <listitem>
- <para>
- The failing test will create a file
- <filename>r/<replaceable>test_name</replaceable>.reject</filename>.
- Examine this file. If it looks okay, copy its content to the
- result file:
- </para>
-
-<programlisting>
-shell> <userinput>cp r/<replaceable>test_name</replaceable>.reject r/<replaceable>test_name</replaceable>.result</userinput>
-</programlisting>
- </listitem>
-
- <listitem>
- <para>
- Run the test again. This time it should succeed:
- </para>
-
-<programlisting>
-shell> <userinput>./mysql-test-run <replaceable>test_name</replaceable></userinput>
-</programlisting>
- </listitem>
-
- </orderedlist>
-
<para>
- The environment can also be set up so that the
- <command>mysqltest</command> program can be called directly. For
- more information about options to the <command>mysqltest</command>
- program run this command:
+ In a source distribution, <filename>mysql-test</filename> is under
+ the root of the source tree. In a binary distribution, the
+ location of <filename>mysql-test</filename> depends on the
+ distribution layout.
</para>
-<programlisting>
-shell> <userinput>mysqltest --help</userinput>
-</programlisting>
-
<para>
- There are also some notes about <command>mysqltest</command> in
- the <filename>README</filename> in the
- <filename>mysql-test</filename> directory.
+ <command>mysql-test-run</command> accepts options on the command
+ line. For example:
</para>
- </section>
-
- <section id="tutorial-running-test-cases">
-
- <title>Running Test Cases</title>
-
- <para>
- This is a brief description of how to write test cases and the
- language used to write them. Normally you run the test suite, when
- developing.
- </para>
-
- <para>
- For example:
- </para>
-
<programlisting>
-shell> <userinput>cd mysql-test</userinput>
-shell> <userinput>mysql-test-run</userinput>
+shell> <userinput>./mysql-test-run --local --force</userinput>
</programlisting>
<para>
- Or:
+ [Q: What is the point of showing <option>--local</option> in the
+ example? Isn't that the default anyway?]
</para>
-<programlisting>
-shell> <userinput>mysql-test-run --local --force</userinput>
-</programlisting>
-
<para>
Normally, <command>mysql-test-run</command> exits if a test case
fails. <option>--force</option> causes execution to continue
@@ -155,48 +80,39 @@
</para>
<para>
- [TODO: the point about the .test extension is a more general
- principle, not specific to following example]
+ Test case files have names like
+ <filename>t/<replaceable>test_name</replaceable>.test</filename>,
+ where <replaceable>test_name</replaceable> is the name of the test
+ case.
</para>
<para>
- For a single test case or group or test cases (the
- <filename>.test</filename> extension exluded)
+ To run one or more specific test cases, give their names on the
+ <command>mysql-test-run</command> command line. (You do not
+ specify the full test case filename.) This command runs the test
+ case in the <filename>t/rpl_abcd.test</filename> file:
</para>
<programlisting>
-shell> <userinput>mysql-test-run --local --do-test=<replaceable>test_name_prefix</replaceable></userinput>
+shell> <userinput>./mysql-test-run rpl_abcd</userinput>
</programlisting>
<para>
- This can be a single test case or a many tests with the same
- prefix.
+ If you have a family of test cases for which the names share a
+ common prefix, you can run all tests that have that prefix by
+ using the <option>--do-tests</option> option:
</para>
- <para>
- Examples:
- </para>
-
- <para>
- [TODO: Fix next example, it's incorrect (doesn't run just one test
- case if there are others with same prefix]
- </para>
-
- <para>
- This will run the <filename>rpl_abcd.test</filename>:
- </para>
-
<programlisting>
-shell> <userinput>mysql-test-run --local --do-test=rpl_abcd</userinput>
+shell> <userinput>./mysql-test-run --do-test=<replaceable>prefix</replaceable></userinput>
</programlisting>
<para>
- This will run all the tests that start with
- <literal>rpl</literal>:
+ For example, the following command runs the replication tests:
</para>
<programlisting>
-shell> <userinput>mysql-test-run --local --do-test=rpl</userinput>
+shell> <userinput>./mysql-test-run --do-tests=rpl</userinput>
</programlisting>
<para>
@@ -214,29 +130,14 @@
<para>
The test case language is a mix of commands that the
<command>mysqltest</command> program understands and SQL
- statements. What it doesn't understand is assumed to be SQL
- statements and are sent to the database server. This makes the
- test case language familiar to those that know how to write SQL
- and powerful enough to add the control needed to write test cases.
+ statements. Input that <command>mysqltest</command> doesn't
+ understand is assumed to be SQL statements and are sent to the
+ database server. This makes the test case language familiar to
+ those that know how to write SQL and powerful enough to add the
+ control needed to write test cases.
</para>
<para>
- [TODO: Merge w/previous stuff]
- </para>
-
- <para>
- Normally, you run the test case either from within a source tree
- (after MySQL has been built), or on a host where the MySQL server
- distribution has been installed. A source tree has a
- <filename>mysql-test</filename> directory, and an installed server
- distribution also should have a <filename>mysql-test</filename>
- directory. To run tests, you should be located in the
- <filename>mysql-test</filename> directory. The program that runs
- the test suite, <command>mysql-test-run</command>, will figure out
- whether you are in a source tree or an installed directory tree.
- </para>
-
- <para>
You need not start a MySQL server first before running tests.
Instead, the <command>mysql-test-run</command> program will start
the server or servers needed on ports that do not conflict with
@@ -245,40 +146,155 @@
</para>
<para>
- [Q: Doesn't the preceding paragraph assume if you have a
+ [Q: Doesn't the preceding paragraph assume that if you have a
production server, it doesn't happen to be using any non-standard
ports used by servers started by the test programs?]
</para>
+ <section id="tutorial-simultaneous-test-runs">
+
+ <title>Constraints on Simultaneous Test Runs</title>
+
+ <para>
+ If you have multiple users that run tests simultaneously on the
+ same machine, you must specify the ports to the
+ <command>mysql-test-run</command> program, so that no test run
+ conflicts with others running concurrently. You add unique port
+ arguments to <command>mysql-test-run</command>, such as
+ <option>--no-manager --master_port=3911
+ --slave_port=3927</option>.
+ </para>
+
+ <para>
+ Also note that only one person can run the
+ <command>mysql-test-run</command> program in the same
+ <filename>mysql-test</filename> directory on a shared drive, at
+ the same time. The <filename>mysql-test/var</filename> directory
+ created and used by <command>mysql-test-run</command> cannot be
+ shared between simultaneous test runs.
+ </para>
+
+ <para>
+ [Q: Is there a <command>mysql-test-run</command> option that can
+ be used to cause each test run to use a separate var directory?]
+ </para>
+
+ </section>
+
+ </section>
+
+ <section id="tutorial-test-case-quick-start">
+
+ <title>Writing a Test Case: Quick Start</title>
+
<para>
- You can run the newly created test case <literal>foo</literal> as
- part of the entire suite:
+ The basic principle is that evaluation of the output resulting
+ from running a test case is based on comparison of that output
+ with the expected result. This is just a <command>diff</command>
+ comparison between the output and a file that the test writer
+ provides. This simplistic method of comparison does not by itself
+ provide any way to handle variation in the output that may occur
+ when a test is run at different times. However, the test language
+ provides some commands for postprocessing result output before the
+ comparison occurs. This enables you to manage expected variations.
</para>
-<programlisting>
-shell> <userinput>./mysql-test-run</userinput>
-</programlisting>
+ <para>
+ Use the following procedure to write a new test case:
+ </para>
<para>
- To run just the <literal>foo</literal> test, give its name on the
- command line:
+ [Q: This procedure seems to ignore the existence of the
+ <option>--record</option> option. Perhaps it was written before
+ that option was available? In any case, is this the procedure that
+ people actually use, or should it be rewritten to use
+ <option>--record</option>?]
</para>
+ <orderedlist>
+
+ <listitem>
+ <para>
+ Change directories to the test directory
+ <filename>mysql-<replaceable>version</replaceable>/mysql-test</filename>:
+ </para>
+
<programlisting>
-shell> <userinput>./mysql-test-run foo</userinput>
+shell> <userinput>cd mysql-<replaceable>version</replaceable>/mysql-test</userinput>
</programlisting>
+ </listitem>
- <para>
- If you have a family of test cases for which the names share a
- common prefix, you can run all tests that have that prefix by
- using the <option>--do-tests</option> option. For example, the
- following command runs the replication tests:
- </para>
+ <listitem>
+ <para>
+ Create the test in a file
+ <filename>t/<replaceable>test_name</replaceable>.test</filename>.
+ You can do this with any test editor. For details of the
+ language used for writing <command>mysqltest</command> test
+ cases, see <xref linkend="mysqltest-reference"/>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Create an empty result file:
+ </para>
+
<programlisting>
-shell> <userinput>./mysql-test-run --do-tests=rpl</userinput>
+shell> <userinput>touch r/<replaceable>test_name</replaceable>.result</userinput>
</programlisting>
+ </listitem>
+ <listitem>
+ <para>
+ Run the test:
+ </para>
+
+<programlisting>
+shell> <userinput>./mysql-test-run <replaceable>test_name</replaceable></userinput>
+</programlisting>
+ </listitem>
+
+ <listitem>
+ <para>
+ The failing test will create a file
+ <filename>r/<replaceable>test_name</replaceable>.reject</filename>.
+ Examine this file. If it looks okay, copy its content to the
+ result file:
+ </para>
+
+<programlisting>
+shell> <userinput>cp r/<replaceable>test_name</replaceable>.reject r/<replaceable>test_name</replaceable>.result</userinput>
+</programlisting>
+ </listitem>
+
+ <listitem>
+ <para>
+ Run the test again. This time it should succeed:
+ </para>
+
+<programlisting>
+shell> <userinput>./mysql-test-run <replaceable>test_name</replaceable></userinput>
+</programlisting>
+
+ <para>
+ You can also run the newly created test case as part of the
+ entire suite:
+ </para>
+
+<programlisting>
+shell> <userinput>./mysql-test-run</userinput>
+</programlisting>
+ </listitem>
+
+ </orderedlist>
+
+ <para>
+ The environment can also be set up so that the
+ <command>mysqltest</command> program can be called directly. For
+ more information about the <command>mysqltest</command> program,
+ see <xref linkend="mysqltest"/>.
+ </para>
+
</section>
<section id="tutorial-writing-test-cases">
@@ -286,9 +302,9 @@
<title>Writing a Test Case</title>
<para>
- You write test cases in any text editor that use LF (linefeed) as
- the end-of-line character. Filenames should be lowercase ASCII
- with no spaces.
+ You write test cases with any text editor that uses linefeed
+ (newline) as the end-of-line character. Filenames should be
+ lowercase ASCII with no spaces.
</para>
<para>
@@ -302,8 +318,8 @@
<para>
<emphasis role="bold">Note</emphasis>: All our test cases are
- published on the Internet, so take care that their contents
- include no confidential information.
+ published on the Internet. Take care that their contents include
+ no confidential information.
</para>
<para>
@@ -333,21 +349,29 @@
sequence</quote> (that is, a number of individual tests that
are grouped together in the same test file).
</para>
+
+ <para>
+ A <quote>command</quote> is an input test that
+ <command>mysqltest</command> recognizes and executes itself. A
+ <quote>statement</quote> is an SQL statement or query that
+ <command>mysqltest</command> sends to the MySQL server to be
+ executed.
+ </para>
</listitem>
</itemizedlist>
<para>
- A test case (that is, a test file) must be self contained and
- independent of other test cases. This means that you are not
+ Each test case (that is, each test file) must be self contained
+ and independent of other test cases. This means that you are not
allowed to create or populate a table in one test case and depend
on the table in a later test case. If you have some common
initialization that needs to be done for multiple test cases,
create an include file. That is, create a file containing the
initialization code in the <filename>mysq-test/include</filename>
directory, and then put a <literal>source</literal> command in
- each test case that requires the code. For example, if two test
- cases both need to have a given table created and filled with
+ each test case that requires the code. For example, if several
+ test cases need to have a given table created and filled with
data, put the statements to do that in a file named
<filename>mysql-test/include/create_my_table.inc</filename>. Then
put this command in each test case file that needs the
@@ -373,22 +397,27 @@
<para>
It is safest to use the ‘<literal>#</literal>’
- character for comments, not to accidently run a mysq-test
- directive/command. For example writing <literal>-- End or test
- 43</literal> will give an error message because
- <literal>end</literal> is something that
+ character for comments, so as not to accidentally execute a
+ <command>mysqltest</command> command. For example, <literal>-- End
+ of test 43</literal> begins with the
+ ‘<literal>--</literal>’ characters, but will result in
+ an error message because <literal>end</literal> is something that
<command>mysqltest</command> thinks is a command.
</para>
- </section>
+ <para>
+ The input syntax for <command>mysqltest</command> test cases is
+ discussed in detail in
+ <xref linkend="mysqltest-input-conventions"/>.
+ </para>
- <section id="tutorial-sample-test-case">
+ <section id="tutorial-sample-test-case">
- <title>Sample Test Case</title>
+ <title>Sample Test Case</title>
- <para>
- Here is a small sample test case:
- </para>
+ <para>
+ Here is a small sample test case:
+ </para>
<programlisting>
--disable_warnings
@@ -401,23 +430,24 @@
INSERT INTO t1 VALUES ("hej");
</programlisting>
- <para>
- The first rows try to clean up from possible earlier runs of the
- test case by dropping the <literal>t1</literal> table. We disable
- storing warnings (<literal>disable_warnings</literal>) in the
- output during this because we are not interested to know whether
- the table <literal>t1</literal> was there or not. Then we enable
- storing warnings again (<literal>enable_warnings</literal>) and
- enable verbose warnings in MySQL using the <literal>SET
- SQL_WARNINGS=1;</literal> statement.
- </para>
+ <para>
+ The first rows try to clean up from possible earlier runs of the
+ test case by dropping the <literal>t1</literal> table. We
+ disable storing warnings (<literal>disable_warnings</literal>)
+ in the output during this because we are not interested to know
+ whether the table <literal>t1</literal> was there or not. Then
+ we enable storing warnings again
+ (<literal>enable_warnings</literal>) and enable verbose warnings
+ in MySQL using the <literal>SET SQL_WARNINGS=1;</literal>
+ statement.
+ </para>
- <para>
- We create the table <literal>t1</literal> and try some operations.
- The creation of the table and first insertion will not create any
- warnings but the last line will. We collect the output from
- running this test and add this to the result file:
- </para>
+ <para>
+ We create the table <literal>t1</literal> and try some
+ operations. The creation of the table and first insertion will
+ not create any warnings but the last line will. We collect the
+ output from running this test and add this to the result file:
+ </para>
<programlisting>
DROP TABLE IF EXISTS t1;
@@ -429,59 +459,64 @@
Warning 1265 Data truncated for column 'a' at row 1
</programlisting>
- <para>
- Note that we see the statements as well in the result. This can be
- disabled with the <literal>disable_query_log</literal> test
- language command. There are several options how to control the
- amount of output from running the tests. If there was an failure
- this will be reported to the screen. You can see the actual output
- from the last unsuccessful run of the test case in the test case
- reject file
- <filename>r/<replaceable>test_name</replaceable>.reject</filename>.
- </para>
+ <para>
+ Note that we see the statements as well in the result. This can
+ be disabled with the <literal>disable_query_log</literal> test
+ language command. There are several options how to control the
+ amount of output from running the tests. If there was an failure
+ this will be reported to the screen. You can see the actual
+ output from the last unsuccessful run of the test case in the
+ test case reject file
+ <filename>r/<replaceable>test_name</replaceable>.reject</filename>.
+ </para>
- </section>
+ </section>
- <section id="tutorial-naming-rules">
+ <section id="tutorial-naming-rules">
- <title>Naming Rules for Database Objects</title>
+ <title>Naming Rules for Database Objects</title>
- <para>
- The test cases might be used to run against a production server
- (we will generally not do that, but our customers might). To
- reduce the risk that running the test suite alters or destroys
- important tables, views, or other objects, you should name them
- using names like this:
- </para>
+ <para>
+ The test cases might be used to run against a production server
+ (generally, we will not do that, but our customers might). To
+ reduce the risk that running the test suite alters or destroys
+ important tables, views, or other objects, you should name them
+ using names like this:
+ </para>
<programlisting>
table names : t1, t2, t3.....
view names : v1, v2, v3.....
</programlisting>
- <para>
- Look in the existing test cases for examples of this. Of course,
- you can name columns and other objects inside tables as you wish.
- Unless you have a special reason not to, use the default database
- named <literal>test</literal> that is already created for you.
- </para>
+ <para>
+ Look in the existing test cases for examples of this. Of course,
+ you can name columns and other objects inside tables as you
+ wish.
+ </para>
- </section>
+ <para>
+ Unless you have a special reason not to, use the default
+ database named <literal>test</literal> that is already created
+ for you.
+ </para>
- <section id="tutorial-cleaning-up">
+ </section>
- <title>Cleaning Up from a Previous Run</title>
+ <section id="tutorial-cleaning-up">
- <para>
- For efficiency, the <command>mysqltest</command> test engine does
- not start with a clean new database for running each test case. So
- a test case generally starts with a <quote>cleaning up
- section.</quote> Assume that our test case will use two tables
- named <literal>t1</literal> and <literal>t2</literal>. The test
- case should begin by making sure that any old tables with those
- names do not exist:
- </para>
+ <title>Cleaning Up from a Previous Run</title>
+ <para>
+ For efficiency, the <command>mysqltest</command> test engine
+ does not start with a clean new database for running each test
+ case. So a test case generally starts with a <quote>cleaning up
+ section.</quote> Assume that our test case will use two tables
+ named <literal>t1</literal> and <literal>t2</literal>. The test
+ case should begin by making sure that any old tables with those
+ names do not exist:
+ </para>
+
<programlisting>
#
# Test of XXXXX
@@ -491,31 +526,31 @@
--enable_warnings
</programlisting>
- <para>
- The <literal>disable_warnings</literal> command instructs the test
- engine not to log any warnings until
- <literal>enable_warnings</literal> or the test case is ended. For
- some unknown reason, MySQL chooses to write a warning if the table
- <literal>t1</literal> or <literal>t2</literal> exists. Surrounding
- this part of the test case with the disable/enable warnings makes
- its output the same whether or not the tables exists before the
- test is started. After ensuring that the tables do not exist, we
- are free to put in any SQL statements that create and use the
- tables <literal>t1</literal> and <literal>t2</literal>. The test
- case should also clean up at the end of the test by dropping any
- tables that it creates.
- </para>
+ <para>
+ The <literal>disable_warnings</literal> command instructs the
+ test engine not to log any warnings until
+ <literal>enable_warnings</literal> or the test case is ended.
+ For some unknown reason, MySQL chooses to write a warning if the
+ table <literal>t1</literal> or <literal>t2</literal> exists.
+ Surrounding this part of the test case with the disable/enable
+ warnings makes its output the same whether or not the tables
+ exists before the test is started. After ensuring that the
+ tables do not exist, we are free to put in any SQL statements
+ that create and use the tables <literal>t1</literal> and
+ <literal>t2</literal>. The test case should also clean up at the
+ end of the test by dropping any tables that it creates.
+ </para>
- <para>
- So let's put in some SQL code into this test case:
- </para>
+ <para>
+ So let's put in some SQL code into this test case:
+ </para>
- <para>
- [Q: There is a disconnect between this test case content and the
- preceding cleanup code: The cleanup code removes
- <literal>t1</literal> and <literal>t2</literal>, but the test
- never uses <literal>t2</literal>.
- </para>
+ <para>
+ [Q: There is a disconnect between this test case content and the
+ preceding cleanup code? (The cleanup code removes
+ <literal>t1</literal> and <literal>t2</literal>, but the test
+ never uses <literal>t2</literal>.)]
+ </para>
<programlisting>
create table t1 (
@@ -532,56 +567,57 @@
drop table t1;
</programlisting>
- </section>
+ </section>
- <section id="tutorial-test-result">
+ <section id="tutorial-test-result">
- <title>Generating a Test Case Result File</title>
+ <title>Generating a Test Case Result File</title>
- <para>
- Note that the test code we just wrote contains no checks of the
- result. The test will report a failure for one of two reasons
- </para>
+ <para>
+ Note that the test code we just wrote contains no checks of the
+ result. The test will report a failure for one of two reasons
+ </para>
- <itemizedlist>
+ <itemizedlist>
- <listitem>
- <para>
- An individual SQL statement failed with an error
- </para>
- </listitem>
+ <listitem>
+ <para>
+ An individual SQL statement failed with an error
+ </para>
+ </listitem>
- <listitem>
- <para>
- The overall test case result does not match what was expected
- </para>
- </listitem>
+ <listitem>
+ <para>
+ The overall test case result does not match what was
+ expected
+ </para>
+ </listitem>
- </itemizedlist>
+ </itemizedlist>
- <para>
- In the first case, <command>mysqltest</command> aborts with an
- error. The second case requires that we have a record of the
- expected result so that it can be compared with the actual result.
- To generate a file that contains the test result, run the test
- with the <option>--record</option> option, like this:
- </para>
+ <para>
+ In the first case, <command>mysqltest</command> aborts with an
+ error. The second case requires that we have a record of the
+ expected result so that it can be compared with the actual
+ result. To generate a file that contains the test result, run
+ the test with the <option>--record</option> option, like this:
+ </para>
<programlisting>
shell> <userinput>cd mysql-test</userinput>
shell> <userinput>./mysql-test-run --record foo</userinput>
</programlisting>
- <para>
- [Q: Is the <quote>protocol</quote> file the result file or the
- reject file? I have seen the term applied to both.]
- </para>
+ <para>
+ [Q: Is the <quote>protocol</quote> file the result file or the
+ reject file? I have seen the term applied to both.]
+ </para>
- <para>
- This creates a file named
- <filename>mysql-test/r/foo.result</filename>, also called the
- <quote>protocol file.</quote> The file has this content
- </para>
+ <para>
+ This creates a file named
+ <filename>mysql-test/r/foo.result</filename>, also called the
+ <quote>protocol file.</quote> The file has this content
+ </para>
<programlisting>
drop table if exists t1,t2;
@@ -602,43 +638,44 @@
drop table t1;
</programlisting>
- <para>
- If we look at this result file, it contains the commands in the
- <filename>foo.test</filename> file together with the output from
- the <literal>SELECT</literal> statements. The output for each
- statement includes a row of column headings followed by data rows.
- Rows have columns separated by Tab characters.
- </para>
+ <para>
+ If we look at this result file, it contains the commands in the
+ <filename>foo.test</filename> file together with the output from
+ the <literal>SELECT</literal> statements. The output for each
+ statement includes a row of column headings followed by data
+ rows. Rows have columns separated by Tab characters.
+ </para>
- <para>
- At this point, you should inspect the result file and determine
- whether its contents are as expected. If so, let it be part of
- your test case.
- </para>
+ <para>
+ At this point, you should inspect the result file and determine
+ whether its contents are as expected. If so, let it be part of
+ your test case.
+ </para>
- <para>
- [Q: And if it's not as expected, what do you do?]
- </para>
+ <para>
+ [Q: And if it's not as expected, what do you do?]
+ </para>
- </section>
+ </section>
- <section id="tutorial-expecting-errors">
+ <section id="tutorial-expecting-errors">
- <title>Specifying When Tests are Expected to Fail</title>
+ <title>Specifying When Tests are Expected to Fail</title>
- <para>
- A good test suite not only tests that operations succeed as they
- ought, but also that they fail as they ought. For example, if a
- statement is illegal, the server should reject it with an error
- message. The test suite should verify that the statement fails,
- and that it fails with the proper error message.
- </para>
+ <para>
+ A good test suite not only tests that operations succeed as they
+ ought, but also that they fail as they ought. For example, if a
+ statement is illegal, the server should reject it with an error
+ message. The test suite should verify that the statement fails,
+ and that it fails with the proper error message.
+ </para>
- <para>
- The test engine let you specify <quote>expected failures.</quote>
- Let's say that after we create <literal>t1</literal>, we try to
- create it again without dropping it first:
- </para>
+ <para>
+ The test engine let you specify <quote>expected
+ failures.</quote> Let's say that after we create
+ <literal>t1</literal>, we try to create it again without
+ dropping it first:
+ </para>
<programlisting>
--disable_warnings
@@ -659,142 +696,114 @@
create table t1 (something smallint(4));
</programlisting>
- <para>
- The result is failure and an error:
- </para>
+ <para>
+ The result is failure and an error:
+ </para>
- <para>
- [Q: word-police note: <literal>create table</literal> is a
- statement, not a query. <quote>statement</quote> would be a better
- term here in the error, because it applies to both queries and
- non-queries.]
- </para>
+ <para>
+ [Q: word-police note: <literal>create table</literal> is a
+ statement, not a query. <quote>statement</quote> would be a
+ better term here in the error, because it applies to both
+ queries and non-queries.]
+ </para>
<programlisting>
At line 19: query 'create table t1 (something smallint(4))'
failed: 1050: Table 't1' already exists
</programlisting>
- <para>
- To handle this and indicate that indeed we do expect MySQL error
- code 1050 to occur, we can put an <literal>error</literal> command
- before the second <literal>create table</literal> statement:
- </para>
+ <para>
+ To handle this and indicate that indeed we do expect MySQL error
+ code 1050 to occur, we can put an <literal>error</literal>
+ command before the second <literal>create table</literal>
+ statement:
+ </para>
<programlisting>
--error 1050
</programlisting>
- <para>
- After making this change and running the test again, the end of
- the result will look like this:
- </para>
+ <para>
+ After making this change and running the test again, the end of
+ the result will look like this:
+ </para>
<programlisting>
create table t1 (something smallint(4));
ERROR 42S01: Table 't1' already exists
</programlisting>
- <para>
- In this case, the result shows the statement that results in the
- error, together with the resulting error message. The fact that
- <command>mysqltest</command> does not terminate and that the error
- message becomes part of the result indicates that the error was
- expected.
- </para>
+ <para>
+ In this case, the result shows the statement that results in the
+ error, together with the resulting error message. The fact that
+ <command>mysqltest</command> does not terminate and that the
+ error message becomes part of the result indicates that the
+ error was expected.
+ </para>
- <para>
- Note: 1050 is the numeric MySQL error number, and 42S01 is the
- corresponding SQLSTATE value. If you like, you can specify
- SQLSTATE values in <literal>error</literal> commands by using an
- <literal>S</literal> prefix:
- </para>
+ <para>
+ Note: 1050 is the numeric MySQL error number, and 42S01 is the
+ corresponding SQLSTATE value. If you like, you can specify
+ SQLSTATE values in <literal>error</literal> commands by using an
+ <literal>S</literal> prefix:
+ </para>
<programlisting>
--error S42S01
</programlisting>
- <para>
- [Q: It's inconsistent and confusing that
- <command>mysqltest</command> displays numeric error codes in one
- error-reporting context and SQLSTATE values in another
- error-reporting context. Wouldn't it be better for
- <command>mysqltest</command> to consistently display the same kind
- of information? Or would this require many test results to be
- updated. ]
- </para>
+ <para>
+ [Q: It's inconsistent and confusing that
+ <command>mysqltest</command> displays numeric error codes in one
+ error-reporting context and SQLSTATE values in another
+ error-reporting context. Wouldn't it be better for
+ <command>mysqltest</command> to consistently display the same
+ kind of information? Or would this require many test results to
+ be updated?]
+ </para>
- </section>
+ </section>
- <section id="tutorial-simultaneous-test-runs">
+ <section id="tutorial-controlling-information">
- <title>Constraints on Simultaneous Test Runs</title>
+ <title>Controlling the Information Produced by a Test Case</title>
- <para>
- If you have multiple users that run tests simultaneously on the
- same machine, you must specify the ports to the
- <command>mysql-test-run</command> program, so that no test run
- conflicts with others running concurrently. You add unique port
- arguments to <command>mysql-test-run</command>, such as
- <option>--no-manager --master_port=3911
- --slave_port=3927</option>.
- </para>
+ <section id="tutorial-more-information">
- <para>
- Also note that only one person can run the
- <command>mysql-test-run</command> program in the same
- <filename>mysql-test</filename> directory on a shared drive, at
- the same time. The <filename>mysql-test/var</filename> directory
- created and used by <command>mysql-test-run</command> cannot be
- shared between simultaneous test runs.
- </para>
+ <title>Getting More Information for Checking the Test Result</title>
- <para>
- [Q: Is there a <command>mysql-test-run</command> option that can
- be used to cause each test run to use a separate var directory?]
- </para>
+ <para>
+ [Q: Isn't the first sentence inaccurate? For example, the
+ result also contains output from <literal>echo</literal>
+ commands, and <literal>exec</literal> (but apparently not from
+ <literal>system</literal>)?]
+ </para>
- </section>
+ <para>
+ The test engine by default only output things from
+ <literal>select</literal>, <literal>show</literal>, and other
+ SQL statements that you expect to produce output (that is,
+ statements that create a result set). But it can be instructed
+ to be more or less verbose, including other data as well.
+ </para>
- <section id="tutorial-controlling-information">
+ <para>
+ Suppose that we want to include in the result the number of
+ rows affected by or returned by SQL statements. To do this,
+ add the following line to the case case file preceding those
+ statements:
+ </para>
- <title>Controlling the Amount of Information Produced by a Test Case</title>
-
- <section id="tutorial-more-information">
-
- <title>Getting More Information for Checking the Test Result</title>
-
- <para>
- [Q: Isn't the first sentence inaccurate? For example, the result
- also contains output from <literal>echo</literal> commands, and
- <literal>exec</literal> (but apparently not from
- <literal>system</literal>)?]
- </para>
-
- <para>
- The test engine by default only output things from
- <literal>select</literal>, <literal>show</literal>, and other
- SQL statements that you expect to produce output (that is,
- statements that create a result set). But it can be instructed
- to be more or less verbose, including other data as well.
- </para>
-
- <para>
- Suppose that we want to include in the result the number of rows
- affected by or returned by SQL statements. To do this, add the
- following line to the case case file preceding those statements:
- </para>
-
<programlisting>
--enable_info
</programlisting>
- <para>
- After rerunning the test by invoking
- <command>mysql-test-run</command> with the
- <option>--record</option> option to record the new result, the
- result file will contain more information:
- </para>
+ <para>
+ After rerunning the test by invoking
+ <command>mysql-test-run</command> with the
+ <option>--record</option> option to record the new result, the
+ result file will contain more information:
+ </para>
<programlisting>
drop table if exists t1,t2;
@@ -821,106 +830,107 @@
affected rows: 0
</programlisting>
- <para>
- To turn off the affected-rows reporting, add this command to the
- test file:
- </para>
+ <para>
+ To turn off the affected-rows reporting, add this command to
+ the test file:
+ </para>
<programlisting>
--disable_info
</programlisting>
- <para>
- Options can in general be enabled and disabled for different
- parts of the test file. Suppose that we are interested in the
- internals of the database as well. We can then enable the
- display of query metadata using
- <literal>enable_metadata</literal>. With this option enabled,
- the test output is a bit verbose. However, as mentioned earlier,
- the option can be enabled and disabled selectively so that it is
- enabled only for those parts of the test case where it interests
- you to know more.
- </para>
+ <para>
+ Options can in general be enabled and disabled for different
+ parts of the test file. Suppose that we are interested in the
+ internals of the database as well. We can then enable the
+ display of query metadata using
+ <literal>enable_metadata</literal>. With this option enabled,
+ the test output is a bit verbose. However, as mentioned
+ earlier, the option can be enabled and disabled selectively so
+ that it is enabled only for those parts of the test case where
+ it interests you to know more.
+ </para>
- </section>
+ </section>
- <section id="tutorial-less-information">
+ <section id="tutorial-less-information">
- <title>Reducing the Amount of Test Case Information</title>
+ <title>Reducing the Amount of Test Case Information</title>
- <para>
- [Q: is the following paragraph inaccurate? Don't the _query_log
- commands apply only to logging SQL statements (and not to "all
- commands in general")?]
- </para>
+ <para>
+ [Q: is the following paragraph inaccurate? Don't the
+ _query_log commands apply only to logging SQL statements (and
+ not to "all commands in general")?]
+ </para>
- <para>
- You might be initializing a table, where we don't really expect
- a failure. You can then temporarily disable the recording of the
- command lines executed with <literal>disable_query_log</literal>
- and then enable the recording again with
- <literal>enable_query_log</literal>. You can disable the
- recording of the output from executing commands using
- <literal>disable_result_log</literal> and enable again with
- <literal>enable_result_log</literal>.
- </para>
+ <para>
+ You might be initializing a table, where we don't really
+ expect a failure. You can then temporarily disable the
+ recording of the command lines executed with
+ <literal>disable_query_log</literal> and then enable the
+ recording again with <literal>enable_query_log</literal>. You
+ can disable the recording of the output from executing
+ commands using <literal>disable_result_log</literal> and
+ enable again with <literal>enable_result_log</literal>.
+ </para>
+ </section>
+
</section>
- </section>
+ <section id="tutorial-output-postprocessing">
- <section id="tutorial-output-postprocessing">
+ <title>Dealing with Output That Varies Per Test Run</title>
- <title>Dealing with Output That Varies Per Test Run</title>
+ <para>
+ It is best to write each test case so that the result it
+ produces does not vary for each test run, or according to
+ factors such as the time of day, differences in how program
+ binaries are compiled, the operating system, and so forth. For
+ example, if the result contains the current date and time, the
+ test engine has no way of verifying that it is correct.
+ </para>
- <para>
- It is best to write each test case so that the result it produces
- does not vary for each test run, or according to factors such as
- the time of day, differences in how program binaries are compiled,
- the operating system, and so forth. For example, if the result
- contains the current date and time, the test engine has no way of
- verifying that it is correct.
- </para>
+ <para>
+ However, sometimes a test result is inherently variable
+ according to external factors, or perhaps there is a part of a
+ result that you simply do not care about.
+ <command>mysqltest</command> provides commands that enable you
+ to postprocess test output to a more standard form. That way,
+ output variation across test runs will not trigger a result
+ mismatch
+ </para>
- <para>
- However, sometimes a test result is inherently variable according
- to external factors, or perhaps there is a part of a result that
- you simply do not care about. <command>mysqltest</command>
- provides commands that enable you to postprocess test output to a
- more standard form. That way, output variation across test runs
- will not trigger a result mismatch
- </para>
+ <para>
+ One such command is <literal>replace_column</literal>, which
+ specifies that you want to replace whatever is in a given column
+ with a string. This makes the output for that column the same
+ for each test run.
+ </para>
- <para>
- One such command is <literal>replace_column</literal>, which
- specifies that you want to replace whatever is in a given column
- with a string. This makes the output for that column the same for
- each test run.
- </para>
+ <para>
+ To see how this command works, add the following row after the
+ first insert:
+ </para>
- <para>
- To see how this command works, add the following row after the
- first insert:
- </para>
-
<programlisting>
insert into t1 values (date_format(now(), '%s'),9999);
</programlisting>
- <para>
- Then record the test result and run the test again:
- </para>
+ <para>
+ Then record the test result and run the test again:
+ </para>
<programlisting>
shell> <userinput>./mysql-test-run --record foo</userinput>
shell> <userinput>./mysql-test-run foo</userinput>
</programlisting>
- <para>
- Most likely, a failure will occur and
- <command>mysql-test-run</command> will display the difference
- between the expected result what we actually got, like this:
- </para>
+ <para>
+ Most likely, a failure will occur and
+ <command>mysql-test-run</command> will display the difference
+ between the expected result what we actually got, like this:
+ </para>
<programlisting>
Below are the diffs between actual and expected results:
@@ -963,13 +973,13 @@
-------------------------------------------------------
</programlisting>
- <para>
- If we are not really interested in the first column, one way to
- eliminate this mismatch is by using the
- <literal>replace_column</literal> command. The duration of the
- effect of this command is the next SQL statement, so we need one
- before each select statement: this:
- </para>
+ <para>
+ If we are not really interested in the first column, one way to
+ eliminate this mismatch is by using the
+ <literal>replace_column</literal> command. The duration of the
+ effect of this command is the next SQL statement, so we need one
+ before each select statement: this:
+ </para>
<programlisting>
--replace_column 1 SECONDS
@@ -980,12 +990,12 @@
select t1.* from t1;
</programlisting>
- <para>
- Note that <literal>SECONDS</literal> could be any string. Its
- purpose is only to map variable output onto a constant value. If
- we record the test result again, we will succeed each time we run
- the test after that. The result file will look like this:
- </para>
+ <para>
+ Note that <literal>SECONDS</literal> could be any string. Its
+ purpose is only to map variable output onto a constant value. If
+ we record the test result again, we will succeed each time we
+ run the test after that. The result file will look like this:
+ </para>
<programlisting>
drop table if exists t1,t2;
@@ -1015,51 +1025,53 @@
affected rows: 2
</programlisting>
- <para>
- [Q: What does the following paragraph add that has not already
- been stated?]
- </para>
+ <para>
+ [Q: What does the following paragraph add that has not already
+ been stated?]
+ </para>
- <para>
- If the result may vary, but just between some known values, you
- can use <literal>replace_result</literal> to match the different
- results, and replace them with a common string, making the result
- file the same even if the result varies some.
- </para>
+ <para>
+ If the result may vary, but just between some known values, you
+ can use <literal>replace_result</literal> to match the different
+ results, and replace them with a common string, making the
+ result file the same even if the result varies some.
+ </para>
- </section>
+ </section>
- <section id="tutorial-server-options">
+ <section id="tutorial-server-options">
- <title>Setting Server Options</title>
+ <title>Setting Server Options</title>
- <para>
- Many server options can be set from within the test case using
- statements such as these:
- </para>
+ <para>
+ Many server options can be set from within the test case using
+ statements such as these:
+ </para>
<programlisting>
set sql_warnings=1;
set sql_mode=no_auto_value_on_zero;
</programlisting>
- <para>
- But sometimes you need to restart the server to use some options
- in the form of extra command line arguments. The test engine will
- restart the server just before running a test case, if there
- exists a file
- <filename>mysql-test/t/<replaceable>test_name</replaceable>-master.opt</filename>
- with the options that you want to use. After the test case is
- completed, the server will be restarted again, with the default
- arguments.
- </para>
+ <para>
+ But sometimes you need to restart the server to use some options
+ in the form of extra command line arguments. The test engine
+ will restart the server just before running a test case, if
+ there exists a file
+ <filename>mysql-test/t/<replaceable>test_name</replaceable>-master.opt</filename>
+ with the options that you want to use. After the test case is
+ completed, the server will be restarted again, with the default
+ arguments.
+ </para>
- <para>
- [Q: Is that correct, or does <command>mysql-test-run</command>
- restart the server only when it notices that a test case requires
- different options than the previous test case?]
- </para>
+ <para>
+ [Q: Is that correct, or does <command>mysql-test-run</command>
+ restart the server only when it notices that a test case
+ requires different options than the previous test case?]
+ </para>
+ </section>
+
</section>
</chapter>
| Thread |
|---|
| • svn commit - mysqldoc@docsrva: r1582 - in trunk: . mysqltest | paul | 14 Mar |
