Author: paul
Date: 2006-03-13 22:36:07 +0100 (Mon, 13 Mar 2006)
New Revision: 1569
Log:
r8640@frost: paul | 2006-03-13 15:35:53 -0600
General revisions.
Modified:
trunk/
trunk/mysqltest/command-reference.xml
trunk/mysqltest/introduction.xml
trunk/mysqltest/mysqltest.xml
trunk/mysqltest/tutorial.xml
Property changes on: trunk
___________________________________________________________________
Name: svk:merge
- b5ec3a16-e900-0410-9ad2-d183a3acac99:/mysqldoc-local/mysqldoc/trunk:8638
bf112a9c-6c03-0410-a055-ad865cd57414:/mysqldoc-local/mysqldoc/trunk:3817
+ b5ec3a16-e900-0410-9ad2-d183a3acac99:/mysqldoc-local/mysqldoc/trunk:8640
bf112a9c-6c03-0410-a055-ad865cd57414:/mysqldoc-local/mysqldoc/trunk:3817
Modified: trunk/mysqltest/command-reference.xml
===================================================================
--- trunk/mysqltest/command-reference.xml 2006-03-13 21:28:12 UTC (rev 1568)
+++ trunk/mysqltest/command-reference.xml 2006-03-13 21:36:07 UTC (rev 1569)
@@ -17,8 +17,8 @@
<listitem>
<para>
- Make the distinction between <quote>command</quote> and
- <quote>statement/query</quote>.
+ Make the distinction between <quote><command>mysqltest</command>
+ command</quote> and <quote>SQL statement/query</quote>.
</para>
</listitem>
@@ -299,10 +299,11 @@
<section id="mysqltest-commands">
- <title><command>mysqltest</command> Command Reference</title>
+ <title><command>mysqltest</command> Commands</title>
<para>
- <command>mysqltest</command> supports the following commands:
+ <command>mysqltest</command> supports the following commands.
+ Command names are not case sensitive.
</para>
<itemizedlist>
@@ -334,8 +335,8 @@
<para>
[Q: To omit an argument, just leave it blank? What value is
- used then? Same as whatever the default is for
- <literal>mysql_real_connect()</literal>?]
+ used then? Same as whatever the default is for the
+ <literal>mysql_real_connect()</literal> C API function?]
</para>
<para>
@@ -424,16 +425,16 @@
</para>
<para>
- Syntax oddities: There must be whitespace between
+ [Syntax oddities: There must be whitespace between
<command>connect</command> and the opening parenthesis, and no
- whitepace after the opening parenthesis.
+ whitepace after the opening parenthesis.]
</para>
<para>
- Bug: leaving off the trailing parenthesis can produce a weird
+ [BUG: leaving off the trailing parenthesis can produce a weird
error. For example, with no paren after the database name,
<command>mysqltest</command> says <literal>mysqltest: At line
- 1: Illegal argument for port: 'h4z'</literal>.
+ 1: Illegal argument for port: 'h4z'</literal>.]
</para>
</listitem>
@@ -765,19 +766,32 @@
MySQL-specific error number or a SQLSTATE value. (These are
the kinds of values returned by the
<literal>mysql_errno()</literal> and
- <literal>mysql_sqlstate()</literal> C API functions.) If you
- specify a SQLSTATE value, it should begin with an
+ <literal>mysql_sqlstate()</literal> C API functions,
+ respectively.)
+ </para>
+
+ <para>
+ If you specify a SQLSTATE value, it should begin with an
<literal>S</literal> to enable <command>mysqltest</command> to
- distinguish it from a MySQL error number. For example, to
- specify the SQLSTATE value <literal>42000</literal>, write it
- as <literal>S42000</literal>:
+ distinguish it from a MySQL error number. For example, the
+ error number 1050 and the SQLSTATE value
+ <literal>42S01</literal> are equivalent, so the following
+ commands specify the same expected error:
</para>
<programlisting>
---error S42000
+--error 1050
+--error S42S01
</programlisting>
<para>
+ [Q: There is not a strict 1−1 relationship between error
+ numbers and SQLSTATE values (some SQLSTATE values are used for
+ more than one error number). Is there some implication for
+ error testing that we should point out here?]
+ </para>
+
+ <para>
[Wiki has some stuff about non-obvious effects for specifying
error 0 in combination with non-zero error codes]
</para>
@@ -968,6 +982,13 @@
Like <literal>eval</literal>, but display the query result
horizontally.
</para>
+
+ <para>
+ [Q: I don't think that's true. <literal>eval</literal> gives
+ you variable expansion, I think
+ <literal>query_horizontal</literal> does not? If there is
+ supposed to be variable expansion, I think there's a bug.]
+ </para>
</listitem>
<listitem>
@@ -979,6 +1000,13 @@
Like <literal>eval</literal>, but display the query result
vertically.
</para>
+
+ <para>
+ [Q: I don't think that's true. <literal>eval</literal> gives
+ you variable expansion, I think
+ <literal>query_vertical</literal> does not? If there is
+ supposed to be variable expansion, I think there's a bug.]
+ </para>
</listitem>
<listitem>
Modified: trunk/mysqltest/introduction.xml
===================================================================
--- trunk/mysqltest/introduction.xml 2006-03-13 21:28:12 UTC (rev 1568)
+++ trunk/mysqltest/introduction.xml 2006-03-13 21:36:07 UTC (rev 1569)
@@ -10,53 +10,134 @@
<title>Introduction</title>
<para>
- Note: The <command>mysql-test-run</command> shell script is being
- replaced by the <command>mysql-test-run.pl</command> Perl script.
- Need to account for that in this document.
+ MySQL distributions include a set of test cases and programs for
+ running them. These tools constitute the MySQL test framework that
+ provideas a means for verifying that MySQL Server and its client
+ programs operate according to expectations. The test cases consist
+ mostly of SQL statements, but can also use test language constructs
+ that control how to run tests and verify their results.
</para>
<para>
- MySQL distributions include a set of test files and programs for
- running them. These tools provide a means for verifying that MySQL
- Server and its client programs operate according to expectations.
+ This document describes the components of the MySQL test framework,
+ how the test programs work, and the language used for writing test
+ cases. It also provides a tutorial for writing and running test
+ cases.
</para>
<para>
- The programs are <command>mysqltest</command>, which runs a test
- case, and <command>mysql-test-run</command>, which invokes
- <command>mysqltest</command> once for each case. Input for each test
- case is stored in a file, and the expected result from running the
- test is stored in another file. The expected result can be compared
- to the actual result from running a test to verify proper processing
- of the input by MySQL. Test case and result files are stored in the
- <filename>t</filename> and <filename>r</filename> subdirectories,
- respectively, of the <filename>mysql-test</filename> directory.
+ The application that runs the test suite is named
+ <command>mysql-test-run</command>. Its location is the
+ <filename>mysql-test</filename> directory, which is present both in
+ source and binary MySQL Server distributions. (Note: The
+ <command>mysql-test-run</command> shell script is being replaced by
+ the <command>mysql-test-run.pl</command> Perl script.)
+ <command>mysql-test-run</command> invokes
+ <command>mysqltest</command> once for each test case.
</para>
<para>
- For a MySQL source distribution, <command>mysqltest</command> is
- located in the <filename>client</filename> directory.
- <command>mysql-test-run</command> and the test input and result
- files are located in the <filename>mysql-test</filename> directory.
- The <filename>client</filename> and <filename>mysql-test</filename>
+ Input for each test case is stored in a file, and the expected
+ result from running the test is stored in another file. The expected
+ result can be compared to the actual result from running a test to
+ verify proper processing of the input by MySQL.
+ </para>
+
+ <para>
+ For a MySQL source distribution, <command>mysql-test-run</command>
+ is located in the <filename>mysql-test</filename> directory.
+ <command>mysqltest</command> is located in the
+ <filename>client</filename> directory. The
+ <filename>mysql-test</filename> and <filename>client</filename>
directories are located in the root directory of the distribution.
</para>
<para>
- For a MySQL binary distribution, <command>mysqltest</command> is
- located in the same directory where other client programs such as
- <command>mysql</command> or <command>mysqladmin</command> are
- installed. <command>mysql-test-run</command> and the test input and
- result files are located in the <filename>mysql-test</filename>
- directory. The locations of the <filename>client</filename> and
- <filename>mysql-test</filename> directories depend on the layout
- used for the distribution format.
+ For a MySQL binary distribution, <command>mysql-test-run</command>
+ is located in the <filename>mysql-test</filename> directory.
+ <command>mysqltest</command> is located in the same directory where
+ other client programs such as <command>mysql</command> or
+ <command>mysqladmin</command> are installed. The locations of the
+ <filename>mysql-test</filename> and <filename>client</filename>
+ directories depend on the layout used for the distribution format.
</para>
<para>
- This document describes the layout of the
- <filename>mysql-test</filename> directory, how the test programs
- work, and the language used for writing test cases.
+ Test case input files and result files are stored in the
+ <filename>mysql-test/t</filename> and
+ <filename>mysql-test/r</filename> directories, respectively. The
+ input and result files have the same basename, which is the test
+ name, but have extensions of <filename>.test</filename> and
+ <filename>.result</filename>, respectively.
</para>
+ <para>
+ Each test file is referred to as one test case, but usually consists
+ of a sequence of related tests. An unexpected failure of a single
+ statement in a test case makes the test fail.
+ </para>
+
+ <para>
+ The <command>mysql-test-run</command> application starts MySQL
+ servers, restarts them as necessary when a specific test case needs
+ different start arguments, and presents the test result. The
+ <command>mysql-test-run</command> application invokes the
+ <command>mysqltest</command> program to run a single test case,
+ intepret the test language constructs, and send SQL statements to
+ the server. <command>mysqltest</command> is also referred to as the
+ <quote>test engine.</quote>
+ </para>
+
+ <para>
+ There are two ways a test case can fail:
+ </para>
+
+ <itemizedlist>
+
+ <listitem>
+ <para>
+ The <command>mysqltest</command> test engine checks the result
+ codes from executing each SQL statement. If the failure is
+ unexpected, the test case fails.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The test case can fail by producing incorrect trace/protocol
+ output. As a test runs, it writes the result from
+ <literal>SELECT</literal>, <literal>SHOW</literal>, and other
+ statements to a file. This file is compared to the expected
+ trace found in the <filename>mysql-test/r</filename> directory
+ (in a file with a <filename>.result</filename> suffix). If the
+ files containing the expected and actual results differ, the
+ test case fails and the difference is presented for evaluation.
+ When test case failure occurs, the actual test result is written
+ to a file in the <filename>mysql-test/r</filename> directory
+ with a <filename>.reject</filename> suffix.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ <para>
+ This method of comparing the result trace puts some restrictions on
+ how test cases can be written. For example, you cannot have a result
+ that varies from run to run, such as the current time. However, if
+ the information that varies is unimportant for test evaluation,
+ there are ways to instruct the test engine to replace those fields
+ in the output with fixed values.
+ </para>
+
+ <para>
+ Because the test cases consist mostly of SQL statements in a text
+ file, there is no direct support for test cases that are written in
+ C, Java, or other languages. Such tests are not within the scope of
+ this test framework. But the framework does support executing your
+ own scripts and initiating them with your own data. Also, a test
+ case can execute an external program, so in some respects, the test
+ framework can be extended for uses other than testing only SQL
+ statements.
+ </para>
+
</chapter>
Modified: trunk/mysqltest/mysqltest.xml
===================================================================
--- trunk/mysqltest/mysqltest.xml 2006-03-13 21:28:12 UTC (rev 1568)
+++ trunk/mysqltest/mysqltest.xml 2006-03-13 21:36:07 UTC (rev 1569)
@@ -50,6 +50,15 @@
<title>Preface</title>
<para>
+ MySQL distributions include a set of test cases and programs for
+ running them. These tools constitute the MySQL test framework that
+ provideas a means for verifying that MySQL Server and its client
+ programs operate according to expectations. The test cases consist
+ mostly of SQL statements, but can also use test language
+ constructs that control how to run tests and verify their results.
+ </para>
+
+ <para>
This manual describes the MySQL test framework. It describes the
programs used to run tests and the language used to write test
cases.
@@ -63,11 +72,72 @@
<xi:include href="introduction.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
- <chapter id="mysql-test-layout">
+ <xi:include href="tutorial.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
- <title>Layout of the <filename>mysql-test</filename> Directory</title>
+ <chapter id="test-framework-components">
+ <title>MySQL Test Framework Components</title>
+
<para>
+ The MySQL test framework includes programs that run tests, and
+ directories and files used by those programs.
+ </para>
+
+ <para>
+ <emphasis role="bold">Test Framework Programs</emphasis>
+ </para>
+
+ <para>
+ The MySQL test framework uses several programs:
+ </para>
+
+ <itemizedlist>
+
+ <listitem>
+ <para>
+ The <command>mysql-test-run</command> script is used to run
+ the test suite.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The <command>mysql-test-run.pl</command> script is currently
+ in development (and eventually will replace
+ <command>mysql-test-run</command>).
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ <command>mysqltest</command> runs a test case.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The <command>mysql-stress-test.pl</command> script is used for
+ stress-testing. (MySQL 5.0 and up only)
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ <para>
+ Except for <command>mysqltest</command>, these programs are
+ located in the <filename>mysql-test</filename> directory.
+ <command>mysqltest</command> is located elsewhere. For a source
+ distribution, <command>mysqltest</command> is in the
+ <filename>client</filename> directory. For a binary distribution,
+ it is in the MySQL <filename>bin</filename> directory.
+ </para>
+
+ <para>
+ <emphasis role="bold">Test Framework Directories and
+ Files</emphasis>
+ </para>
+
+ <para>
The test suite is located in the <filename>mysql-test</filename>
directory. For a source distribution,
<filename>mysql-test</filename> is found under the source tree
@@ -85,43 +155,10 @@
<listitem>
<para>
- Programs that are used for running tests:
+ The <command>mysql-test-run</command> and
+ <command>mysql-stress-test.pl</command> programs that are used
+ for running tests.
</para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- The <command>mysql-test-run</command> script that is used
- to run the test suite.
- </para>
- </listitem>
-
- <listitem>
- <para>
- The <command>mysql-test-run.pl</command> script that is
- currently in development (and that eventually will replace
- <command>mysql-test-run</command>).
- </para>
- </listitem>
-
- <listitem>
- <para>
- The <command>mysql-stress-test.pl</command> stress-testing
- script. (MySQL 5.0 and up only)
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- Note: The <command>mysqltest</command> program is used by
- <command>mysql-test-run</command> but is located elsewhere.
- For a source distribution, <command>mysqltest</command> is in
- the <filename>client</filename> directory. For a binary
- distribution, it is in the MySQL <filename>bin</filename>
- directory.
- </para>
</listitem>
<listitem>
@@ -173,8 +210,12 @@
<para>
A filename of the form
<filename><replaceable>test_name</replaceable>-im.opt</filename>
- [Q: is for ...?]
+ ...
</para>
+
+ <para>
+ [Q: is for what ...?]
+ </para>
</listitem>
</itemizedlist>
@@ -214,10 +255,18 @@
</para>
<para>
- [Q: If a .reject file is created because a test fails,
- does the file get removed automatically later the next
- time the test succeeds?]
+ [Q: If a <filename>.reject</filename> file is created
+ because a test fails, does the file get removed
+ automatically later the next time the test succeeds?]
</para>
+
+ <para>
+ [Q: <command>mysql-test-run</command> refers to
+ <filename>.eval</filename> files. What are these for? Do
+ they result from the <literal>eval_result</literal>
+ command and if they exist they are used instead of the
+ corresponding <filename>.result</filename> files?
+ </para>
</listitem>
</itemizedlist>
@@ -266,6 +315,10 @@
</itemizedlist>
<para>
+ <emphasis role="bold">Test Execution and Evaluation</emphasis>
+ </para>
+
+ <para>
[The following stuff would be better for an overview section?]
</para>
@@ -391,8 +444,9 @@
</itemizedlist>
<para>
- [Q: <command>mysqltest</command> does not write the
- <filename>.reject</filename> files?
+ [Q: Which program writes the <filename>.reject</filename> files?
+ <command>mysql-test-run</command> or
+ <command>mysqltest</command>?]
</para>
<para>
@@ -406,6 +460,12 @@
and write output.
</para>
+ <para>
+ [TODO: would be useful to have an example command line here
+ showning how <command>mysql-test-run</command> invokes
+ <command>mysqltest</command>.]
+ </para>
+
</chapter>
<chapter id="test-programs">
@@ -512,7 +572,7 @@
</para>
<para>
- Features:
+ Features of <command>mysqltest</command>:
</para>
<itemizedlist>
@@ -542,7 +602,9 @@
</listitem>
<listitem>
- <para></para>
+ <para>
+ Can execute external commands
+ </para>
</listitem>
</itemizedlist>
@@ -558,7 +620,7 @@
</programlisting>
<para>
- You can also name the test case with a
+ You can also name the test case file with a
<option>--test-file=<replaceable>file_name</replaceable></option>
option.
</para>
@@ -1099,7 +1161,7 @@
<para>
[Q: And does what with the result from the
comparison? Is this similar to using the
- <literal>result</literal> command?
+ <literal>result</literal> command?]
</para>
</listitem>
@@ -1232,10 +1294,15 @@
</para>
<para>
- [Bug: <option>--sleep=0</option> has no effect. That is,
+ [BUG: <option>--sleep=0</option> has no effect. That is,
it doesn't cause a sleep of 0 seconds. Instead,
<literal>sleep</literal> commands in the test file sleep
- by the amount specified in the file.]
+ by the amount specified in the file. Probably we could
+ fix this by setting <literal>opt_sleep</literal> to -1
+ (not 0) and using that as the "no setting given" value,
+ and preventing negative values from being given on the
+ command line by using GET_UINT (not GET_INT) in the
+ option structure.]
</para>
</listitem>
@@ -1509,13 +1576,18 @@
</programlisting>
<para>
+ Each <replaceable>test_name</replaceable> argument names a
+ test case. The test case file corresponding to the test name
+ is
+ <filename>t/<replaceable>test_name</replaceable>.test</filename>.
+ </para>
+
+ <para>
For each <replaceable>test_name</replaceable> argument,
- <command>mysql-test-run</command> runs [Q: the test found in
- the
- <filename>t/<replaceable>test_name</replaceable>.test</filename>
- file?] With no <replaceable>test_name</replaceable>
- arguments, <command>mysql-test-run</command> [Q: does what?
- Runs all <filename>.test</filename> files in the
+ <command>mysql-test-run</command> runs the named test case.
+ With no <replaceable>test_name</replaceable> arguments,
+ <command>mysql-test-run</command> [Q: does what? Runs all
+ <filename>.test</filename> files in the
<filename>t</filename> subdirectory?]
</para>
@@ -1625,10 +1697,12 @@
</para>
<para>
- <emphasis role="bold">Note</emphasis>: In fact,
- <command>mysql-test-run</command> does not have a
- <option>--help</option> option, which is a deficiency.
- (Corrected by <command>mysql-test-run.pl</command>.)
+ [BUG: In fact, <command>mysql-test-run</command> does
+ not have a <option>--help</option> option. That is a
+ deficiency that is corrected in
+ <command>mysql-test-run.pl</command>, but there is no
+ runtime command for <command>mysql-test-run</command> to
+ discover its options.]
</para>
</listitem>
@@ -3611,8 +3685,6 @@
<xi:include href="command-reference.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
- <xi:include href="tutorial.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
-
<chapter id="mysqltest-misc">
<title><command>mysqltest</command> Misc. Stuff</title>
Modified: trunk/mysqltest/tutorial.xml
===================================================================
--- trunk/mysqltest/tutorial.xml 2006-03-13 21:28:12 UTC (rev 1568)
+++ trunk/mysqltest/tutorial.xml 2006-03-13 21:36:07 UTC (rev 1569)
@@ -17,102 +17,6 @@
[TODO: Add tutorial introductory text here]
</para>
- <section id="test-framework">
-
- <title>The MySQL Test Framework</title>
-
- <para>
- [TODO: Merge this into main document introduction]
- </para>
-
- <para>
- The MySQL test framework is used primarily to run the MySQL Server
- regression suite. The test cases consist mostly of SQL statements,
- but can also use test language constructs that control how to run
- tests and verify their results.
- </para>
-
- <para>
- The application that runs the test suite is named
- <command>mysql-test-run</command>. Its location is the
- <filename>mysql-test</filename> directory, which is present both
- in source and binary MySQL Server distributions.
- </para>
-
- <para>
- The test cases are located in the
- <filename>mysql-test/t</filename> directory, each in a file having
- a suffix of <filename>.text</filename>. Each test file is referred
- to as one test case, but usually consists of a sequence of related
- tests. An unexpected failure of a single statement in a test case
- makes the test fail.
- </para>
-
- <para>
- The <command>mysql-test-run</command> application starts MySQL
- servers, restarts them as necessary when a specific test case
- needs different start arguments, and presents the test result. The
- <command>mysql-test-run</command> application invokes the
- <command>mysqltest</command> program to run a single test case,
- intepret the test language constructs, and send SQL statements to
- the server. <command>mysqltest</command> is also referred to as
- the <quote>test engine.</quote>
- </para>
-
- <para>
- There are two ways a test case can fail:
- </para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- The <command>mysqltest</command> test engine checks the result
- codes from executing each SQL statement. If the failure is
- unexpected, the test case fails.
- </para>
- </listitem>
-
- <listitem>
- <para>
- The test case can fail by producing incorrect trace/protocol
- output. When running the test, it writes the result from
- <literal>SHOW</literal> and other statements to a file. This
- file is compared to the expected trace found in the
- <filename>mysql-test/r</filename> directory (in a file with a
- <filename>.result</filename> suffix). If the two files
- containing the expected and actual results differ, the test
- case fails and the difference is presented for evaluation.
- When test case failure occurs, the actual test result is
- written to a file in the <filename>mysql-test/r</filename>
- directory with a <filename>.reject</filename> suffix.
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- This method of comparing the result trace puts some restrictions
- on how test cases can be written. For example, you cannot have a
- result that varies from run to run, such as the current time.
- However, if the information that varies is unimportant for test
- evaluation, there are ways to instruct the test engine to replace
- those fields in the output with fixed values.
- </para>
-
- <para>
- Because the test cases consist mostly of SQL statements in a text
- file, there is no direct support for test cases that are written
- in C, Java, or other languages. Such tests are not within the
- scope of this test framework. But the framework does support
- executing your own scripts and initiating them with your own data.
- Also, a test case can execute an external program, so in some
- respects the test framework can be extended for uses other than
- testing only SQL statements.
- </para>
-
- </section>
-
<section id="tutorial-test-case-basics">
<title>Test Cases Basics</title>
| Thread |
|---|
| • svn commit - mysqldoc@docsrva: r1569 - in trunk: . mysqltest | paul | 13 Mar |
