Author: paul
Date: 2006-03-11 01:21:34 +0100 (Sat, 11 Mar 2006)
New Revision: 1556
Log:
r8570@frost: paul | 2006-03-10 18:20:13 -0600
Rearrange tutorial material.
Add section id attributes needed to make .info builds work.
Re-enable Makefile .info machinery.
Fix Makefile dependency.
Modified:
trunk/
trunk/mysqltest/Makefile
trunk/mysqltest/mysqltest.xml
Property changes on: trunk
___________________________________________________________________
Name: svk:merge
- b5ec3a16-e900-0410-9ad2-d183a3acac99:/mysqldoc-local/mysqldoc/trunk:8569
bf112a9c-6c03-0410-a055-ad865cd57414:/mysqldoc-local/mysqldoc/trunk:3817
+ b5ec3a16-e900-0410-9ad2-d183a3acac99:/mysqldoc-local/mysqldoc/trunk:8570
bf112a9c-6c03-0410-a055-ad865cd57414:/mysqldoc-local/mysqldoc/trunk:3817
Modified: trunk/mysqltest/Makefile
===================================================================
--- trunk/mysqltest/Makefile 2006-03-11 00:21:07 UTC (rev 1555)
+++ trunk/mysqltest/Makefile 2006-03-11 00:21:34 UTC (rev 1556)
@@ -1,4 +1,4 @@
-# Makefile for MySQL test platform manual
+# Makefile for MySQL test framework manual
# Formats we need for docsrva
# chm
@@ -6,6 +6,7 @@
# html-chapter Done
# html-section Done
# html-web Done
+# info Done
# pdf-a4 Done
# pdf Done
# txt Done (suboptimal)
@@ -42,9 +43,9 @@
MYSQLTEST_SRCS_EXTRA = fixedchars.ent
-MYSQLTEST_SRCS = $(MYSQLTEST_SRCS_EXTRA) mysqltest.xml
+MYSQLTEST_SRCS = $(MYSQLTEST_SRCS_EXTRA) mysqltest.xml introduction.xml command-reference.xml tutorial.xml
-mysqltest-prepped.xml = $(MYSQLTEST_SRCS)
+mysqltest-prepped.xml: $(MYSQLTEST_SRCS)
#%.txt: %.html
# lynx -dump $< > $@
@@ -67,6 +68,6 @@
include $(MAKE_DIR)/xml-pdf
include $(MAKE_DIR)/xml-toc
include $(MAKE_DIR)/xml-txt
-#include $(MAKE_DIR)/xml-texi
-#include $(MAKE_DIR)/texi-info
+include $(MAKE_DIR)/xml-texi
+include $(MAKE_DIR)/texi-info
include $(MAKE_DIR)/xml-depend
Modified: trunk/mysqltest/mysqltest.xml
===================================================================
--- trunk/mysqltest/mysqltest.xml 2006-03-11 00:21:07 UTC (rev 1555)
+++ trunk/mysqltest/mysqltest.xml 2006-03-11 00:21:34 UTC (rev 1556)
@@ -16,14 +16,14 @@
]>
<book lang="en">
- <title>The MySQL Test Platform</title>
+ <title>The MySQL Test Framework</title>
<bookinfo>
<abstract>
<para>
- This manual describes the MySQL test platform.
+ This manual describes the MySQL test framework.
</para>
<para>
@@ -50,7 +50,7 @@
<title>Preface</title>
<para>
- This manual describes the MySQL test platform. It describes the
+ This manual describes the MySQL test framework. It describes the
programs used to run tests and the language used to write test
cases.
</para>
@@ -61,64 +61,8 @@
</preface>
- <chapter id="introduction">
+ <xi:include href="introduction.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
- <title>Introduction</title>
-
- <para>
- Note: The <command>mysql-test-run</command> shell script is being
- replaced by the <command>mysql-test-run.pl</command> Perl script.
- Need to account for that in this document.
- </para>
-
- <para>
- MySQL distributions include a set of test files and programs for
- running them. These tools provide a means for verifying that MySQL
- Server and its client programs operate according to expectations.
- </para>
-
- <para>
- The programs are <command>mysqltest</command>, which runs a test
- case, and <command>mysql-test-run</command>, which invokes
- <command>mysqltest</command> once for each case. Input for each
- test case is stored in a file, and the expected result from
- running the test is stored in another file. The expected result
- can be compared to the actual result from running a test to verify
- proper processing of the input by MySQL. Test case and result
- files are stored in the <filename>t</filename> and
- <filename>r</filename> subdirectories, respectively, of the
- <filename>mysql-test</filename> directory.
- </para>
-
- <para>
- For a MySQL source distribution, <command>mysqltest</command> is
- located in the <filename>client</filename> directory.
- <command>mysql-test-run</command> and the test input and result
- files are located in the <filename>mysql-test</filename>
- directory. The <filename>client</filename> and
- <filename>mysql-test</filename> directories are located in the
- root directory of the distribution.
- </para>
-
- <para>
- For a MySQL binary distribution, <command>mysqltest</command> is
- located in the same directory where other client programs such as
- <command>mysql</command> or <command>mysqladmin</command> are
- installed. <command>mysql-test-run</command> and the test input
- and result files are located in the
- <filename>mysql-test</filename> directory. The locations of the
- <filename>client</filename> and <filename>mysql-test</filename>
- directories depend on the layout used for the distribution format.
- </para>
-
- <para>
- This document describes the layout of the
- <filename>mysql-test</filename> directory, how the test programs
- work, and the language used for writing test cases.
- </para>
-
- </chapter>
-
<chapter id="mysql-test-layout">
<title>Layout of the <filename>mysql-test</filename> Directory</title>
@@ -478,7 +422,7 @@
<para>
This chapter describes the test programs that run test cases. For
information about the language used for writing test cases, see
- <xref linkend="mysqltest-language"/>.
+ <xref linkend="mysqltest-reference"/>.
</para>
<para>
@@ -3665,2591 +3609,96 @@
</chapter>
- <chapter id="mysqltest-language">
+ <xi:include href="command-reference.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
- <title>The <command>mysqltest</command> Command Language</title>
+ <xi:include href="tutorial.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/>
- <para>
- To do:
- </para>
+ <chapter id="mysqltest-misc">
- <itemizedlist>
+ <title><command>mysqltest</command> Misc. Stuff</title>
- <listitem>
- <para>
- Make the distinction between <quote>command</quote> and
- <quote>statement/query</quote>.
- </para>
- </listitem>
+<para>
+[TODO: All this stuff needs to be moved somewhere else]
+</para>
- <listitem>
- <para>
- Verify the syntax of each command.
- </para>
- </listitem>
-
- <listitem>
- <para>
- Note: For now, the commands are in a single lexical list.
- Eventually, they should probably be categorized.
- </para>
- </listitem>
-
- </itemizedlist>
-
<para>
- This chapter describes the test language implemented by
- <command>mysqltest</command>. The language allows input to contain
- a mix of comments, commands executed by
- <command>mysqltest</command> itself, and SQL statements that
- <command>mysqltest</command> sends to a MySQL server for
- execution. Here is an example of input for
- <command>mysqltest</command>:
+ If an expected error is specified and that error occurs,
+ <command>mysqltest</command> continues reading input. If the
+ command is successful or a different error occurs,
+ <command>mysqltest</command> aborts.
</para>
-<programlisting>
-# this is a comment that is ignored
--- this is a comment because it does not begin with a command word
--- echo this is a mysqltest command because it begins with a command word
-select 'this is a SQL statement to be sent to the server';
-echo this is a mysqltest command because it begins with a command word;
-</programlisting>
-
<para>
- When <command>mysqltest</command> starts, it opens a connection
- named <literal>default</literal>, using any connection parameters
- specified by the command options. [Q: When run from
- <command>mysql-test-run</command>, the default username will be
- <literal>root</literal>?]
+ If no expected error is specified, <command>mysqltest</command>
+ aborts unless the command is successful. (It is implicit that you
+ expect <literal>$mysql_errno</literal> to be 0.)
</para>
- <para>
- You can use the <literal>connection</literal> command to open
- other connections, the <literal>connection</literal> command to
- switch between connections, and the <literal>disconnect</literal>
- command to close connections. The capability for switching
- connections which means that <literal>default</literal> need not
- be the connection in use at a given time. To avoid ambiguity, this
- document uses the term <quote>current connection</quote> to
- indicate the connection currently in use, which might be different
- from <quote>the connection named
- <literal>default</literal>.</quote>
- </para>
+ <section id="environment-overview">
- <section id="mysqltest-input-conventions">
+ <title>Test Environment Overview</title>
- <title><command>mysqltest</command> Input Conventions</title>
-
<para>
- <command>mysqltest</command> reads input lines and processes
- them as follows:
+ There are two ways the same test environment should handle. One
+ is to talk to an installed running server. The second case is to
+ be able to start a server from the binaries in the source tree,
+ i.e. an uninstalled MySQL server. Because of this in general you
+ don't start a server and create a connection from the test case,
+ this is done by the <command>mysql-test-run</command> script
+ before your test case is run.
</para>
- <itemizedlist>
-
- <listitem>
- <para>
- <quote>End of line</quote> means a newline (linefeed)
- character. A carriage return/linefeed (CRLF) pair also is
- allowable as as a line terminator (the carriage return is
- ignored), but carriage return by itself is
- <emphasis>not</emphasis> allowed as a line terminator..
- </para>
- </listitem>
-
- <listitem>
- <para>
- A line that begins with ‘<literal>#</literal>’
- as the first non-whitespace character is treated as a
- comment that extends to the end of the line and is ignored.
- Example:
- </para>
-
-<programlisting>
-# this is a comment
-</programlisting>
- </listitem>
-
- <listitem>
- <para>
- A line that begins with ‘<literal>--</literal>’
- as the first non-whitespace content also is treated as a
- comment that extends to the end of the line. However, unlike
- ‘<literal>#</literal>’ comments, if the first
- word of the comment is a valid <command>mysqltest</command>
- command, <command>mysqltest</command> executes the line from
- that word to the end of the line as a command. This syntax
- is useful for writing commands that contain embedded
- instances of the command delimiter. Examples:
- </para>
-
-<programlisting>
---disconnect conn1
--- echo write this text; it goes to the result file
-</programlisting>
- </listitem>
-
- <listitem>
- <para>
- Other input is taken as normal command input. The command
- extends to the next occurrence of the command delimiter,
- which is semicolon (‘<literal>;</literal>’) by
- default. The delimiter can be changed with the
- <literal>delimiter</literal> command.
- </para>
-
- <para>
- Because the command extends to the delimiter, a given input
- line can contain multiple commands, and a given command can
- span multiple lines. If the first line of the
- delimiter-terminated command begins with a word that is a
- <command>mysqltest</command> command,
- <command>mysqltest</command> executes it as such. Otherwise,
- <command>mysqltest</command> assumes that the command is an
- SQL statement and sends it to the MySQL server to be
- executed.
- </para>
-
- <para>
- After <command>mysqltest</command> reads a command up to a
- delimiter and executes it, input reading restarts following
- the delimiter and any following input on the line that
- contains the delimiter is treated as though it begins on a
- new line. Consider the following two input lines:
- </para>
-
-<programlisting>
-select 1; select 'hello'; select
-'goodbye';
-</programlisting>
-
- <para>
- Three separate commands result from that input:
- </para>
-
-<programlisting>
-select 1
-select 'hello'
-select 'goodbye'
-</programlisting>
-
- <para>
- Similarly, ‘<literal>#</literal>’ comments or
- ‘<literal>--</literal>’ comments can be given on
- a command line following a delimiter:
- </para>
-
-<programlisting>
-select 'hello'; # select a string value
-select 'hello'; -- echo some text
-</programlisting>
-
- <para>
- The ability to write multiple-line statements is useful for
- making long statements more readable, such as a
- <literal>create table</literal> statement for a table that
- has many columns.
- </para>
- </listitem>
-
- </itemizedlist>
-
<para>
- Note that ‘<literal>--</literal>’ comments and
- normal commands have complementary properties with regard to how
- <command>mysqltest</command> reads them:
+ FIXME add the most important options to use calling
+ <command>mysql-test-run</command>
</para>
- <itemizedlist>
-
- <listitem>
- <para>
- A ‘<literal>--</literal>’ comment is terminated
- by a newline, regardless of how many delimiters it contains.
- </para>
- </listitem>
-
- <listitem>
- <para>
- A normal command (without
- ‘<literal>--</literal>’) is terminated by the
- delimiter (semicolon), no matter how many newlines it
- contains.
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- <command>mysqltest</command> commands can be written either as
- comments (with a leading ‘<literal>--</literal>’) or
- as normal command input (no leading
- ‘<literal>--</literal>’). Use the command delimiter
- only in the latter case. Thus, these two lines are equivalent:
- </para>
-
-<programlisting>
---sleep 2
-sleep 2;
-</programlisting>
-
- <para>
- The equivalence is true even for the
- <literal>delimiter</literal> command. For example, to set the
- delimiter to <literal>//</literal>, either of these commands
- work:
- </para>
-
-<programlisting>
---delimiter //
-delimiter //;
-</programlisting>
-
- <para>
- To set the delimiter back to ;, use either of these commands:
- </para>
-
-<programlisting>
---delimiter ;
-delimiter ;//
-</programlisting>
-
- <para>
- [Ambiguity] Although it is <emphasis>possible</emphasis> to
- write a non-command comment that begins with
- ‘<literal>--</literal>’, it is better to use
- ‘<literal>#</literal>’ instead. For example, the
- line <literal>--switch to conn1</literal> is take as a comment
- currently, but if <command>mysqltest</command> is extended in
- the future to add a <literal>switch</literal> command, that line
- will be treated as a command instead. If you use
- ‘<literal>#</literal>’ for all comments, this
- problem will not occur.
- </para>
-
- <para>
- [Ambiguity] Because a non-comment line can contain either a
- <command>mysqltest</command> command or an SQL statement,
- presumably this means that no <command>mysqltest</command>
- command should be the same as any keyword that can begin an SQL
- statement? Presumably also this means that, should extensions to
- SQL be implemented in the future, it's possible that a new SQL
- keyword could be impossible for <command>mysqltest</command> to
- recognize as such if that keyword is already used as a
- <command>mysqltest</command> command.
- </para>
-
- <para>
- On a multiple-line command, ‘<literal>#</literal>’
- or ‘<literal>--</literal>’ at the beginning of the
- second or following lines is not special. Thus, the second and
- third lines of the following command are not taken as comments:
- </para>
-
-<programlisting>
-let $a = This is a variable
-# assignment that sets a variable
--- to a multiple-line value;
- </programlisting>
-
- <para>
- Describe quoting and escaping rules. How to include a literal
- quote? Double it? Escape it? Can you escape a delimiter as "\;"?
- But then what if the delimiter is multiple characters? (I guess
- that wouldn't matter. If the first char is not taken as a
- delimiter beginning, the rest wouldn't be recognized, either.
- Escaping appears to be allowable only in limited contexts?
- </para>
-
</section>
- <section id="mysqltest-commands">
+ <section id="language-overview">
- <title><command>mysqltest</command> Command Reference</title>
+ <title>Language Overview</title>
<para>
- <command>mysqltest</command> supports the following commands:
+ FIXME make descriptions shorter by introducing a syntax to
+ inform that an argument could be a test language variable. Why
+ not all?
</para>
- <itemizedlist>
-
- <listitem>
- <para>
- <literal>character_set
- <replaceable>charset_name</replaceable></literal>
- </para>
-
- <para>
- Set the default character set to
- <replaceable>charset_name</replaceable>. Initially, the
- character set is <literal>latin1</literal>.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>connect (<replaceable>name</replaceable>,
- <replaceable>host_name</replaceable>,
- <replaceable>user_name</replaceable>,
- <replaceable>password</replaceable>,
- <replaceable>db_name</replaceable>
- [,<replaceable>port_num</replaceable>
- [,<replaceable>socket</replaceable>
- [,<replaceable>options</replaceable>]]])</literal>
- </para>
-
- <para>
- [Q: To omit an argument, just leave it blank? What value is
- used then? Same as whatever the default is for
- <literal>mysql_real_connect()</literal>?]
- </para>
-
- <para>
- Open a connection to the server. [Q: and make the connection
- the current connection?]
- </para>
-
- <para>
- [Put a couple of examples here, including one with blank
- arguments]
- </para>
-
- <para>
- The arguments to <literal>connect</literal> are:
- </para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- <replaceable>name</replaceable> is the name for the
- connection (for use with the
- <literal>connection</literal> and
- <literal>disconnect</literal> commands). This name must
- not already be in use by an open connection.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <replaceable>host_name</replaceable> indicates the host
- where the server is running.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <replaceable>user_name</replaceable> and
- <replaceable>password</replaceable> are the username and
- password of the MySQL account to use.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <replaceable>db_name</replaceable> is the default
- database to use. As a special case,
- <literal>*NO-ONE*</literal> means that no default
- database should be selected.
- </para>
-
- <para>
- [Q: you can't just leave it blank to select no
- database?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <replaceable>port_num</replaceable>, if given, is the
- TCP/IP port number to use for the connection. This
- parameter can be given by using a variable.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <replaceable>socket</replaceable>, if given, is the
- socket file to use for connections to
- <literal>localhost</literal>. This parameter can be
- given by using a variable.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <replaceable>options</replaceable> can be one or more of
- the words <literal>SSL</literal> and
- <literal>COMPRESS</literal>. These specify the use of
- SSL and the compressed client/server protocol,
- respectively.
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- [Q: If a connection attempt fails initially, there is some
- kind of retry behavior?]
- </para>
-
- <para>
- Syntax oddities: There must be whitespace between
- <command>connect</command> and the opening parenthesis, and
- no whitepace after the opening parenthesis.
- </para>
-
- <para>
- Bug: leaving off the trailing parenthesis can produce a
- weird error. For example, with no paren after the database
- name, <command>mysqltest</command> says <literal>mysqltest:
- At line 1: Illegal argument for port: 'h4z'</literal>.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>connection
- <replaceable>connection_name</replaceable></literal>
- </para>
-
- <para>
- Select <replaceable>connection_name</replaceable> as the
- current connection. To select the connection that
- <command>mysqltest</command> opens when it starts, use the
- name <literal>default</literal>.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>dec $<replaceable>var_name</replaceable></literal>
- </para>
-
- <para>
- Decrement a numeric variable. If the variable is not
- numeric, the result is undefined.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>delimiter <replaceable>str</replaceable></literal>
- </para>
-
- <para>
- Set the command delimiter to <replaceable>str</replaceable>,
- which may consist of 1 to 15 characters. The default
- delimiter is the semicolon character
- (‘<literal>;</literal>’).
- </para>
-
- <para>
- [Q: The Wiki docs state a max length of 16, but that doesn't
- work; perhaps this is a bug? Or else MAX_DELIMITER includes
- the terminating null byte?]
- </para>
-
-<programlisting>
--- delimiter 1234567890123456
-mysqltest: At line 2: End of line junk detected: "6"
-</programlisting>
- </listitem>
-
- <listitem>
- <para>
- <literal>dirty_close
- <replaceable>connection_name</replaceable></literal>
- </para>
-
- <para>
- Close the named connection. This is like
- <literal>disconnect</literal> except that it calls
- <literal>vio_delete()</literal> before it closes the
- connection.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_abort_on_error</literal>,
- <literal>enable_abort_on_error</literal>
- </para>
-
- <para>
- Disable or enable abort-on-error behavior. This setting is
- enabled by default. With this setting enabled,
- <command>mysqltest</command> aborts the test when a
- statement sent to the server results in an unexpected error
- [Q: and does not generate the <filename>.reject</filename>
- file?]
- </para>
-
- <para>
- [Q: I would have expected
- <literal>disable_abort_on_error</literal> also to suppress
- aborts for a statement that succeeds when it shouldn't, or a
- statement that fails with an error different from that
- specified as expected.. But <command>mysqltest</command>
- aborts regardless.]
- </para>
-
- <para>
- [Wiki has some why-this-is-useful notes for these options]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_info</literal>,
- <literal>enable_info</literal>
- </para>
-
- <para>
- Disable or enable additional information about SQL statement
- results. This setting is disabled by default. With this
- setting enabled, <command>mysqltest</command> displays the
- affected-rows count and the output from the
- <literal>mysql_info()</literal> C API function. The
- <quote>affected-rows</quote> value is <quote>rows
- selected</quote> for queries such as
- <literal>SELECT</literal> and <quote>rows modified</quote>
- for statements that change data.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_metadata</literal>,
- <literal>enable_metadata</literal>
- </para>
-
- <para>
- Disable or enable query metadata display. This setting is
- enabled by default. With this setting enabled,
- <command>mysqltest</command> adds query metadata to the
- result. This information consists of the values
- corresponding to the members of the
- <literal>MYSQL_FIELD</literal> C API data structure, for
- each column of the result.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_parsing</literal>,
- <literal>enable_parsing</literal>
- </para>
-
- <para>
- Increment or decrement the parsing-disabled level. The level
- is 0 by default.
- </para>
-
- <para>
- [Q: What does this apply to? SQL statements? Non-comment
- commands? Comment (--) commnds?]
- </para>
-
- <para>
- [Q: If parsing is disabled, commands are treated as
- comments? The only commands recognized are --disable_parsing
- and --enable_parsing until the parsing level goes > 0?]
- </para>
-
- <para>
- [Q: Do these commands work only when given in "--" form?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_ps_protocol</literal>,
- <literal>enable_ps_protocol</literal>
- </para>
-
- <para>
- Disable or enable prepared-statement protocol. This setting
- is disabled by default unless the
- <option>--ps-protocol</option> option is given.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_ps_warnings</literal>,
- <literal>enable_ps_warnings</literal>
- </para>
-
- <para>
- Disable or enable prepared-statement warnings. This setting
- is enabled by default.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_query_log</literal>,
- <literal>enable_query_log</literal>
- </para>
-
- <para>
- Disable or enable query logging. This setting is enabled by
- default. With this setting enabled,
- <command>mysqltest</command> echoes input SQL statements to
- the test result.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_reconnect</literal>,
- <literal>enable_reconnect</literal>
- </para>
-
- <para>
- Disable or enable automatic reconnect for dropped
- connections. [Q: Is the default whatever the default is for
- the client library (which varies for different versions)?]
- </para>
-
- <para>
- [Q: When does this apply? To connections made afterward?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_result_log</literal>,
- <literal>enable_result_log</literal>
- </para>
-
- <para>
- Disable or enable the result log. This setting is enabled by
- default. With this setting enabled,
- <command>mysqltest</command> displays query results (and
- results from commands such as <literal>echo</literal> and
- <literal>exec</literal>).
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_rpl_parse</literal>,
- <literal>enable_rpl_parse</literal>
- </para>
-
- <para>
- Disable or enable parsing of statements to determine whether
- they go to the master or slave.
- </para>
-
- <para>
- [Q: The default is whatever the default is for the C API
- library?] [MySQL 4.0 and up only]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disable_warnings</literal>,
- <literal>enable_warnings</literal>
- </para>
-
- <para>
- Disable or enable warnings. This setting is enabled by
- default. With this setting enabled,
- <command>mysqltest</command> uses <literal>SHOW
- WARNINGS</literal> to display any warnings produced by SQL
- statements.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>disconnect
- <replaceable>connection_name</replaceable></literal>
- </para>
-
- <para>
- Close the named connection. If the connection is the current
- connection, you should use the <literal>connection</literal>
- command to switch to a different connection before executing
- further SQL statements.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>echo <replaceable>text</replaceable></literal>
- </para>
-
- <para>
- Echo the text to the test result.
- </para>
-
- <para>
- [Q: This is is supposed to allow some kind of variable
- expansion, but I find that this works only if the text to be
- echoed consists only of a single variable reference. That's
- not what the comments in the code lead me to expect.]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>end</literal>
- </para>
-
- <para>
- End an <literal>if</literal> or <literal>while</literal>
- block. If there is no such block open,
- <command>mysqltest</command> exits with an error.
- </para>
-
- <para>
- [<command>mysqltest</command> appears to consider
- <literal>}</literal> and <literal>end</literal> the same]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>end_timer</literal>
- </para>
-
- <para>
- Stop the timer. By default, the timer does not stop until
- just before <command>mysqltest</command> exits.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>error <replaceable>error_code</replaceable> [,
- <replaceable>error_code</replaceable>] ...</literal>
- </para>
-
- <para>
- Specify one or more comma-separated error values that the
- next command is expected to return. Each
- <replaceable>error_code</replaceable> value is either a
- MySQL-specific error number or a SQLSTATE value. (These are
- the kinds of values returned by the
- <literal>mysql_errno()</literal> and
- <literal>mysql_sqlstate()</literal> C API functions.) If you
- specify a SQLSTATE value, it should begin with an
- <literal>S</literal> to enable <command>mysqltest</command>
- to distinguish it from a MySQL error number. For example, to
- specify the SQLSTATE value <literal>42000</literal>, write
- it as <literal>S42000</literal>:
- </para>
-
-<programlisting>
---error S42000
-</programlisting>
-
- <para>
- [Wiki has some stuff about non-obvious effects for
- specifying error 0 in combination with non-zero error codes]
- </para>
-
- <para>
- [Q: 0 and S00000 are equivalent for specifying <quote>no
- error</quote>?]
- </para>
-
- <para>
- [Q: --error 0 is the same as saying <quote>no error
- expected, the statement must succeed</quote>?]
- </para>
-
- <para>
- [Q: Can you use <literal>error</literal> to specify shell
- status values for testing the value of shell commands
- (<literal>exec</literal>, <literal>system</literal>)?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>eval <replaceable>statement</replaceable></literal>
- </para>
-
- <para>
- Evaluate the statement and send it to the server as a
- statement to be executed. Variable references are expanded.
- Use ‘<literal>\$</literal>’ to specify a literal
- ‘<literal>$</literal>’ character.
- </para>
-
- <para>
- [Q: What is the advantage of using <literal>eval</literal>
- over not using <literal>eval</literal>? Variable reference
- expansion?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>eval_result</literal>
- </para>
-
- <para>
- [Unknown]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>exec <replaceable>command</replaceable>
- [<replaceable>arg</replaceable>] ...</literal>
- </para>
-
- <para>
- Execute the shell command using the
- <literal>popen()</literal> library call.
- </para>
-
- <para>
- On Cygwin, the command is executed from
- <command>cmd.exe</command>, so commands such as
- <command>rm</command> cannot be executed with
- <literal>exec</literal>. Use <literal>system</literal>
- instead.
- </para>
-
- <para>
- [Q: Are variable references expanded?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>exit</literal>
- </para>
-
- <para>
- Terminate the test case.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>horizontal_results</literal>
- </para>
-
- <para>
- Set the default query result display format to horizontal.
- Initially, the default is to display results horizontally.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>if (<replaceable>expr</replaceable>)</literal>
- </para>
-
- <para>
- Begin an <literal>if</literal> block, which continues until
- an <literal>end</literal> line. <command>mysqltest</command>
- executes the block if the expression is true. See
- <xref linkend="mysqltest-flow-control"/>, for information on
- flow-control constructs.
- </para>
-
- <para>
- [Q: There is no provision for <literal>else</literal>?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>inc $<replaceable>var_name</replaceable></literal>
- </para>
-
- <para>
- Increment a numeric variable. If the variable is not
- numeric, the result is undefined.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>let $<replaceable>var_name</replaceable> =
- <replaceable>value</replaceable></literal>
- </para>
-
- <para>
- Assign a value to a variable. <command>mysqltest</command>
- aborts with an error if the value is erroneous.
- </para>
-
- <para>
- If the <literal>let</literal> command is specified as a
- normal command (that is, not beginning with
- ‘<literal>--</literal>’), the value contains
- everything up to the command delimiter, and thus can span
- multiple lines.
- </para>
-
- <para>
- [Q: The <literal>$</literal> seems to be implied if not
- present?]
- </para>
-
- <para>
- [Q: The Wiki says there can be no space between the var name
- and the <literal>=</literal> sign. Is that really true? It
- also seems that whitespace following the
- <literal>=</literal> sign is discarded.]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>ping</literal>
- </para>
-
- <para>
- Ping the server. This executes the
- <literal>mysql_ping()</literal> C API function. The function
- result is discarded.
- </para>
-
- <para>
- [Q: I suppose the effect is that if the connection has
- dropped and reconnect is enabled, pinging the server causes
- a reconnect?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>query</literal>
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>query_horizontal
- <replaceable>statement</replaceable></literal>
- </para>
-
- <para>
- Like <literal>eval</literal>, but display the query result
- horizontally.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>query_vertical<replaceable>statement</replaceable></literal>
- </para>
-
- <para>
- Like <literal>eval</literal>, but display the query result
- vertically.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>real_sleep <replaceable>num</replaceable></literal>
- </para>
-
- <para>
- Sleep <replaceable>num</replaceable> seconds.
- <replaceable>num</replaceable> can have a fractional part.
- Unlike the <literal>sleep</literal> command,
- <literal>real_sleep</literal> is not affected, by the
- <option>--sleep</option> command option.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>reap</literal>
- </para>
-
- <para>
- Receive the result of the statement most recently sent with
- the <literal>send</literal> command.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>replace_column <replaceable>col_num</replaceable>
- <replaceable>value</replaceable>
- [<replaceable>col_num</replaceable>
- <replaceable>value</replaceable>] ...</literal>
- </para>
-
- <para>
- Replace strings in the result. The value in
- <replaceable>col_num</replaceable> is replaced by the
- corresponding <replaceable>value</replaceable>. There can be
- more than one
- <replaceable>col_num</replaceable>/<replaceable>value</replaceable>
- pair. Column numbers start with 1.
- </para>
-
- <para>
- [Q: This command applies only to the next command? Is that
- true for all the replace_xxx commands? Can values be quoted
- or specified using variables?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>replace_regex
- /<replaceable>pattern</replaceable>/<replaceable>replacement</replaceable>/[i]
- ...</literal>
- </para>
-
- <para>
- Find strings that match the pattern
- <replaceable>pattern</replaceable> and replace them with
- <replaceable>replacement</replaceable>. Each instance of a
- string in the line that matches the pattern is replaced.
- Matching is case sensitive by default. Specify the optional
- <literal>i</literal> modifier to cause matching to be case
- insensitive.
- </para>
-
- <para>
- The syntax for allowable patterns is the same as for the
- <literal>REGEXP</literal> SQL operator. In addition, the
- pattern can contain parentheses to mark substrings matched
- by parts of the pattern. These substrings can be referenced
- in the replacement string: An instance of
- <literal>\<replaceable>N</replaceable></literal> in the
- replacement string causes insertion of the
- <replaceable>N</replaceable>-th substring matched by the
- pattern. For example, the following command matches
- <literal>strawberry</literal> and replaces it with
- <literal>raspberry and strawberry</literal>:
- </para>
-
-<programlisting>
---replace_regex /(strawberry)/raspberry and \1/
-</programlisting>
-
- <para>
- Multiple pattern/replacement pairs may be given. The
- following command replaces instances of <literal>A</literal>
- with <literal>C</literal> (the first pattern replaces
- <literal>A</literal> with <literal>B</literal>, the second
- replaces <literal>B</literal> with <literal>C</literal>):
- </para>
-
-<programlisting>
---replace_regex /A/B/ /B/C/
-</programlisting>
-
- <para>
- [Better to use patterns with metacharacters for this
- example]
- </para>
-
- <para>
- If a pattern is not found, no error occurs and the input is
- unchanged.
- </para>
-
- <para>
- The <literal>replace_regex</literal> command was added in
- MySQL 5.1.6.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>replace_result <replaceable>from_val
- to_val</replaceable> [<replaceable>from_val
- to_val</replaceable>] ...</literal>
- </para>
-
- <para>
- Replace strings in the result. Each occurrence of
- <replaceable>from_val</replaceable> is replaced by the
- corresponding <replaceable>to_val</replaceable>. There can
- be more than
- <replaceable>from_val</replaceable>/<replaceable>to_val</replaceable>
- pair. Arguments can be quoted with single quotes or double
- quotes. [Q: Variable references within the arguments are
- expanded before replacement occurs?] Values are matched
- literally. To use patterns, use the
- <literal>replace_regex</literal> command.
- </para>
-
- <para>
- [Q: This applies both to the line of column names at the
- beginning of statement output, and to the data lines that
- follow? (Is that true for other replace_xxx commands?)]
- </para>
-
- <para>
- [Q: The behavior differs from
- <literal>replace_regex</literal>. For example
- <literal>replace_result a b b c</literal> does not change a
- to c. Apparently, result replacement for a column stops as
- soon as a match is found?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>require
- <replaceable>file_name</replaceable></literal>
- </para>
-
- <para>
- Compare the result [Q: from the current test?] with the
- contents of the named file. If the content does not match or
- there is some other error, the test aborts with an
- "abort_not_supported_test" error message.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>require_manager</literal>
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>result
- <replaceable>file_name</replaceable></literal>
- </para>
-
- <para>
- Compare the result [from the current test?] with the
- contents of the named file. If the content does not match or
- there is some other error, write the result to
- <filename>r/<replaceable>file_name</replaceable>.reject</filename>.
- </para>
-
- <para>
- [Q: So this assumes that there is an <filename>r</filename>
- directory under the current directory?
- </para>
-
- <para>
- [Ask these questions, but base them on the Wiki text, not my
- rewrite in the previous para.] [Q: So, does that mean
- <replaceable>file_name</replaceable> is given without the
- <filename>.result</filename> extension? Is the "current"
- test the most recent command executed within the test case,
- or the result of the entire test case up to the
- <literal>result</literal> command? Is "write the result"
- actually "write the diff of the expected and actual
- results"?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>rpl_probe</literal>
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>save_master_pos</literal>
- </para>
-
- <para>
- For a master replication server, save the current binary log
- filename and position. These values can be used for
- subsequent <literal>sync_with_master</literal> or
- <literal>sync_slave_with_master</literal> commands.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>send <replaceable>statement</replaceable></literal>
- </para>
-
- <para>
- Send a statement to the server but do not wait for the
- result. The result must be received with the
- <literal>reap</literal> command.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>server_start
- <replaceable>server_name</replaceable></literal>
- </para>
-
- <para>
- Contact the manager to start a server. The timeout value is
- 10 seconds.
- </para>
-
- <para>
- [Q: Does "manager" mean the Instance Manager? If so, is the
- connection to the manager made using the parameters
- specified by the
- <option>--manager-<replaceable>xxx</replaceable></option>
- command-line options?]
- </para>
-
- <para>
- [Q: This command is unavailable for
- <command>mysqltest_embedded</command>, the version of
- <command>mysqltest</command> built with support for
- <literal>libmysqld</literal>?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>server_stop
- <replaceable>server_name</replaceable></literal>
- </para>
-
- <para>
- Contact the manager to stop a server. The timeout value is
- 10 seconds.
- </para>
-
- <para>
- [Q: This command is unavailable for
- <command>mysqltest_embedded</command>, the version of
- <command>mysqltest</command> built with support for
- <literal>libmysqld</literal>?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>sleep <replaceable>num</replaceable></literal>
- </para>
-
- <para>
- Sleep <replaceable>num</replaceable> seconds.
- <replaceable>num</replaceable> can have a fractional part.
- If the <option>--sleep</option> command option was given,
- the option value overrides the value given in the
- <literal>sleep</literal> command. For example, if
- <command>mysqltest</command> is started with
- <option>--sleep=10</option>, the command <literal>sleep
- 15</literal> sleeps 10 seconds, not 15.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>source
- <replaceable>file_name</replaceable></literal>
- </para>
-
- <para>
- Reads test input from the named file. A sourced file can use
- <literal>source</literal> to read other files.
- </para>
-
- <para>
- If you find that several test case files contain a common
- section of commands (for example, statements that create a
- standard set of tables), you can put those commands in
- another file and those test cases that need the file can
- include it by means of a <literal>source
- <replaceable>file_name</replaceable></literal> command. This
- enables you to write the code just once rather than in
- multiple test cases.
- </para>
-
- <para>
- [Q: Max recursion level?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>start_timer</literal>
- </para>
-
- <para>
- Restart the timer, overriding any timer start that occurred
- earlier. By default, the timer starts when
- <command>mysqltest</command> begins execution.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>sync_slave_with_master
- [<replaceable>connection_name</replaceable>]</literal>
- </para>
-
- <para>
- Executing this command is equivalent to executing the
- following commands:
- </para>
-
-<programlisting>
-save_master_pos
-connection <replaceable>connection_name</replaceable>
-sync_with_master 0
-</programlisting>
-
- <para>
- If <replaceable>connection_name</replaceable> is not
- specified, the connection named <literal>slave</literal> is
- used.
- </para>
-
- <para>
- The effect is to save the replication coordinates (binary
- log filename and position) for the server on the current
- connection (which is assumed to be a master replication
- server), and then switch to a slave server and wait until it
- catches up with the saved coordinates. Note that this
- command implicitly changes the current connection.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>sync_with_master
- <replaceable>offset</replaceable></literal>
- </para>
-
- <para>
- For a slave replication server, wait until it has caught up
- with the master. The position to synchronize to is the
- position saved by the most recent
- <literal>save_master_pos</literal> command plus
- <replaceable>offset</replaceable>.
- </para>
-
- <para>
- To use this command, <literal>save_master_pos</literal> must
- have been executed at some point earlier in the test case to
- cause <command>mysqltest</command> to save the master's
- replication coordinates.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>system <replaceable>command</replaceable>
- [<replaceable>arg</replaceable>] ...</literal>
- </para>
-
- <para>
- Execute the shell command using the
- <literal>system()</literal> library call.
- </para>
-
- <para>
- On Cygwin, the command is executed from
- <command>cmd.exe</command>, so commands such as
- <command>rm</command> cannot be executed with
- <literal>exec</literal>. Use <literal>system</literal>
- instead.
- </para>
-
- <para>
- [Q: Are variable references expanded?]
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>vertical_results</literal>
- </para>
-
- <para>
- Set the default query result display format to vertical.
- Initially, the default is to display results horizontally.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>wait_for_slave_to_stop</literal>
- </para>
-
- <para>
- Poll the current connection, which is assumed to be a
- connection to a slave replication server, by using the
- <literal>SHOW STATUS LIKE 'Slave_running'</literal>
- statement until the result is <literal>OFF</literal>.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>while (<replaceable>expr</replaceable>)</literal>
- </para>
-
- <para>
- Begin a <literal>while</literal> loop block, which continues
- until an <literal>end</literal> line.
- <command>mysqltest</command> executes the block repeatedly
- as long as the expression is true. See
- <xref linkend="mysqltest-flow-control"/>, for information on
- flow-control constructs.
- </para>
-
- <para>
- Make sure that the loop includes some exit condition that
- eventually occurs. This can be done by writing
- <replaceable>expr</replaceable> so that it goes false at
- some point.
- </para>
- </listitem>
-
- </itemizedlist>
-
- </section>
-
- <section id="mysqltest-variables">
-
- <title><command>mysqltest</command> Variables</title>
-
<para>
- [Cover <literal>let</literal>, <literal>dec</literal>,
- <literal>inc</literal>. Cover use in <literal>eval</literal>,
- <literal>exec</literal>, <literal>system</literal>, and
- flow-control statements.
+ FIXME some environment variables are passed from the environment
+ to test language variables, list them. Why not all??
</para>
<para>
- Variables:
+ The following is a description of the language used in
+ .test-files.
</para>
<para>
- <literal>$mysql_errno</literal> is a built-in variable that
- contains the numeric error returned by the most recent statement
- sent to the server, or 0 if the command executes successfully.
- <literal>$mysql_errno</literal> has a value of −1 if no
- command has yet been sent.
+ The <command>mysqltest</command> program is a small interpreter
+ that interpret a special test language. Lines that it doesn't
+ recognize are assumed to be SQL queries that can be sent to the
+ MySQL server. In effect this makes the test files look like SQL
+ with some small extensions.
</para>
<para>
- [Q: Is $mysql_errno set only by SQL statements, or is it also
- set by exec, system, mysqltest internal commands?]
+ Lines starting with a "#" in the test files are comments.
</para>
<para>
- You can refer to environment variables. [Q: How is a variable
- determined to be such? If it's not recognized as a built-in
- variable or a variable defined in the script?]
+ Commands to the test engine itself are often put into comments,
+ like --cmd. This sometimes leads to conflicts with comments that
+ start with the same word as a command.
</para>
<para>
- [Q: Variable names appear not to be case sensitive. This leads
- to an ambiguity: If you refer to a variable named
- <literal>$path</literal>, is that a <command>mysqltest</command>
- variable, or the <literal>$PATH</literal> environment variable?]
+ FIXME what is @filename ?!
</para>
</section>
- <section id="mysqltest-flow-control">
-
- <title><command>mysqltest</command> Flow Control Constructs</title>
-
- <para>
- [Cover <literal>exit</literal> here?]
- </para>
-
- <para>
- The syntax for <literal>if</literal> and
- <literal>while</literal> blocks looks like this:
- </para>
-
-<programlisting>
-if (<replaceable>expr</replaceable>)
-{
- <replaceable>command list</replaceable>
-}
-</programlisting>
-
-<programlisting>
-while (<replaceable>expr</replaceable>)
-{
- <replaceable>command list</replaceable>
-}
-</programlisting>
-
- <para>
- If the expression begins with <literal>!</literal>, the sense of
- the test is reversed.
- </para>
-
- <para>
- [Q: The opening <literal>{</literal> must be on the next line
- following <literal>if</literal> or <literal>while</literal>?]
- </para>
-
- <para>
- No spaces allowed between parens and the expression within the
- parens? What is the allowable syntax for
- <literal>expr</literal>?]
- </para>
-
- <para>
- [Q: I'm confused by this in several respects. Is a block written
- using <literal>{</literal> and <literal>}</literal>, or does it
- end with <literal>end</literal>? Also, the comments in the code
- indicate that <replaceable>expr</replaceable> is true if greater
- than zero and that adding <literal>!</literal> makes
- <replaceable>expr</replaceable> true if it's zero. But those are
- not complementary. What if <replaceable>expr</replaceable> is
- less than zero?]
- </para>
-
- <para>
- Hmm: A block can begin with <literal>{</literal> and end with
- <literal>end</literal>.
- </para>
-
- </section>
-
- <section id="mysqltest-misc">
-
- <title><command>mysqltest</command> Misc. Stuff</title>
-
- <para>
- If an expected error is specified and that error occurs,
- <command>mysqltest</command> continues reading input. If the
- command is successful or a different error occurs,
- <command>mysqltest</command> aborts.
- </para>
-
- <para>
- If no expected error is specified, <command>mysqltest</command>
- aborts unless the command is successful. (It is implicit that
- you expect <literal>$mysql_errno</literal> to be 0.)
- </para>
-
- </section>
-
</chapter>
- <chapter id="mysqltest-wiki">
-
- <title><command>mysqltest</command> Stuff from the Wiki</title>
-
- <section id="test-framework">
-
- <title>The <command>mysqltest</command> Test Framework</title>
-
- <para>
- The MySQL test framework is used primarily to run the MySQL
- server regression suite. The test cases are for most part SQL
- commands, mixed by some MySQL Test language constructs to
- control how the test is to be run and how the result is to be
- verified.
- </para>
-
- <para>
- The application that run the test suite is named
- <command>mysql-test-run</command> and is located in the
- <filename>mysql-test</filename> directory of both source and
- binary MySQL server distributions.
- </para>
-
- <para>
- The test cases are located in the
- <filename>mysql-test/t</filename> directory. Each test file is
- usually refered to as one test, but usually consist of a
- sequence of related tests. An unexpected failure of a single
- statement in the test file, will make this test fail.
- </para>
-
- <para>
- The <command>mysql-test-run</command> application will start
- MySQL servers, restart it if a specific test case needs
- different start arguments, and present the test result. You may
- hear referenses to the mysqltest application, this is the
- application that is called by <command>mysql-test-run</command>
- to run a single test case, intepret the test language
- constructs, and send SQL commands to the server.
- </para>
-
- <para>
- There is two ways a test can fail. The test engine check the
- result codes from executing each SQL command. If the failure is
- unexpected, the test case will fail.
- </para>
-
- <para>
- The other way it can fail, is by producing the incorrect
- trace/protocol output. When running the test, it writes out the
- result from SHOW and other commands to a file. This file is then
- compared to the expected trace found in the
- <filename>mysql-test/r</filename> directory. If it is different,
- the test case will fail and the difference will be presented for
- evaluation.
- </para>
-
- <para>
- This way of comparing the result trace, puts some restrictions
- on how a test case can be written. You can't have a result that
- varies from run to run, like the use the current time. If the
- data that varies is unimportant for the evaluation there are
- ways to instruct the test engine to replace those fields in the
- output with fixed data.
- </para>
-
- <para>
- Because the test cases are written mostly in SQL in a text file,
- there is no direct support to do test cases that are written in
- C, Java or other languages. This is not the scope of this test
- framework. But the test framework support executing your own
- scripts, initiating with your own data and in a test case
- execute any program so it may be used for some other testing
- than just SQL statements.
- </para>
-
- </section>
-
- <section id="mysqltest-tutorial">
-
- <title><command>mysqltest</command> Tutorial</title>
-
- <section>
-
- <title>Writing a test case</title>
-
- <para>
- You write test cases in any text editor that use LF (linefeed)
- as the end-of-line character. File names are in lower case
- ASCII, with no spaces.
- </para>
-
- <para>
- We are adding support for multiple <quote>suites.</quote>
- Until then, all test cases are to be located in
- <filename>mysql-test/t</filename>.
- </para>
-
- <para>
- One test file can be a collection of tests that you think
- belongs together. But this also means that if one of the tests
- fail, the whole test case will fail. While this makes it
- tempting to write each small test into a single file, this
- will be to inefficient and make the test runs unbearably slow.
- So make the test files not too big, not too small.
- </para>
-
- <para>
- To make us talk about the same thing, let us define
- </para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- One <quote>test file</quote> is one <quote>test
- case.</quote>
- </para>
- </listitem>
-
- <listitem>
- <para>
- One <quote>test case</quote> might contain a <quote>test
- sequence</quote> (that is, a number of individual tests
- that are grouped together in the same test file).
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- Also, a test case (i.e. a test file) is to be self contained.
- This means you are not allowed to create and/or fill a table
- in one test case and depend on this in a later test case. If
- you have some common initiation to be done, you put this into
- a file in the <filename>mysq-test/include</filename>
- directory, and in each test case that needs it, you do a
- <literal>--source include/xxxxx.inc</literal>. We take this by
- example. Lets say you add a test case called
- <literal>foo</literal>. You then create a file
- <filename>mysql-test/t/foo.test</filename>.
- </para>
-
- <para>
- A comment in a test case can be started with the '#'
- character, or the "--" characters. Note that the first word
- after the "--" might be a command to the test engine itself,
- More about that later.
- </para>
-
- <para>
- It is safest to use the '#' character for comments, not to
- accidently run a mysq-test directive/command. For example
- writing "-- End or test 43" will give an error message because
- "end" is something that <command>mysqltest</command> thinks is
- a directive/command.
- </para>
-
- <para>
- Note that because all our test cases will be published to all
- Internet, be careful that your comments inside the test cases
- contain no confidential information.
- </para>
-
- </section>
-
- <section>
-
- <title>Naming rules for tables, views etc</title>
-
- <para>
- The test cases might be used to run against a production
- server (we will generally not do that, but our customers
- might). To reduce the risk that by running the test suite,
- important tables are altered or destroyed, you are to name
- your tables, views and other objects like
- </para>
-
-<programlisting>
-table names : t1, t2, t3.....
-view names : v1, v2, v3.....
-</programlisting>
-
- <para>
- Look in the other test cases for examples of this. Of course
- you can name field names and other objects inside tables as
- you wish. If you don't have a special reason to, you are to
- use the default database already created for you 'test'.
- </para>
-
- </section>
-
- <section>
-
- <title>Cleaning up from an old run</title>
-
- <para>
- For efficiency our test engine will not start with a clean new
- database for running each test case. So a test case generally
- starts with a "cleaning up section". Let us assume we are to
- use two tables in our new test case <literal>foo</literal>.
- You then begin your test file with
- </para>
-
-<programlisting>
-#
-# Test of XXXXX
-#
---disable_warnings
-drop table if exists t1,t2;
---enable_warnings
-</programlisting>
-
- <para>
- The "disable_warnings" command instructs the test engine not
- to log any warnings until "enable_warnings" or the test case
- is ended. For some unknown reason MySQL choose to write a
- warning if the table 't1' or 't2' exists. Surrounding this
- part with the disable/enable warnings makes the test case
- output the same if the tables exists or not before the test is
- started. Now we are free to put in any SQL commands that
- create, use and drop the tables 't1' and 't2'. The rule is
- that you are to have a corresponding drop of each table you
- create in the test.
- </para>
-
- <para>
- So lets put in some SQL code into this test case
- </para>
-
-<programlisting>
-create table t1 (
- Period smallint(4) unsigned zerofill default '0000' not null,
- Varor_period smallint(4) unsigned default '0' not null
-);
-
-insert into t1 values (9410,9412);
-
-select period from t1;
-select * from t1;
-select t1.* from t1;
-
-drop table t1;
-</programlisting>
-
- </section>
-
- <section>
-
- <title>Determining whether a test fails</title>
-
- <para>
- Note that the test code we just wrote contains no checks of
- the result. The test will report a failure for one of two
- reasons
- </para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- A SQL statement failed with an error
- </para>
- </listitem>
-
- <listitem>
- <para>
- The result is not what was expected
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- So we are to specify the expected result somehow. You do this
- by running the test with the flag "--record", like
- </para>
-
-<programlisting>
-% cd mysql-test
-% ./mysql-test-run --record foo
-</programlisting>
-
- <para>
- [Q: Is the <quote>protocol</quote> file the result file or the
- reject file? I have seen the term applied to both.]
- </para>
-
- <para>
- This will create a file
- <filename>mysql-test/r/foo.result</filename>, also called the
- <quote>protocol file,</quote> with the content
- </para>
-
-<programlisting>
-drop table if exists t1,t2;
-create table t1 (
-Period smallint(4) unsigned zerofill default '0000' not null,
-Varor_period smallint(4) unsigned default '0' not null
-);
-insert into t1 values (9410,9412);
-select period from t1;
-period
-9410
-select * from t1;
-Period Varor_period
-9410 9412
-select t1.* from t1;
-Period Varor_period
-9410 9412
-drop table t1;
-</programlisting>
-
- <para>
- If we look at this file, it contains the commands in the
- <filename>foo.test</filename> file together with the output
- from the select commands. The output is the table headings and
- fields separated by the TAB character. So now you are to
- inspect this file, decide if the result is the expected one,
- and let it be part of your test case.
- </para>
-
- </section>
-
- <section>
-
- <title>Running test cases</title>
-
- <para>
- You will normally either run the test case from within a
- source tree, or on an installed server. In source tree you
- will be located in the <filename>mysql-test</filename>
- directory. The installed server also have a
- <filename>mysql-test</filename> directory. The program running
- the test suite, <command>mysql-test-run</command>, will figure
- out if you are in a source tree, or in an installed directory
- tree.
- </para>
-
- <para>
- You will not run against a started server. Instead the
- <command>mysql-test-run</command> program will start the
- server or servers needed on ports that doesn't conflicts with
- a production server, if you happen to have one on the same
- machine.
- </para>
-
- <para>
- You can run our newly created test case <literal>foo</literal>
- as part of the whole suite, or run it by itself like
- </para>
-
-<programlisting>
-% ./mysql-test-run foo
-</programlisting>
-
- <para>
- If you have a family of test cases, prefixed with something
- common, you can use
- </para>
-
-<programlisting>
-% ./mysql-test-run --do-tests=mytest
-</programlisting>
-
- </section>
-
- <section>
-
- <title>Mutilple users on the same machine, or shared directories</title>
-
- <para>
- If you are multiple users that run test on the same machine,
- you need to specify the ports to the
- <command>mysql-test-run</command> program, so that your run
- doesn't conflict with others. You add unique port arguments to
- <command>mysql-test-run</command>, such as
- <option>--no-manager --master_port=3911
- --slave_port=3927</option>.
- </para>
-
- <para>
- Also note that only one person can run the
- <command>mysql-test-run</command> program in the same
- <filename>mysql-test</filename> directory on a shared drive,
- at the same time. The <filename>mysql-test/var</filename>
- directory created can't be shared between mutiple users.
- </para>
-
- </section>
-
- <section>
-
- <title>How to get more information to check the result against</title>
-
- <para>
- The test engine by default only output things from "select",
- "show" and other SQL commands you expect to produce output,
- i.e. commands that create a data set. But it can be instructed
- to be more or less verbose, including other data as well.
- </para>
-
- <para>
- Lets say we are a bit more interested how many rows are
- effected by an operation, we add "--enable_info" to the test
- case before we create the table. Now, after we rerun the test
- with "--record", we get some more information in our result
- file
- </para>
-
-<programlisting>
-drop table if exists t1,t2;
-create table t1 (
-Period smallint(4) unsigned zerofill default '0000' not null,
-Varor_period smallint(4) unsigned default '0' not null
-);
-affected rows: 0
-insert into t1 values (9410,9412);
-affected rows: 1
-select period from t1;
-period
-9410
-affected rows: 1
-select * from t1;
-Period Varor_period
-9410 9412
-affected rows: 1
-select t1.* from t1;
-Period Varor_period
-9410 9412
-affected rows: 1
-drop table t1;
-</programlisting>
-
- <para>
- Options can in general be enabled and disabled for different
- parts of the test file. Lets say we are interested in the
- internals of the database as well. We can then enable the
- output of "meta data" using "--enable_metadata". The output
- now begins to be a bit verbose, but as said earlier, this can
- be enabled and disabled just for a part of the test where it
- interests you to know more.
- </para>
-
- </section>
-
- <section>
-
- <title>How to get less information</title>
-
- <para>
- You might be initializing a table, where we don't really
- expect a failure. You can then temporarily disable the
- recording of the command lines executed with
- "--disable_query_log" and then enable the recording again with
- "--enable_query_log". You can disable the recording of the
- output from executing commands using "--disable_result_log"
- and enable again with "--enable_result_log".
- </para>
-
- </section>
-
- <section>
-
- <title>What if the output varies?</title>
-
- <para>
- In general you are to avoid having tests that give different
- output for each run, or for differences in how the mysql
- binaries are created, operating system etc. The first example
- that comes to mind is the use of the current date and time.
- The test engine have no way of verifying that it is correct.
- </para>
-
- <para>
- Now, you might have a test that have a column that will vary,
- but you are not really interested in what is in that column
- anyway. The test engine let you specify that you want to
- replace whatever is in that column with a string. This makes
- the output from our run the same each time.
- </para>
-
- <para>
- Just to illustrate the functionality, we add after our first
- insert, the row
- </para>
-
-<programlisting>
-insert into t1 values (date_format(now(), '%s'),9999);
-</programlisting>
-
- <para>
- We now record and run the test case
- </para>
-
-<programlisting>
-% ./mysql-test-run --record foo
-% ./mysql-test-run foo
-</programlisting>
-
- <para>
- Now we will, most likely, get a failure and the test engine
- will display the difference between what we got and the
- expected result, like
- </para>
-
-<programlisting>
-Below are the diffs between actual and expected results:
--------------------------------------------------------
-*** r/foo.result Thu Jan 20 18:38:37 2005
---- r/foo.reject Thu Jan 20 18:39:00 2005
-***************
-*** 11,25 ****
- select period from t1;
- period
- 9410
-! 0037
- affected rows: 2
- select * from t1;
- Period Varor_period
- 9410 9412
-! 0037 9999
- affected rows: 2
- select t1.* from t1;
- Period Varor_period
- 9410 9412
-! 0037 9999
- affected rows: 2
---- 11,25 ----
- select period from t1;
- period
- 9410
-! 0059
- affected rows: 2
- select * from t1;
- Period Varor_period
- 9410 9412
-! 0059 9999
- affected rows: 2
- select t1.* from t1;
- Period Varor_period
- 9410 9412
-! 0059 9999
- affected rows: 2
--------------------------------------------------------
-</programlisting>
-
- <para>
- If we are not really interested in the first column, one way
- to solve this by using the directive/command
- "--replace_column". The duration of the effect of this
- directive/command, is the next SQL statement, so we need one
- before each select call, like
- </para>
-
-<programlisting>
---replace_column 1 SECONDS
-select period from t1;
---replace_column 1 SECONDS
-select * from t1;
---replace_column 1 SECONDS
-select t1.* from t1;
-</programlisting>
-
- <para>
- Note that "SECONDS" could be any string, it is just to display
- something. If we now record again and run the test, we will
- succeed each time. The result file now looks like
- </para>
-
-<programlisting>
-drop table if exists t1,t2;
-create table t1 (
-Period smallint(4) unsigned zerofill default '0000' not null,
-Varor_period smallint(4) unsigned default '0' not null
-);
-affected rows: 0
-insert into t1 values (9410,9412);
-affected rows: 1
-insert into t1 values (date_format(now(), '%s'),9999);
-affected rows: 1
-select period from t1;
-period
-SECONDS
-SECONDS
-affected rows: 2
-select * from t1;
-Period Varor_period
-SECONDS 9412
-SECONDS 9999
-affected rows: 2
-select t1.* from t1;
-Period Varor_period
-SECONDS 9412
-SECONDS 9999
-affected rows: 2
-</programlisting>
-
- <para>
- If the result may vary, but just between some know values, you
- can use "--replace_result" to match the different result, and
- replace them with a common string, making the result file the
- same even if the result varies some.
- </para>
-
- </section>
-
- <section>
-
- <title>Expected failures</title>
-
- <para>
- A good test suite not only test when things go right, but also
- that we get the correct error message, when it goes wrong. The
- test engine let you specify "expected failures". Lets say we
- try to create 't1' again without dropping it in between, like
- </para>
-
-<programlisting>
---disable_warnings
-drop table if exists t1,t2;
---enable_warnings
-
-create table t1 (
- Period smallint(4) unsigned zerofill default '0000' not null,
- Varor_period smallint(4) unsigned default '0' not null
-);
-
-insert into t1 values (9410,9412);
-
-select period from t1;
-select * from t1;
-select t1.* from t1;
-
-create table t1 (something smallint(4));
-</programlisting>
-
- <para>
- This gives a failure and an error
- </para>
-
-<programlisting>
-At line 19: query 'create table t1 (something smallint(4))'
- failed: 1050: Table 't1' already exists
-</programlisting>
-
- <para>
- We put in a "--error 1050" just before the second "create
- table", to mark that this is an expected failure, and that we
- expected it to return the MySQL error code 1050. If we look at
- the end of the result file we now have
- </para>
-
-<programlisting>
-create table t1 (something smallint(4));
-ERROR 42S01: Table 't1' already exists
-</programlisting>
-
- <para>
- So we still can match on the expected error, but it will not
- terminate the test case run because of that error. The "42S01"
- is the SQLSTATE code. Alternatively you might match on that
- error code, by using the 'S' prefix, like "--error S42S01".
- </para>
-
- </section>
-
- <section>
-
- <title>Setting server options</title>
-
- <para>
- Many server options can be set from within the test case, like
- </para>
-
-<programlisting>
-set sql_warnings=1;
-set sql_mode=no_auto_value_on_zero;
-</programlisting>
-
- <para>
- But sometimes you need to restart the server to use some
- options in the form of extra command line arguments. The test
- engine will restart the server just before running a test
- case, if there exists a file
- <filename>mysql-test/t/<replaceable>test_name</replaceable>-master.opt</filename>
- with the options you want to use. After the test case is
- completed, the server will be restarted again, with the
- default arguments.
- </para>
-
- </section>
-
- </section>
-
- <section id="mysqltest-test-cases">
-
- <title>More on Writing Test Cases</title>
-
- <section>
-
- <title>Running Test Case Overview</title>
-
- <para>
- This is a brief description of how to write test cases and the
- language used to write them. Normally you run the test suite,
- when developing.
- </para>
-
- <para>
- For example:
- </para>
-
-<programlisting>
-% cd mysql-test
-% mysql-test-run
-</programlisting>
-
- <para>
- Or:
- </para>
-
-<programlisting>
-% mysql-test-run --local --force
-</programlisting>
-
- <para>
- Use --force if you want mysql-test to continue on failure of a
- test case
- </para>
-
- <para>
- For a single test case or group or test cases(the .test
- extension exluded)
- </para>
-
-<programlisting>
-% mysql-test-run --local --do-test=<replaceable>test_name_prefix</replaceable>
-</programlisting>
-
- <para>
- This can be a single test case or a many tests with the same
- prefix.
- </para>
-
- <para>
- Examples:
- </para>
-
-<programlisting>
-% mysql-test-run --local --do-test=rpl_abcd
-</programlisting>
-
- <para>
- This will run the rpl_abcd.test
- </para>
-
-<programlisting>
-% mysql-test-run --local --do-test=rpl
-</programlisting>
-
- <para>
- This will run all the tests that start with rpl.
- </para>
-
- <para>
- Please view the README file in the mysql-test directory for
- more details about how to invoke the test script
- <command>mysql-test-run</command>. The
- <command>mysql-test-run</command> is setting up the
- environment for calling the program mysqltest and start a
- server etc. mysqltest is the program that will read the test
- case language text and start managers, create connections,
- send queries to a server etc.
- </para>
-
- <para>
- The test case language is a mix of commands that the
- <command>mysqltest</command> program understand and SQL. What
- it doesn't understand is assumed to be SQL queries and are
- sent to the database server. This makes the test case language
- familiar to those that know how to write SQL and powerful
- enough to add the control needed to write test cases.
- </para>
-
- </section>
-
- <section>
-
- <title>Writing Test Cases Basics</title>
-
- <para>
- The base principle is that the test script compares the output
- from running the tests with the expected result. This is just
- a diff between the output and a file the test writer provide.
- With this simplistic way of comparing you can't handle
- variations in the output from running the script but the test
- language provide ways to suppress unwanted output making this
- usable. The steps to write a new test case are
- </para>
-
- <orderedlist>
-
- <listitem>
- <para>
- Change directories to the test directory
- <filename>mysql-<replaceable>version</replaceable>/mysql-test</filename>
- </para>
-
-<programlisting>
-% cd mysql-<replaceable>version</replaceable>/mysql-test
-</programlisting>
- </listitem>
-
- <listitem>
- <para>
- Create the test in a file
- "t/<replaceable>test_name</replaceable>.test"
- </para>
- </listitem>
-
- <listitem>
- <para>
- Create an empty result file
- </para>
-
-<programlisting>
-% touch r/<replaceable>test_name</replaceable>.result
-</programlisting>
- </listitem>
-
- <listitem>
- <para>
- Run the test
- </para>
-
-<programlisting>
-% ./mysql-test-run <replaceable>test_name</replaceable>
-</programlisting>
- </listitem>
-
- <listitem>
- <para>
- The failing test will create a file
- "r/<replaceable>test_name</replaceable>.reject". Examine
- this file and if it looks ok you copy its content to the
- result file
- </para>
-
-<programlisting>
-% cp r/<replaceable>test_name</replaceable>.reject r/<replaceable>test_name</replaceable>.result
-</programlisting>
- </listitem>
-
- <listitem>
- <para>
- Run the test again, this time it should succeed
- </para>
-
-<programlisting>
-% ./mysql-test-run <replaceable>test_name</replaceable>
-</programlisting>
- </listitem>
-
- </orderedlist>
-
- <para>
- The environment can also be set up so the
- <command>mysqltest</command> program can be called directly.
- For more information about options to the
- <command>mysqltest</command> program run
- </para>
-
-<programlisting>
-% mysqltest --help
-</programlisting>
-
- <para>
- There are also some notes about <command>mysqltest</command>
- in the README in this directory.
- </para>
-
- </section>
-
- <section>
-
- <title>Sample Test Case</title>
-
- <para>
- Small sample test case
- </para>
-
-<programlisting>
---disable_warnings
-DROP TABLE IF EXISTS t1;
---enable_warnings
-SET SQL_WARNINGS=1;
-
-CREATE TABLE t1 (a INT);
-INSERT INTO t1 VALUES (1);
-INSERT INTO t1 VALUES ("hej");
-</programlisting>
-
- <para>
- The first rows try to cleanup from possibly earlier runs of
- the script by dropping the table "t1". We disable storing
- warnings (--disable_warnings) in the output during this
- because we are not interested to know if the table "t1" was
- there or not. Then we enable storing warnings again
- (--enable_warnings) and enable verbose warnings in MySQL using
- the SET SQL_WARNINGS=1;" line.
- </para>
-
- <para>
- We create the table "t1" and try some operations. The creation
- of the table and first insertion will not create any warnings
- but the last line will. We collect the output from running
- this test and add this to the result file
- </para>
-
-<programlisting>
-DROP TABLE IF EXISTS t1;
-SET SQL_WARNINGS=1;
-CREATE TABLE t1 (a INT);
-INSERT INTO t1 VALUES (1);
-INSERT INTO t1 VALUES ("hej");
-Warnings:
-Warning 1265 Data truncated for column 'a' at row 1
-</programlisting>
-
- <para>
- Note that we see the commands as well in the result. This can
- be disabled with the "--disable_query_log" test language
- command. There are several options how to control the amount
- of output from running the tests. If there was an failure this
- will be reported to the screen. You can see the actual output
- from the last unsuccessful run of the testcase in the test
- case reject file
- "r/<replaceable>test_name</replaceable>.reject".
- </para>
-
- </section>
-
- <section>
-
- <title>Test Environment Overview</title>
-
- <para>
- There are two ways the same test environment should handle.
- One is to talk to an installed running server. The second case
- is to be able to start a server from the binaries in the
- source tree, i.e. an uninstalled MySQL server. Because of this
- in general you don't start a server and create a connection
- from the test case, this is done by the
- <command>mysql-test-run</command> script before your test case
- is run.
- </para>
-
- <para>
- FIXME add the most important options to use calling
- <command>mysql-test-run</command>
- </para>
-
- </section>
-
- <section>
-
- <title>Language Overview</title>
-
- <para>
- FIXME make descriptions shorter by introducing a syntax to
- inform that an argument could be a test language variable. Why
- not all?
- </para>
-
- <para>
- FIXME some environment variables are passed from the
- environment to test language variables, list them. Why not
- all??
- </para>
-
- <para>
- The following is a description of the language used in
- .test-files.
- </para>
-
- <para>
- The <command>mysqltest</command> program is a small
- interpreter that interpret a special test language. Lines that
- it doesn't recognize are assumed to be SQL queries that can be
- sent to the MySQL server. In effect this makes the test files
- look like SQL with some small extensions.
- </para>
-
- <para>
- Lines starting with a "#" in the test files are comments.
- </para>
-
- <para>
- Commands to the test engine itself are often put into
- comments, like --cmd. This sometimes leads to conflicts with
- comments that start with the same word as a command.
- </para>
-
- <para>
- FIXME what is @filename ?!
- </para>
-
- </section>
-
- </section>
-
- </chapter>
-
</book>
| Thread |
|---|
| • svn commit - mysqldoc@docsrva: r1556 - in trunk: . mysqltest | paul | 11 Mar |
