| List: | Commits | « Previous MessageNext Message » | |
| From: | paul | Date: | March 15 2006 6:17pm |
| Subject: | svn commit - mysqldoc@docsrva: r1590 - in trunk: . mysqltest | ||
| View as plain text | |||
Author: paul Date: 2006-03-15 18:17:26 +0100 (Wed, 15 Mar 2006) New Revision: 1590 Log: r3923@kite-hub: paul | 2006-03-15 11:16:10 -0600 General revisions. Modified: trunk/ trunk/mysqltest/command-reference.xml trunk/mysqltest/introduction.xml trunk/mysqltest/tutorial.xml Property changes on: trunk ___________________________________________________________________ Name: svk:merge - b5ec3a16-e900-0410-9ad2-d183a3acac99:/mysqldoc-local/mysqldoc/trunk:8693 bf112a9c-6c03-0410-a055-ad865cd57414:/mysqldoc-local/mysqldoc/trunk:3907 + b5ec3a16-e900-0410-9ad2-d183a3acac99:/mysqldoc-local/mysqldoc/trunk:8693 bf112a9c-6c03-0410-a055-ad865cd57414:/mysqldoc-local/mysqldoc/trunk:3923 Modified: trunk/mysqltest/command-reference.xml =================================================================== --- trunk/mysqltest/command-reference.xml 2006-03-15 15:30:18 UTC (rev 1589) +++ trunk/mysqltest/command-reference.xml 2006-03-15 17:17:26 UTC (rev 1590) @@ -17,13 +17,6 @@ <listitem> <para> - Make the distinction between <quote><command>mysqltest</command> - command</quote> and <quote>SQL statement/query</quote>. - </para> - </listitem> - - <listitem> - <para> Verify the syntax of each command. </para> </listitem> @@ -46,27 +39,44 @@ </para> <para> - <emphasis role="bold">Terminology note:</emphasis> + <emphasis role="bold">Terminology notes:</emphasis> </para> - <para> - When <command>mysqltest</command> starts, it opens a connection - named <literal>default</literal> to the MySQL server, using any - connection parameters specified by the command options. [Q: When run - from <command>mysql-test-run</command>, the default username will be - <literal>root</literal>?] You can use the <literal>connect</literal> - command to open other connections, the <literal>connection</literal> - command to switch between connections, and the - <literal>disconnect</literal> command to close connections. However, - the capability for switching connections means that - <literal>default</literal> need not be the connection in use at a - given time. To avoid ambiguity, this document avoids the term - <quote>default connection.</quote> It uses the term <quote>current - connection</quote> to mean <quote>the connection currently in - use,</quote> which might be different from <quote>the connection - named <literal>default</literal>.</quote> - </para> + <itemizedlist> + <listitem> + <para> + A <quote>command</quote> is an input test that + <command>mysqltest</command> recognizes and executes itself. A + <quote>statement</quote> is an SQL statement or query that + <command>mysqltest</command> sends to the MySQL server to be + executed. + </para> + </listitem> + + <listitem> + <para> + When <command>mysqltest</command> starts, it opens a connection + named <literal>default</literal> to the MySQL server, using any + connection parameters specified by the command options. [Q: When + run from <command>mysql-test-run</command>, the default username + will be <literal>root</literal>?] You can use the + <literal>connect</literal> command to open other connections, + the <literal>connection</literal> command to switch between + connections, and the <literal>disconnect</literal> command to + close connections. However, the capability for switching + connections means that <literal>default</literal> need not be + the connection in use at a given time. To avoid ambiguity, this + document avoids the term <quote>default connection.</quote> It + uses the term <quote>current connection</quote> to mean + <quote>the connection currently in use,</quote> which might be + different from <quote>the connection named + <literal>default</literal>.</quote> + </para> + </listitem> + + </itemizedlist> + <section id="mysqltest-input-conventions"> <title><command>mysqltest</command> Input Conventions</title> Modified: trunk/mysqltest/introduction.xml =================================================================== --- trunk/mysqltest/introduction.xml 2006-03-15 15:30:18 UTC (rev 1589) +++ trunk/mysqltest/introduction.xml 2006-03-15 17:17:26 UTC (rev 1590) @@ -12,7 +12,7 @@ <para> MySQL distributions include a set of test cases and programs for running them. These tools constitute the MySQL test framework that - provideas a means for verifying that MySQL Server and its client + provides a means for verifying that MySQL Server and its client programs operate according to expectations. The test cases consist mostly of SQL statements, but can also use test language constructs that control how to run tests and verify their results. @@ -31,12 +31,22 @@ <filename>mysql-test</filename> directory, which is present both in source and binary MySQL Server distributions. (Note: The <command>mysql-test-run</command> shell script is being replaced by - the <command>mysql-test-run.pl</command> Perl script.) - <command>mysql-test-run</command> invokes - <command>mysqltest</command> once for each test case. + the <command>mysql-test-run.pl</command> Perl script, which + currently is under development.) </para> <para> + The <command>mysql-test-run</command> application starts MySQL + servers, restarts them as necessary when a specific test case needs + different start arguments, and presents the test result. The + <command>mysql-test-run</command> application invokes the + <command>mysqltest</command> program to run a single test case, + intepret the test language constructs, and send SQL statements to + the server. <command>mysqltest</command> is also referred to as the + <quote>test engine.</quote> + </para> + + <para> Input for each test case is stored in a file, and the expected result from running the test is stored in another file. The expected result can be compared to the actual result from running a test to @@ -78,17 +88,6 @@ </para> <para> - The <command>mysql-test-run</command> application starts MySQL - servers, restarts them as necessary when a specific test case needs - different start arguments, and presents the test result. The - <command>mysql-test-run</command> application invokes the - <command>mysqltest</command> program to run a single test case, - intepret the test language constructs, and send SQL statements to - the server. <command>mysqltest</command> is also referred to as the - <quote>test engine.</quote> - </para> - - <para> There are two ways a test case can fail: </para> @@ -122,11 +121,11 @@ <para> This method of comparing the result trace puts some restrictions on - how test cases can be written. For example, you cannot have a result - that varies from run to run, such as the current time. However, if - the information that varies is unimportant for test evaluation, - there are ways to instruct the test engine to replace those fields - in the output with fixed values. + how test cases can be written. For example, the result cannot + contain information that varies from run to run, such as the current + time. However, if the information that varies is unimportant for + test evaluation, there are ways to instruct the test engine to + replace those fields in the output with fixed values. </para> <para> @@ -135,7 +134,7 @@ C, Java, or other languages. Such tests are not within the scope of this test framework. But the framework does support executing your own scripts and initiating them with your own data. Also, a test - case can execute an external program, so in some respects, the test + case can execute an external program, so in some respects the test framework can be extended for uses other than testing only SQL statements. </para> Modified: trunk/mysqltest/tutorial.xml =================================================================== --- trunk/mysqltest/tutorial.xml 2006-03-15 15:30:18 UTC (rev 1589) +++ trunk/mysqltest/tutorial.xml 2006-03-15 17:17:26 UTC (rev 1590) @@ -10,19 +10,19 @@ <title>Tutorial</title> <para> - [This chapter is based on the information that was in the Wiki] + [This chapter is based on the information that was in the Wiki, so + some of it probably is outdated.] </para> <para> This chapter provides a tutorial on running and writing test cases - for the MySQL test framework. It gives a brief description of how to - write test cases and the language used to write them. Normally, you - run the test suite during the development process to ensure that - your changes do not cause existing test cases to break. You can also - write new test cases or add tests to existing cases. This happens - when you fix a bug (so that the bug cannot reappear later without - being detected) or when you add new capabilities to the server or - other MySQL programs. + for the MySQL test framework. Normally, you run the test suite + during the development process to ensure that your changes do not + cause existing test cases to break. You can also write new test + cases or add tests to existing cases. This happens when you fix a + bug (so that the bug cannot reappear later without being detected) + or when you add new capabilities to the server or other MySQL + programs. </para> <section id="tutorial-running-test-cases"> @@ -32,13 +32,15 @@ <para> Normally, you run the test suite either from within a source tree (after MySQL has been built), or on a host where the MySQL server - distribution has been installed. A source tree has a - <filename>mysql-test</filename> directory, and an installed server - distribution also should have a <filename>mysql-test</filename> - directory. To run tests, you should be located in the - <filename>mysql-test</filename> directory. The program that runs - the test suite, <command>mysql-test-run</command>, will figure out - whether you are in a source tree or an installed directory tree. + distribution has been installed. To run tests, you should be + located in the <filename>mysql-test</filename> directory of your + source tree or installed distribution. In a source distribution, + <filename>mysql-test</filename> is under the root of the source + tree. In a binary distribution, the location of + <filename>mysql-test</filename> depends on the distribution + layout. The program that runs the test suite, + <command>mysql-test-run</command>, will figure out whether you are + in a source tree or an installed directory tree. </para> <para> @@ -53,13 +55,6 @@ </programlisting> <para> - In a source distribution, <filename>mysql-test</filename> is under - the root of the source tree. In a binary distribution, the - location of <filename>mysql-test</filename> depends on the - distribution layout. - </para> - - <para> <command>mysql-test-run</command> accepts options on the command line. For example: </para> @@ -74,12 +69,17 @@ </para> <para> - Normally, <command>mysql-test-run</command> exits if a test case + By default, <command>mysql-test-run</command> exits if a test case fails. <option>--force</option> causes execution to continue regardless of test case failure. </para> <para> + For a full list of the supported options, see + <xref linkend="mysql-test-run"/>. + </para> + + <para> Test case files have names like <filename>t/<replaceable>test_name</replaceable>.test</filename>, where <replaceable>test_name</replaceable> is the name of the test @@ -116,22 +116,19 @@ </programlisting> <para> - Please view the <filename>README</filename> file in the - <filename>mysql-test</filename> directory for more details about - how to invoke the <command>mysql-test-run</command> test script. <command>mysql-test-run</command> sets up the environment for calling the <command>mysqltest</command> program and starting a server, and so forth. <command>mysqltest</command> handles - operations such as reading the test case language text, starting - managers, creating connections, and sending SQL statements to a - server. + operations such as reading input from the test case file, starting + managers, creating server connections, and sending SQL statements + to servers. </para> <para> The test case language is a mix of commands that the <command>mysqltest</command> program understands and SQL statements. Input that <command>mysqltest</command> doesn't - understand is assumed to be SQL statements and are sent to the + understand is assumed to be SQL statements to be sent to the database server. This makes the test case language familiar to those that know how to write SQL and powerful enough to add the control needed to write test cases. @@ -191,12 +188,13 @@ The basic principle is that evaluation of the output resulting from running a test case is based on comparison of that output with the expected result. This is just a <command>diff</command> - comparison between the output and a file that the test writer - provides. This simplistic method of comparison does not by itself - provide any way to handle variation in the output that may occur - when a test is run at different times. However, the test language - provides some commands for postprocessing result output before the - comparison occurs. This enables you to manage expected variations. + comparison between the output and an expected-result file that the + test writer provides. This simplistic method of comparison does + not by itself provide any way to handle variation in the output + that may occur when a test is run at different times. However, the + test language provides commands for postprocessing result output + before the comparison occurs. This enables you to manage expected + variation. </para> <para> @@ -349,7 +347,9 @@ sequence</quote> (that is, a number of individual tests that are grouped together in the same test file). </para> + </listitem> + <listitem> <para> A <quote>command</quote> is an input test that <command>mysqltest</command> recognizes and executes itself. A @@ -406,9 +406,8 @@ </para> <para> - The input syntax for <command>mysqltest</command> test cases is - discussed in detail in - <xref linkend="mysqltest-input-conventions"/>. + <xref linkend="mysqltest-input-conventions"/>, discusses in detail + the input syntax for <command>mysqltest</command> test cases. </para> <section id="tutorial-sample-test-case"> @@ -445,8 +444,8 @@ <para> We create the table <literal>t1</literal> and try some operations. The creation of the table and first insertion will - not create any warnings but the last line will. We collect the - output from running this test and add this to the result file: + not create any warnings, but the last line will. The output that + results from running the test looks like this: </para> <programlisting> @@ -460,13 +459,17 @@ </programlisting> <para> - Note that we see the statements as well in the result. This can + Note that the result includes not only the output from SQL + statements, but the statements themselves. Statement logging can be disabled with the <literal>disable_query_log</literal> test - language command. There are several options how to control the - amount of output from running the tests. If there was an failure - this will be reported to the screen. You can see the actual - output from the last unsuccessful run of the test case in the - test case reject file + language command. There are several options for controlling the + amount of output from running the tests. + </para> + + <para> + If there was a test failure, it will be reported to the screen. + You can see the actual output from the last unsuccessful run of + the test case in the test case reject file <filename>r/<replaceable>test_name</replaceable>.reject</filename>. </para> @@ -477,11 +480,11 @@ <title>Naming Rules for Database Objects</title> <para> - The test cases might be used to run against a production server - (generally, we will not do that, but our customers might). To + It is possible to run test cases against a production server. + (Generally, we will not do that, but our customers might.) To reduce the risk that running the test suite alters or destroys - important tables, views, or other objects, you should name them - using names like this: + important tables, views, or other objects, you should create + them using names like this: </para> <programlisting> @@ -490,9 +493,9 @@ </programlisting> <para> - Look in the existing test cases for examples of this. Of course, - you can name columns and other objects inside tables as you - wish. + Look in the existing test cases for examples of how to name + objects. Of course, you can name columns and other objects + inside tables as you wish. </para> <para> @@ -510,7 +513,7 @@ <para> For efficiency, the <command>mysqltest</command> test engine does not start with a clean new database for running each test - case. So a test case generally starts with a <quote>cleaning up + case, so a test case generally starts with a <quote>cleaning up section.</quote> Assume that our test case will use two tables named <literal>t1</literal> and <literal>t2</literal>. The test case should begin by making sure that any old tables with those @@ -530,15 +533,15 @@ The <literal>disable_warnings</literal> command instructs the test engine not to log any warnings until <literal>enable_warnings</literal> or the test case is ended. - For some unknown reason, MySQL chooses to write a warning if the - table <literal>t1</literal> or <literal>t2</literal> exists. - Surrounding this part of the test case with the disable/enable - warnings makes its output the same whether or not the tables - exists before the test is started. After ensuring that the - tables do not exist, we are free to put in any SQL statements - that create and use the tables <literal>t1</literal> and - <literal>t2</literal>. The test case should also clean up at the - end of the test by dropping any tables that it creates. + (MySQL generates a warning if the table <literal>t1</literal> or + <literal>t2</literal> does not exist.) Surrounding this part of + the test case with the disable/enable warnings makes its output + the same regardless of whether the tables exist before the test + is started. After ensuring that the tables do not exist, we are + free to put in any SQL statements that create and use the tables + <literal>t1</literal> and <literal>t2</literal>. The test case + should also clean up at the end of the test by dropping any + tables that it creates. </para> <para> @@ -574,15 +577,15 @@ <title>Generating a Test Case Result File</title> <para> - Note that the test code we just wrote contains no checks of the - result. The test will report a failure for one of two reasons + The test code we just wrote contains no checks of the result. + The test will report a failure for one of two reasons: </para> <itemizedlist> <listitem> <para> - An individual SQL statement failed with an error + An individual SQL statement fails with an error </para> </listitem> @@ -614,9 +617,9 @@ </para> <para> - This creates a file named + Running the test as shown creates a file named <filename>mysql-test/r/foo.result</filename>, also called the - <quote>protocol file.</quote> The file has this content + <quote>protocol file.</quote> The file has this content: </para> <programlisting> @@ -639,11 +642,11 @@ </programlisting> <para> - If we look at this result file, it contains the commands in the - <filename>foo.test</filename> file together with the output from - the <literal>SELECT</literal> statements. The output for each - statement includes a row of column headings followed by data - rows. Rows have columns separated by Tab characters. + If we look at this result file, it contains the statements in + the <filename>foo.test</filename> file together with the output + from the <literal>SELECT</literal> statements. The output for + each statement includes a row of column headings followed by + data rows. Rows have columns separated by Tab characters. </para> <para> @@ -666,12 +669,12 @@ A good test suite not only tests that operations succeed as they ought, but also that they fail as they ought. For example, if a statement is illegal, the server should reject it with an error - message. The test suite should verify that the statement fails, + message. The test suite should verify that the statement fails and that it fails with the proper error message. </para> <para> - The test engine let you specify <quote>expected + The test engine enables you to specify <quote>expected failures.</quote> Let's say that after we create <literal>t1</literal>, we try to create it again without dropping it first: @@ -703,8 +706,8 @@ <para> [Q: word-police note: <literal>create table</literal> is a statement, not a query. <quote>statement</quote> would be a - better term here in the error, because it applies to both - queries and non-queries.] + better term here in the error message, because it applies to + both queries and non-queries.] </para> <programlisting> @@ -713,8 +716,8 @@ </programlisting> <para> - To handle this and indicate that indeed we do expect MySQL error - code 1050 to occur, we can put an <literal>error</literal> + To handle this error and indicate that indeed we do expect MySQL + error code 1050 to occur, we can put an <literal>error</literal> command before the second <literal>create table</literal> statement: </para> @@ -734,7 +737,7 @@ </programlisting> <para> - In this case, the result shows the statement that results in the + In this case, the result shows the statement that causes the error, together with the resulting error message. The fact that <command>mysqltest</command> does not terminate and that the error message becomes part of the result indicates that the @@ -768,42 +771,37 @@ <title>Controlling the Information Produced by a Test Case</title> - <section id="tutorial-more-information"> + <para> + [Q: Isn't the first sentence inaccurate? For example, the result + also contains output from <literal>echo</literal> commands, and + <literal>exec</literal> (but apparently not from + <literal>system</literal>)?] + </para> - <title>Getting More Information for Checking the Test Result</title> + <para> + By default, the test engine produces output only from + <literal>select</literal>, <literal>show</literal>, and other + SQL statements that you expect to produce output (that is, + statements that create a result set). But it can be instructed + to be more or less verbose, including other data as well. + </para> - <para> - [Q: Isn't the first sentence inaccurate? For example, the - result also contains output from <literal>echo</literal> - commands, and <literal>exec</literal> (but apparently not from - <literal>system</literal>)?] - </para> + <para> + Suppose that we want to include in the result the number of rows + affected by or returned by SQL statements. To do this, add the + following line to the case case file preceding those statements: + </para> - <para> - The test engine by default only output things from - <literal>select</literal>, <literal>show</literal>, and other - SQL statements that you expect to produce output (that is, - statements that create a result set). But it can be instructed - to be more or less verbose, including other data as well. - </para> - - <para> - Suppose that we want to include in the result the number of - rows affected by or returned by SQL statements. To do this, - add the following line to the case case file preceding those - statements: - </para> - <programlisting> --enable_info </programlisting> - <para> - After rerunning the test by invoking - <command>mysql-test-run</command> with the - <option>--record</option> option to record the new result, the - result file will contain more information: - </para> + <para> + After rerunning the test by invoking + <command>mysql-test-run</command> with the + <option>--record</option> option to record the new result, the + result file will contain more information: + </para> <programlisting> drop table if exists t1,t2; @@ -830,52 +828,50 @@ affected rows: 0 </programlisting> - <para> - To turn off the affected-rows reporting, add this command to - the test file: - </para> + <para> + To turn off the affected-rows reporting, add this command to the + test file: + </para> <programlisting> --disable_info </programlisting> - <para> - Options can in general be enabled and disabled for different - parts of the test file. Suppose that we are interested in the - internals of the database as well. We can then enable the - display of query metadata using - <literal>enable_metadata</literal>. With this option enabled, - the test output is a bit verbose. However, as mentioned - earlier, the option can be enabled and disabled selectively so - that it is enabled only for those parts of the test case where - it interests you to know more. - </para> + <para> + Options can in general be enabled and disabled for different + parts of the test file. Suppose that we are interested in the + internals of the database as well. We can then enable the + display of query metadata using + <literal>enable_metadata</literal>. With this option enabled, + the test output is a bit verbose. However, as mentioned earlier, + the option can be enabled and disabled selectively so that it is + enabled only for those parts of the test case where it interests + you to know more. + </para> - </section> + <title>Reducing the Amount of Test Case Information</title> - <section id="tutorial-less-information"> + <para> + [Q: Is the following paragraph inaccurate? Don't the _query_log + commands apply only to logging SQL statements (and not to "all + commands in general")?] + </para> - <title>Reducing the Amount of Test Case Information</title> + <para> + If you perform an operation for which you have no interest in + seeing the statements logged to the result, you can disable + statement logging. For example, you might be initializing a + table where you don't really expect a failure, and you are not + interested in seeing the initialization statements in the test + result. You can use the <literal>disable_query_log</literal> + command to temporarily disable recording of the command lines, + and enable recording again with + <literal>enable_query_log</literal>. You can disable the + recording of the output from executing commands using + <literal>disable_result_log</literal> and enable again with + <literal>enable_result_log</literal>. + </para> - <para> - [Q: is the following paragraph inaccurate? Don't the - _query_log commands apply only to logging SQL statements (and - not to "all commands in general")?] - </para> - - <para> - You might be initializing a table, where we don't really - expect a failure. You can then temporarily disable the - recording of the command lines executed with - <literal>disable_query_log</literal> and then enable the - recording again with <literal>enable_query_log</literal>. You - can disable the recording of the output from executing - commands using <literal>disable_result_log</literal> and - enable again with <literal>enable_result_log</literal>. - </para> - - </section> - </section> <section id="tutorial-output-postprocessing"> @@ -888,7 +884,7 @@ factors such as the time of day, differences in how program binaries are compiled, the operating system, and so forth. For example, if the result contains the current date and time, the - test engine has no way of verifying that it is correct. + test engine has no way to verify that the result is correct. </para> <para> @@ -896,9 +892,9 @@ according to external factors, or perhaps there is a part of a result that you simply do not care about. <command>mysqltest</command> provides commands that enable you - to postprocess test output to a more standard form. That way, + to postprocess test output to a more standard form, so that output variation across test runs will not trigger a result - mismatch + mismatch. </para> <para> @@ -929,7 +925,7 @@ <para> Most likely, a failure will occur and <command>mysql-test-run</command> will display the difference - between the expected result what we actually got, like this: + between the expected result and what we actually got, like this: </para> <programlisting> @@ -978,7 +974,7 @@ eliminate this mismatch is by using the <literal>replace_column</literal> command. The duration of the effect of this command is the next SQL statement, so we need one - before each select statement: this: + before each <literal>select</literal> statement: this: </para> <programlisting> @@ -992,7 +988,7 @@ <para> Note that <literal>SECONDS</literal> could be any string. Its - purpose is only to map variable output onto a constant value. If + only purpose is to map variable output onto a constant value. If we record the test result again, we will succeed each time we run the test after that. The result file will look like this: </para> @@ -1025,18 +1021,6 @@ affected rows: 2 </programlisting> - <para> - [Q: What does the following paragraph add that has not already - been stated?] - </para> - - <para> - If the result may vary, but just between some known values, you - can use <literal>replace_result</literal> to match the different - results, and replace them with a common string, making the - result file the same even if the result varies some. - </para> - </section> <section id="tutorial-server-options"> @@ -1044,7 +1028,7 @@ <title>Setting Server Options</title> <para> - Many server options can be set from within the test case using + Many server options can be set from within test cases by using statements such as these: </para> @@ -1055,9 +1039,9 @@ <para> But sometimes you need to restart the server to use some options - in the form of extra command line arguments. The test engine - will restart the server just before running a test case, if - there exists a file + in the form of extra command-line options. + <command>mysql-test-run</command> will restart the server just + before running a test case, if there exists a file <filename>mysql-test/t/<replaceable>test_name</replaceable>-master.opt</filename> with the options that you want to use. After the test case is completed, the server will be restarted again, with the default
| Thread | ||
|---|---|---|
| • svn commit - mysqldoc@docsrva: r1590 - in trunk: . mysqltest | paul | 15 Mar |
© 1995-2008 MySQL AB, 2008-2009 Sun Microsystems, Inc.
Page generated in 0.808 sec. using MySQL 5.1.14-beta-log