Hi Ingo,
Lots of nice new comments and less silent errors. Only a couple of style
issues below and perhaps the bugfix could have been in seperate commit.
Approved to push.
On Wed, 2007-07-25 at 12:08 +0200, Ingo Struewing wrote:
> Below is the list of changes that have just been committed into a local
> 5.1 repository of istruewing. When istruewing does a push these changes will
> be propagated to the main repository and, within 24 hours after the
> push, to the public repository.
> For information on how to access the public repository
> see http://dev.mysql.com/doc/mysql/en/installing-source-tree.html
>
> ChangeSet@stripped, 2007-07-25 12:08:30+02:00, istruewing@stripped +5 -0
> Bug#29838 - myisam corruption using concurrent select ... and update
>
> When using concurrent insert with parallel index reads, it could
> happen that reading sessions found keys that pointed to records
> yet to be written to the data file. The result was a report of
> a corrupted table. But it was false alert.
>
> When inserting a record in a table with indexes, the keys are
> inserted into the indexes before the record is written to the data
> file. When the insert happens concurrently to selects, an
> index read can find a key that references the record that is not
> yet written to the data file. To avoid any access to such record,
> the select saves the current end of file position when it starts.
> Since concurrent inserts are always appended at end of the data
> file, the select can easily ignore any concurrently inserted record.
>
> The problem was that the ignore was only done for non-exact key
> searches (partial key or using >, >=, < or <=).
>
> The fix is to ignore concurrently inserted records also for
> exact key searches.
>
> No test case. Concurrent inserts cannot be tested with the test
> suite. Test cases are attached to the bug report.
>
> The real fix is in mi_rkey.c. The other changes are for improved
> diagnostic. They were useful to detect the above problem and some
> follow-up problems. They should be useful for future problems too.
>
> storage/myisam/mi_cache.c@stripped, 2007-07-25 12:08:25+02:00, istruewing@stripped
> +55 -10
> Bug#29838 - myisam corruption using concurrent select ... and update
> Added logging of a message on cache errors.
> Added comments.
>
> storage/myisam/mi_dynrec.c@stripped, 2007-07-25 12:08:26+02:00, istruewing@stripped
> +83 -7
> Bug#29838 - myisam corruption using concurrent select ... and update
> Added logging of messages on record errors.
> Added marking table crashed on fatal record errors.
> Avoided reading of block headers past enf of file.
> Added DBUG_PRINTs.
> Added comments.
>
> storage/myisam/mi_info.c@stripped, 2007-07-25 12:08:26+02:00, istruewing@stripped
> +17 -0
> Bug#29838 - myisam corruption using concurrent select ... and update
> Added a function for logging of a message to the error log.
>
> storage/myisam/mi_rkey.c@stripped, 2007-07-25 12:08:26+02:00, istruewing@stripped
> +46 -33
> Bug#29838 - myisam corruption using concurrent select ... and update
> Fixed mi_rkey() to always ignore records beyond saved eof.
>
> storage/myisam/myisamdef.h@stripped, 2007-07-25 12:08:26+02:00,
> istruewing@stripped +1 -0
> Bug#29838 - myisam corruption using concurrent select ... and update
> Added declaration for a function for logging of a message
> to the error log.
>
> diff -Nrup a/storage/myisam/mi_cache.c b/storage/myisam/mi_cache.c
> --- a/storage/myisam/mi_cache.c 2007-05-10 11:59:32 +02:00
> +++ b/storage/myisam/mi_cache.c 2007-07-25 12:08:25 +02:00
> @@ -43,6 +43,7 @@ int _mi_read_cache(IO_CACHE *info, uchar
> uchar *in_buff_pos;
> DBUG_ENTER("_mi_read_cache");
>
> + /* If read position is below cache contents, read directly. */
> if (pos < info->pos_in_file)
> {
> read_length=length;
> @@ -56,6 +57,10 @@ int _mi_read_cache(IO_CACHE *info, uchar
> pos+=read_length;
> buff+=read_length;
> }
> + /*
> + If read position is within cache, copy as much as possible.
> + Set in_buff_length to how much has been copied from cache.
> + */
> if (pos >= info->pos_in_file &&
> (offset= (my_off_t) (pos - info->pos_in_file)) <
> (my_off_t) (info->read_end - info->request_pos))
> @@ -64,14 +69,21 @@ int _mi_read_cache(IO_CACHE *info, uchar
> in_buff_length= min(length, (size_t) (info->read_end-in_buff_pos));
> memcpy(buff,info->request_pos+(uint) offset,(size_t) in_buff_length);
> if (!(length-=in_buff_length))
> + {
> + /* If everything was in cache, we are done. */
> DBUG_RETURN(0);
> + }
> pos+=in_buff_length;
> buff+=in_buff_length;
> }
> else
> + {
> + /* Request is completely past cache. */
> in_buff_length=0;
> + }
> if (flag & READING_NEXT)
> {
> + /* Basically sequential read. Read new contents into cache. */
> if (pos != (info->pos_in_file +
> (uint) (info->read_end - info->request_pos)))
> {
> @@ -82,26 +94,59 @@ int _mi_read_cache(IO_CACHE *info, uchar
> else
> info->read_pos=info->read_end; /* All block used */
> if (!(*info->read_function)(info,buff,length))
> + {
> + /* If sufficient stuff read, we are done. */
> DBUG_RETURN(0);
> + }
> read_length=info->error;
> }
> else
> {
> + /* Out of band read. Bypass cache. */
> info->seek_not_done=1;
> if ((read_length=my_pread(info->file,buff,length,pos,MYF(0))) == length)
> + {
> + /* If sufficient stuff read, we are done. */
> DBUG_RETURN(0);
> + }
> }
> - if (!(flag & READING_HEADER) || (int) read_length == -1 ||
> - read_length+in_buff_length < 3)
> - {
> - DBUG_PRINT("error",
> - ("Error %d reading next-multi-part block (Got %d bytes)",
> - my_errno, (int) read_length));
> - if (!my_errno || my_errno == -1)
> - my_errno=HA_ERR_WRONG_IN_RECORD;
> - DBUG_RETURN(1);
> - }
> + /*
> + Insufficient contents read. read_length tells how much we got.
> + This can be ok if we try to read the header (first block of record)
> + and did not get a read error.
> + */
> + if (!(flag & READING_HEADER) || (int) read_length == -1)
> + goto err;
> + /*
> + If we have nothing (EOF), then this is ok, but we must still
> + return non-zero.
> + */
> + if (!read_length && !in_buff_length)
> + goto eof;
> + /*
> + If we got something, then we must have >= 3 bytes (the smallest
> + possible block header).
> + */
> + if (read_length + in_buff_length < 3)
> + goto err;
> + /*
> + We may have got a valid header, but less than we expected. Clear up
> + to the maximum block header size to avoid misinterpretation.
> + */
> bzero(buff+read_length,MI_BLOCK_INFO_HEADER_LENGTH - in_buff_length -
> read_length);
> DBUG_RETURN(0);
> +
> +err:
> + DBUG_PRINT("error",
> + ("Error %d reading next-multi-part block (Got %d bytes)",
> + my_errno, (int) read_length));
> + /* If not a read error, then we have a crashed file. */
> + if (!my_errno || my_errno == -1)
> + {
> + mi_log_message("_mi_read_cache read_length");
> + my_errno=HA_ERR_WRONG_IN_RECORD;
Coding standards says that for assignments, there should be only one
space after the equal sign.
> + }
> +eof:
> + DBUG_RETURN(1);
> } /* _mi_read_cache */
> diff -Nrup a/storage/myisam/mi_dynrec.c b/storage/myisam/mi_dynrec.c
> --- a/storage/myisam/mi_dynrec.c 2007-05-10 11:59:32 +02:00
> +++ b/storage/myisam/mi_dynrec.c 2007-07-25 12:08:26 +02:00
> @@ -365,6 +365,8 @@ static int _mi_find_writepos(MI_INFO *in
> BLOCK_DELETED))
> {
> DBUG_PRINT("error",("Delete link crashed"));
> + mi_log_message("_mi_find_writepos delete link crashed");
> + mi_mark_crashed(info);
> my_errno=HA_ERR_WRONG_IN_RECORD;
> DBUG_RETURN(-1);
> }
> @@ -422,7 +424,10 @@ static bool unlink_deleted_block(MI_INFO
> /* Unlink block from the previous block */
> if (!(_mi_get_block_info(&tmp,info->dfile,block_info->prev_filepos)
> & BLOCK_DELETED))
> + {
> + mi_mark_crashed(info);
> DBUG_RETURN(1); /* Something is wrong */
> + }
> mi_sizestore(tmp.header+4,block_info->next_filepos);
> if (info->s->file_write(info,(char*) tmp.header+4,8,
> block_info->prev_filepos+4, MYF(MY_NABP)))
> @@ -432,7 +437,10 @@ static bool unlink_deleted_block(MI_INFO
> {
> if (!(_mi_get_block_info(&tmp,info->dfile,block_info->next_filepos)
> & BLOCK_DELETED))
> + {
> + mi_mark_crashed(info);
> DBUG_RETURN(1); /* Something is wrong */
> + }
> mi_sizestore(tmp.header+12,block_info->prev_filepos);
> if (info->s->file_write(info,(char*) tmp.header+12,8,
> block_info->next_filepos+12,
> @@ -476,6 +484,8 @@ static int update_backward_delete_link(M
> {
> MI_BLOCK_INFO block_info;
> DBUG_ENTER("update_backward_delete_link");
> + DBUG_PRINT("enter", ("delete_block: %lu filepos: %lu",
> + (ulong) delete_block, (ulong) filepos));
>
> if (delete_block != HA_OFFSET_ERROR)
> {
> @@ -490,6 +500,7 @@ static int update_backward_delete_link(M
> }
> else
> {
> + mi_mark_crashed(info);
> my_errno=HA_ERR_WRONG_IN_RECORD;
> DBUG_RETURN(1); /* Wrong delete link */
> }
> @@ -508,6 +519,9 @@ static int delete_dynamic_record(MI_INFO
> int error;
> my_bool remove_next_block;
> DBUG_ENTER("delete_dynamic_record");
> + DBUG_PRINT("enter", ("dellink: %lu filepos: %lu second_read: %u",
> + (ulong) info->s->state.dellink, (ulong) filepos,
> + second_read));
>
> /* First add a link from the last block to the new one */
> error= update_backward_delete_link(info, info->s->state.dellink, filepos);
> @@ -522,13 +536,16 @@ static int delete_dynamic_record(MI_INFO
> (length=(uint) (block_info.filepos-filepos) +block_info.block_len) <
> MI_MIN_BLOCK_LENGTH)
> {
> + mi_mark_crashed(info);
> my_errno=HA_ERR_WRONG_IN_RECORD;
> DBUG_RETURN(1);
> }
> - /* Check if next block is a delete block */
> + /* Check if next block is a delete block, but only if not past EOF. */
> del_block.second_read=0;
> remove_next_block=0;
> - if (_mi_get_block_info(&del_block,info->dfile,filepos+length) &
> + if ((filepos + length + MI_BLOCK_INFO_HEADER_LENGTH <=
> + info->state->data_file_length) &&
> + _mi_get_block_info(&del_block,info->dfile,filepos+length) &
> BLOCK_DELETED && del_block.block_len+length < MI_DYN_MAX_BLOCK_LENGTH)
> {
> /* We can't remove this yet as this block may be the head block */
> @@ -773,7 +790,10 @@ static int update_dynamic_record(MI_INFO
> {
> DBUG_PRINT("error",("Got wrong block info"));
> if (!(error & BLOCK_FATAL_ERROR))
> + {
> + mi_mark_crashed(info);
> my_errno=HA_ERR_WRONG_IN_RECORD;
> + }
> goto err;
> }
> length=(ulong) (block_info.filepos-filepos) + block_info.block_len;
> @@ -1249,6 +1269,8 @@ ulong _mi_rec_unpack(register MI_INFO *i
> DBUG_RETURN(found_length);
>
> err:
> + mi_log_message("_mi_rec_unpack err");
> + mi_mark_crashed(info);
> my_errno= HA_ERR_WRONG_IN_RECORD;
> DBUG_PRINT("error",("to_end: 0x%lx -> 0x%lx from_end: 0x%lx -> 0x%lx",
> (long) to, (long) to_end, (long) from, (long) from_end));
> @@ -1353,7 +1375,8 @@ int _mi_read_dynamic_record(MI_INFO *inf
> uchar *to;
> MI_BLOCK_INFO block_info;
> File file;
> - DBUG_ENTER("mi_read_dynamic_record");
> + DBUG_ENTER("_mi_read_dynamic_record");
> + DBUG_PRINT("enter", ("filepos: %lu", (ulong) filepos));
>
> if (filepos != HA_OFFSET_ERROR)
> {
> @@ -1366,24 +1389,47 @@ int _mi_read_dynamic_record(MI_INFO *inf
> {
> /* A corrupted table can have wrong pointers. (Bug# 19835) */
> if (filepos == HA_OFFSET_ERROR)
> + {
> + mi_log_message("_mi_read_dynamic_record offset error");
> goto panic;
> + }
> + /* Wrong code could try to read past EOF. (Bug# 29838) */
> + if (filepos + MI_BLOCK_INFO_HEADER_LENGTH >
> + info->state->data_file_length)
> + {
> + DBUG_PRINT("error", ("filepos: %lu data_file_length: %lu",
> + (ulong) filepos,
> + (ulong) info->state->data_file_length));
> + mi_log_message("_mi_read_dynamic_record eof");
> + goto err;
> + }
> + /* May need to flush write cache first. */
> if (info->opt_flag & WRITE_CACHE_USED &&
> info->rec_cache.pos_in_file < filepos + MI_BLOCK_INFO_HEADER_LENGTH
> &&
> flush_io_cache(&info->rec_cache))
> + {
> + mi_log_message("_mi_read_dynamic_record flush_io_cache error @1");
> goto err;
> + }
> + /* Read block at filepos. */
> info->rec_cache.seek_not_done=1;
> if ((b_type= _mi_get_block_info(&block_info, file, filepos))
> & (BLOCK_DELETED | BLOCK_ERROR | BLOCK_SYNC_ERROR |
> BLOCK_FATAL_ERROR))
> {
> - if (b_type & (BLOCK_SYNC_ERROR | BLOCK_DELETED))
> - my_errno=HA_ERR_RECORD_DELETED;
> - goto err;
> + if (b_type & (BLOCK_SYNC_ERROR | BLOCK_DELETED))
> + my_errno=HA_ERR_RECORD_DELETED;
ditto re coding style.
> + if (my_errno == HA_ERR_WRONG_IN_RECORD)
> + goto panic;
> + goto err;
> }
> if (block_of_record++ == 0) /* First block */
> {
> if (block_info.rec_len > (uint) info->s->base.max_pack_length)
> + {
> + mi_log_message("_mi_read_dynamic_record rec_len");
> goto panic;
> + }
> if (info->s->base.blobs)
> {
> if (!(to=mi_alloc_rec_buff(info, block_info.rec_len,
> @@ -1395,7 +1441,10 @@ int _mi_read_dynamic_record(MI_INFO *inf
> left_length=block_info.rec_len;
> }
> if (left_length < block_info.data_len || ! block_info.data_len)
> + {
> + mi_log_message("_mi_read_dynamic_record left_length");
> goto panic; /* Wrong linked record */
> + }
> /* copy information that is already read */
> {
> uint offset= (uint) (block_info.filepos - filepos);
> @@ -1415,10 +1464,14 @@ int _mi_read_dynamic_record(MI_INFO *inf
> /* read rest of record from file */
> if (block_info.data_len)
> {
> + /* May need to flush write cache first. */
> if (info->opt_flag & WRITE_CACHE_USED &&
> info->rec_cache.pos_in_file < filepos + block_info.data_len
> &&
> flush_io_cache(&info->rec_cache))
> + {
> + mi_log_message("_mi_read_dynamic_record flush_io_cache error @2");
> goto err;
> + }
> /*
> What a pity that this method is not called 'file_pread' and that
> there is no equivalent without seeking. We are at the right
> @@ -1426,7 +1479,10 @@ int _mi_read_dynamic_record(MI_INFO *inf
> */
> if (info->s->file_read(info, (uchar*) to, block_info.data_len,
> filepos, MYF(MY_NABP)))
> + {
> + mi_log_message("_mi_read_dynamic_record file_read error");
> goto panic;
> + }
> left_length-=block_info.data_len;
> to+=block_info.data_len;
> }
> @@ -1442,6 +1498,8 @@ int _mi_read_dynamic_record(MI_INFO *inf
> DBUG_RETURN(-1); /* Wrong data to read */
>
> panic:
> + mi_log_message("_mi_read_dynamic_record panic");
> + mi_mark_crashed(info);
> my_errno=HA_ERR_WRONG_IN_RECORD;
> err:
> VOID(_mi_writeinfo(info,0));
> @@ -1534,8 +1592,11 @@ int _mi_cmp_dynamic_record(register MI_I
> my_errno=HA_ERR_RECORD_CHANGED;
> goto err;
> }
> - } else if (reclength < block_info.data_len)
> + }
> + else if (reclength < block_info.data_len)
> {
> + mi_log_message("_mi_cmp_dynamic_record reclength");
> + mi_mark_crashed(info);
> my_errno=HA_ERR_WRONG_IN_RECORD;
> goto err;
> }
> @@ -1765,7 +1826,11 @@ int _mi_read_rnd_dynamic_record(MI_INFO
> if (my_read(info->dfile,(uchar*) to,block_info.data_len,MYF(MY_NABP)))
> {
> if (my_errno == -1)
> + {
> + mi_log_message("_mi_read_rnd_dynamic_record read error");
> + mi_mark_crashed(info);
> my_errno= HA_ERR_WRONG_IN_RECORD; /* Unexpected end of file */
> + }
> goto err;
> }
> }
> @@ -1792,6 +1857,8 @@ int _mi_read_rnd_dynamic_record(MI_INFO
> DBUG_RETURN(my_errno); /* Wrong record */
>
> panic:
> + mi_log_message("_mi_read_rnd_dynamic_record panic");
> + mi_mark_crashed(info);
> my_errno=HA_ERR_WRONG_IN_RECORD; /* Something is fatal wrong */
> err:
> save_errno=my_errno;
> @@ -1817,7 +1884,10 @@ uint _mi_get_block_info(MI_BLOCK_INFO *i
> VOID(my_seek(file,filepos,MY_SEEK_SET,MYF(0)));
> if (my_read(file,(char*) header,sizeof(info->header),MYF(0)) !=
> sizeof(info->header))
> + {
> + mi_log_message("_mi_get_block_info read size error");
> goto err;
> + }
> }
> DBUG_DUMP("header",(uchar*) header,MI_BLOCK_INFO_HEADER_LENGTH);
> if (info->second_read)
> @@ -1837,7 +1907,10 @@ uint _mi_get_block_info(MI_BLOCK_INFO *i
> if ((info->block_len=(uint) mi_uint3korr(header+1)) <
> MI_MIN_BLOCK_LENGTH ||
> (info->block_len & (MI_DYN_ALIGN_SIZE -1)))
> + {
> + mi_log_message("_mi_get_block_info invalid del blk lgt");
> goto err;
> + }
> info->filepos=filepos;
> info->next_filepos=mi_sizekorr(header+4);
> info->prev_filepos=mi_sizekorr(header+12);
> @@ -1848,7 +1921,10 @@ uint _mi_get_block_info(MI_BLOCK_INFO *i
> (mi_uint4korr(header+12) != 0 &&
> (mi_uint4korr(header+12) != (ulong) ~0 ||
> info->prev_filepos != (ulong) ~0)))
> + {
> + mi_log_message("_mi_get_block_info invalid del blk ptr");
> goto err;
> + }
> #endif
> return return_val | BLOCK_DELETED; /* Deleted block */
>
> diff -Nrup a/storage/myisam/mi_info.c b/storage/myisam/mi_info.c
> --- a/storage/myisam/mi_info.c 2007-04-13 09:43:59 +02:00
> +++ b/storage/myisam/mi_info.c 2007-07-25 12:08:26 +02:00
> @@ -130,3 +130,20 @@ void mi_report_error(int errcode, const
> DBUG_VOID_RETURN;
> }
>
> +
> +
> +/**
> + @brief Write a message to the error log.
> +
> + @param[in] message message
> +*/
> +
> +void mi_log_message(const char *message)
> +{
> + size_t lgt;
> + DBUG_ENTER("mi_log_message");
> + DBUG_PRINT("enter",("message '%s'", message));
> +
> + my_printf_error(HA_ERR_GENERIC, "%.200s", MYF(0), message);
> + DBUG_VOID_RETURN;
> +}
> diff -Nrup a/storage/myisam/mi_rkey.c b/storage/myisam/mi_rkey.c
> --- a/storage/myisam/mi_rkey.c 2007-05-10 11:59:32 +02:00
> +++ b/storage/myisam/mi_rkey.c 2007-07-25 12:08:26 +02:00
> @@ -93,43 +93,56 @@ int mi_rkey(MI_INFO *info, uchar *buf, i
> if (!_mi_search(info, keyinfo, key_buff, use_key_length,
> myisam_read_vec[search_flag],
> info->s->state.key_root[inx]))
> {
> - /*
> - If we searching for a partial key (or using >, >=, < or <=) and
> - the data is outside of the data file, we need to continue searching
> - for the first key inside the data file
> - */
> - if (info->lastpos >= info->state->data_file_length &&
> - (search_flag != HA_READ_KEY_EXACT ||
> - last_used_keyseg != keyinfo->seg + keyinfo->keysegs))
> + if (info->lastpos >= info->state->data_file_length)
> {
> - do
> + /*
> + If we searching for a partial key (or using >, >=, < or <=)
> and
> + the data is outside of the data file, we need to continue searching
> + for the first key inside the data file
> + */
> + if (search_flag != HA_READ_KEY_EXACT ||
> + last_used_keyseg != keyinfo->seg + keyinfo->keysegs)
> + {
> + do
> + {
> + uint not_used[2];
> + /*
> + Skip rows that are inserted by other threads since we got a lock
> + Note that this can only happen if we are not searching after an
> + full length exact key, because the keys are sorted
> + according to position
> + */
> + if (_mi_search_next(info, keyinfo, info->lastkey,
> + info->lastkey_length,
> + myisam_readnext_vec[search_flag],
> + info->s->state.key_root[inx]))
> + break;
> + /*
> + Check that the found key does still match the search.
> + _mi_search_next() delivers the next key regardless of its
> + value.
> + */
> + if (search_flag == HA_READ_KEY_EXACT &&
> + ha_key_cmp(keyinfo->seg, key_buff, info->lastkey,
> + use_key_length, SEARCH_FIND, not_used))
> + {
> + my_errno= HA_ERR_KEY_NOT_FOUND;
> + info->lastpos= HA_OFFSET_ERROR;
> + break;
> + }
> + } while (info->lastpos >= info->state->data_file_length);
> + }
> + else
> {
> - uint not_used[2];
> - /*
> - Skip rows that are inserted by other threads since we got a lock
> - Note that this can only happen if we are not searching after an
> - full length exact key, because the keys are sorted
> - according to position
> - */
> - if (_mi_search_next(info, keyinfo, info->lastkey,
> - info->lastkey_length,
> - myisam_readnext_vec[search_flag],
> - info->s->state.key_root[inx]))
> - break;
> /*
> - Check that the found key does still match the search.
> - _mi_search_next() delivers the next key regardless of its
> - value.
> + We cannot use rows that are inserted by other threads since
> + we got a lock ("concurrent inserts"). The record may not
> + even be present yet. Keys are inserted before the record is.
> + (Bug #29838)
> */
> - if (search_flag == HA_READ_KEY_EXACT &&
> - ha_key_cmp(keyinfo->seg, key_buff, info->lastkey,
> use_key_length,
> - SEARCH_FIND, not_used))
> - {
> - my_errno= HA_ERR_KEY_NOT_FOUND;
> - info->lastpos= HA_OFFSET_ERROR;
> - break;
> - }
> - } while (info->lastpos >= info->state->data_file_length);
> + my_errno= HA_ERR_KEY_NOT_FOUND;
> + info->lastpos= HA_OFFSET_ERROR;
> + }
> }
> }
> }
> diff -Nrup a/storage/myisam/myisamdef.h b/storage/myisam/myisamdef.h
> --- a/storage/myisam/myisamdef.h 2007-05-10 11:59:33 +02:00
> +++ b/storage/myisam/myisamdef.h 2007-07-25 12:08:26 +02:00
> @@ -712,6 +712,7 @@ extern void _myisam_log_record(enum myis
> const uchar *record,my_off_t filepos,
> int result);
> extern void mi_report_error(int errcode, const char *file_name);
> +extern void mi_log_message(const char *message);
> extern my_bool _mi_memmap_file(MI_INFO *info);
> extern void _mi_unmap_file(MI_INFO *info);
> extern uint save_pack_length(uint version, uchar *block_buff, ulong length);
>
Regards,
--
Antony T Curtis, Senior Software Developer
MySQL Inc, www.mysql.com
SIP: 4468@stripped