2011-01-25 03:25, Alfranio Correia wrote:
> Hi all,
>
> On 01/24/2011 10:10 AM, anders wrote:
>
>> On Mon, 2011-01-24 at 14:06 +0800, Daogang Qu wrote:
>>
>>> 2011-01-21 19:12, Alfranio Correia wrote:
>>>
>>>> STATUS
>>>> ------
>>>>
>>>> Not approved as there are several problems
>>>> with the current design and implementation.
>>>>
>>>> REQUESTS
>>>> --------
>>>>
>>>> 1. It is missing a set of variables to implement an automated mechanism
> to
>>>> pre-allocate binary logs. For example,
>>>>
>>>> 1.1 How do I know if there is any pre-allocated file that is not in
> use?
>>>>
>>>>
>>> The unused files which are preallocated previously will be used when
>>> server restart.
>>> Why we need to know the unused file? We can know it by mysqlbinlog
>>> client tool
>>> if we really want to know it.
>>>
>>>> 1.2 How do I get its maximum size?
>>>>
>>>>
>>> The maximum size is limited by max-binlog-size, The file size is specified
>>> when preallocating and it's equal to max-binlog-size by default.
>>>
>> But It is possible that users use a alloc-size different from
>> max-binlog-size.
>> I wander why we need to know the size of pre-allocated file?
>> MYSQL_BIN_LOG will expand the file if its capacity is not enough,
>> and MYSQL_BIN_LOG can truncate binlog files without knowing the size.
>>
>>
>>
> Try to answer the following question: How can I automate the task of
> pre-allocating files? To do that, you need to know which files are pre-allocated,
> their size, and the used size of the current file, etc, etc...
>
> In other words, what I want is to provide the means to build an automated
> routine that will pre-allocate files. We cannot force users to periodically check
> the servers to see if they are running out of pre-allocated files. Besides, it is
> not
> possible to provide such routine, files will be created by demand thus generating
> a negative impact on performance.
>
The user don't need the above info base on the
assumption that user just can do static preallocation
when server is off. The automatic
preallocation will be used when server is on.
>
>
>>>> 1.3 How do I get the actual size of the current file?
>>>>
>>>>
>>> The "ulonglong actual_size" member variable in MYSQL_BIN_LOG class is
> introduced for
>>> managing the latest binlog's actual size.
>>>
>>>
>>>
>>>> Note this is not a complete list and we may provide different
> alternatives to
>>>> achieve the same goal. Look at the questions above as just examples.
>>>>
>>>> 2. Why do we need a separate client to pre-allocate binary logs?
>>>>
>>>>
>>> One issue is that in some OS/filesystems (i.e. ext3 on RHEL5)
> posix_fallocate()
>>> is defined but does zero filling internally, which will take a lot of time
> and
>>> massive writes. To avoid this scenario, we introduced the separate client
> for
>>> pre-allocating binary logs before mysql server start.
>>>
>>>
>>>> I don't buy the argument of performance as described in the WL because
> the time
>>>> to pre-allocate the files is taking into account to do the performance
> assessment
>>>> and creating the client does not fix anything.
>>>>
>>>>
>>> The time of static pre-allocate files should not be taken into account to
>>> performance assessment. Because it's done before the server start.
>>>
>>>
>>>
>>>
>>>> The issue here should be seen from a different perspective:
> "Pre-allocating files
>>>> will increase disk activity and as such will interfere in the
> performance". So
>>>> the files need to be created when the server is idle and when there is
> really a
>>>> need to do so.
>>>>
>>>>
>>> These files are preallocated by the client tool when the server is stopped
>>> or idle.
>>>
>>>> Note that I am ok with the client provided item 3 is not a problem.
> However I
>>>> prefer a new command.
>>>>
>>>>
>>> What's your suggestion for the new command?
>>>
>>>> 3. Isn't there any race condition between rotation and pre-allocation?
>>>>
>>>> server ---> decides to use master.X
>>>>
>>>> mysqlbinlogalloc ---> creates master.X
>>>>
>>>> server ---> fails while creating master.X (booom)
>>>>
>> Not sure if it will fail, but this case exists, I think.
>> A way to avoid this case is that mysqlbinlogalloc create the file with a
>> temporary name and then rename it.
>>
> Sure, we need to check this.
>
See above and reply to your and Libing's previous comments in last mails.
>
>>>>
>>>>
>>> No. Static and automatic preallocation are compatible. One will not
>>> preallocate the file if another is a preallocating it. So no conflict here.
>>>
>> [snip]
>>
>>>
>>>> 4. Using the binlog_preallocate to decide if either the old or new
> behavior should
>>>> be used is not a good idea. We may pre-allocate files and start the
> server with
>>>> binlog_preallocate<> 0 and then forget and restart the
> server with binlog_preallocate
>>>> = 0. In this case, we are going to have dangling files consuming
> lots of disk space.
>>>>
>>>> If We can I think we must avoid situations where an user's mistake
> may cause problems.
>>>>
>>>>
>>> Maybe. But how to avoid this?
>>>
>>
>>>> 5. Using binlog_preallocate> 1 to automatically pre-allocate binary
> log files is not
>>>> a good idea because users will not have control when the
> pre-allocation will happen.
>>>> See item 1.
>>>>
>>>>
>>> I don't think the user should control something in the process as it's a
>>> automatic preallocation.
>>>
>>>> 6. Why do you need to keep the old behavior where new files are created
> after searching
>>>> in the filesystem? I know that this done just for safety in order to
> avoid overwriting
>>>> a possible useful binary log. However, this case will only happen if
> someone manually
>>>> copies files around.
>>>>
>>>>
>>> The old behavior is good. It can guarantee that Static and automatic
>>> preallocation are compatible.
>>>
>>>> Besides, we have committed several patches towards making the index
> file the source
>>>> of truth. So why don't we keep just the new behavior where the index
> file is the source
>>>> of truth?
>>>>
>>>> If overwriting a file is really a concern. I think we should use two
> states in the
>>>> binary log:
>>>> . LOG_EVENT_BINLOG_IN_USE_F
>>>> . LOG_EVENT_BINLOG_PRE_ALLOC
>>>>
>>>> So, if the new file is pre-allocated, its status would be
> LOG_EVENT_BINLOG_PRE_ALLOC
>>>> and the server would be able to know if it is safe to overwrite it
> or not.
>>>>
>>>>
>> I agree with Alfranio. binlog_preallocate is not necessary.
>> There are other ways that can keep the both behaviors.
>> 1. Just like Alfranio said.
>> 2. Use a new BINLOG_MAGIC
>> 3. Add a prefix in the file name.
>> /mysqlbinlogalloc --start=1 --end=10 --alloc-size=2G
> --binlog-name=mysql-binlog
>>
>> pre-mysql-binlog.000001 .. pre-mysql-binlog.000010 are generated.
>>
>> When server need a new binlog file, it will try to find a
>> pre-mysql-binlog.* and rename it to a regular binlog name.
>>
>> I prefer 3.
>>
> I buy the idea, although there are some important details missing here, and
> it will fix the possible problem in item (3). However, a different client still
> sounds strange to me.
>
See reply to Libing's comments in last mail. Thanks!
Best Regards,
Daogang
> Cheers.
>
>
>
>
[snip]
