Hi Ingo,
Thanks for the comments. Some explanations follow.
Ingo Strüwing wrote:
> When enabling debug, I see
>
> ...
> | >str_read
> | | stream: want bytes: 512
> | | stream: read bytes: 512
> | <str_read
> | >str_read
> | | stream: want bytes: 15872
> | | stream: read bytes: 15872
> | <str_read
> | >str_read
> | | stream: want bytes: 512
> | | stream: read bytes: 512
> | <str_read
> | >str_read
> | | stream: want bytes: 15872
> | | stream: read bytes: 15872
> | <str_read
> ...
>
>> In any case, the library does not try to read data in 16K blocks since
>> it assumes that OS is doing this in the background. What the library
>> tries to do, is to keep as few data as possible inside internal buffers
>> to avoid double buffering.
>
> This does not exactly match my analyze of stream_v1_transport.c. There
> seems to be an input buffer, which is filled with load_buffer() when
> insufficient data is in the buffer.
>
Ok, you caught me on a "simplyfing lie" - sorry for missinforming. This is
right, I'm having a small input/output buffer because I need to read fragment
headers before I know how to read fragment contents. I was trying to avoid
double buffering (by reading directly to the provided buffer) but with the
assumption that it will be relevant only when huge data chunks are read.
Your analysis exhibits that perhaps the optimization does not work as good as
expected. I'd have to think about this...
> The first read is 512 bytes, which is sufficient for the header and part
> of the catalog. In my small test case it does even span into the first
> table data chunk.
>
> It looks like you read 512 bytes minimum and then, if requested the rest
> of a block_size. That could explain the above numbers.
>
>> Inside the library it is assumed that it is ok and efficient to call
>> low-level I/O for arbitrary amounts of data. E.g. to read single bytes.
>
> It does not cost a disk access each time, but a system call has pretty
> much overhead. It almost always pays off to buffer in the application.
>
>> Making this efficient is the task of the as_read_m and as_write_m
>> callbacks which can implement internal buffering if needed.
>
> Ok. But then we would end up with triple buffering.
>
Yes, but hopefully only for small amounts of data. I.e., the low-level read
loads a 16K block into its buffer (or gets pointer to system buffer). The
library copies few bytes to the internal buffer to read and interpret the
headers (currently 512 bytes but this can be adjusted). The rest of the data
should be copied directly from the low-level buffer to the buffer provided by
the library user. At least that was the idea.
Btw. a pity you were not reviewing this design...
Rafal
