Hi Andreas,
OK, so let's consider that HLS is finished now. Next action then would be
for you to write LLD and get it approved by Ingo. Since you don't have
access to our WL database, I can be a proxy - just send me your propositions
and I'll put them into WL.
I think LLD should contain description of what you want to implement on the
level of C API. It should contain enough information for Ingo so that he can
get an idea of how the implementation would look like and approve it. This
does not need to be very detailed, because implementation itself is also
reviewed.
When designing the API, it would be good to think not only about functions
in the API but also about what data types are needed and will be used for
passing the information around.
Please let me know if you think anything should be changed in HLS. We can
still do it if we learn more when working on lower-level details.
Andreas Almroth wrote:
> I concur with Ingo. Let us get started, we are already in November, leaving
> me very little time for implementation...
>
> Comments:
>
> A1 can be seamlessly be implemented by backup kernel in the future, if the
> initial current bytes are not enough. The backup system beneath BSM will
> most likely have to store out-of-band meta-info anyway, with perhaps the
> exception of the default file BSM (perhaps solvable with use of table in
> system database/information schema etc, haven't analyzed all the
> possibilities yet).
I imagine that default file BSM will store meta-info the same as it is doing
now - in a 10 byte prefix of the stream. The specification is general enough
to allow such implementation and at the same time allows for out-of-band
implementations for BSMs for which it is more natural. The main
architectural decision is whether storing meta-info is a responsibility of
BSM or backup kernel does it. The current design is that it is a
responsibility of BSM (and the interface is adapted to that). The
alternative A1 is that backup kernel stores meta-info, and it is rejected now.
>
> A3 is, I feel, is like being stuck between a rock and a hard place... Code
> must be added for async I/O. The buffer sent from storage engine must be
> processed and used in chunks if it can't be written in whole. On the other
> hand; if the BSM report a preferred buffer size smaller than storage engine
> deliver, we still have to implement the code. Sooooo, A3 will be implemented
> regardless, but not because of an async I/O need.
>
> Also note; Most backup systems are slow, meaning that we can expect anything
> from milliseconds to days, before the data can actually be written. This has
> to do with the resource allocation within the backup system. I intend to
> implement a timeout watchdog thread, which will abort waiting I/O
> operations, and thus abort the backup. Usual default value is 300 seconds. A
> timeout variable should be implemented for the BSM as well. This approach is
> de facto standard and quite common in the backup world.
I'm not sure which option you are arguing for. Anyway, the current design is
that "write bytes" service can write only part of the bytes (also 0 bytes is
possible) and the user of the service can/should retry writing remaining data.
Rafal
