List:Backup« Previous MessageNext Message »
From:Rafal Somla Date:November 6 2009 10:28am
Subject:Re: HLS approved: WL#5046 - Online Backup: Pluggable Storage Modules
View as plain text  
Hi Andreas,

OK, so let's consider that HLS is finished now. Next action then would be 
for you to write LLD and get it approved by Ingo. Since you don't have 
access to our WL database, I can be a proxy - just send me your propositions 
and I'll put them into WL.

I think LLD should contain description of what you want to implement on the 
level of C API. It should contain enough information for Ingo so that he can 
get an idea of how the implementation would look like and approve it. This 
does not need to be very detailed, because implementation itself is also 
reviewed.

When designing the API, it would be good to think not only about functions 
in the API but also about what data types are needed and will be used for 
passing the information around.

Please let me know if you think anything should be changed in HLS. We can 
still do it if we learn more when working on lower-level details.

Andreas Almroth wrote:
> I concur with Ingo. Let us get started, we are already in November, leaving 
> me very little time for implementation...
> 
> Comments:
> 
> A1 can be seamlessly be implemented by backup kernel in the future, if the 
> initial current bytes are not enough. The backup system beneath BSM will 
> most likely have to store out-of-band meta-info anyway, with perhaps the 
> exception of the default file BSM (perhaps solvable with use of table in 
> system database/information schema etc, haven't analyzed all the 
> possibilities yet).

I imagine that default file BSM will store meta-info the same as it is doing 
now - in a 10 byte prefix of the stream. The specification is general enough 
to allow such implementation and at the same time allows for out-of-band 
implementations for BSMs for which it is more natural. The main 
architectural decision is whether storing meta-info is a responsibility of 
BSM or backup kernel does it. The current design is that it is a 
responsibility of BSM (and the interface is adapted to that). The 
alternative A1 is that backup kernel stores meta-info, and it is rejected now.

> 
> A3 is, I feel, is like being stuck between a rock and a hard place... Code 
> must be added for async I/O. The buffer sent from storage engine must be 
> processed and used in chunks if it can't be written in whole. On the other 
> hand; if the BSM report a preferred buffer size smaller than storage engine 
> deliver, we still have to implement the code. Sooooo, A3 will be implemented 
> regardless, but not because of an async I/O need.
> 
> Also note; Most backup systems are slow, meaning that we can expect anything 
> from milliseconds to days, before the data can actually be written. This has 
> to do with the resource allocation within the backup system. I intend to 
> implement a timeout watchdog thread, which will abort waiting I/O 
> operations, and thus abort the backup. Usual default value is 300 seconds. A 
> timeout variable should be implemented for the BSM as well. This approach is 
> de facto standard and quite common in the backup world.

I'm not sure which option you are arguing for. Anyway, the current design is 
that "write bytes" service can write only part of the bytes (also 0 bytes is 
possible) and the user of the service can/should retry writing remaining data.

Rafal

Thread
HLS approved: WL#5046 - Online Backup: Pluggable Storage ModulesIngo Strüwing4 Nov
  • Re: HLS approved: WL#5046 - Online Backup: Pluggable Storage ModulesRafal Somla4 Nov
    • Re: HLS approved: WL#5046 - Online Backup: Pluggable Storage ModulesRafal Somla4 Nov
    • Re: HLS approved: WL#5046 - Online Backup: Pluggable Storage ModulesIngo Strüwing4 Nov
      • RE: HLS approved: WL#5046 - Online Backup: Pluggable Storage ModulesAndreas Almroth4 Nov
  • Re: HLS approved: WL#5046 - Online Backup: Pluggable Storage ModulesRafal Somla6 Nov