List:Backup« Previous MessageNext Message »
From:Rafal Somla Date:November 3 2009 1:52pm
Subject:Re: WL#4056: Pluggable storage modules for backup
View as plain text  
Hi Ingo,

I've come up with an example where my "asynchronous" I/O interface could 
provide extra value as compared to your "simplified one".

Consider a streaming storage module which connects over internet to some 
destination host and sends data there. Now suppose that backup kernel asks 
for writing 10Mb of data. The module will write 9Mb and then the connection 
will drop and no more data can be written at the moment. With my interface, 
storage module can report to the kernel that 9Mb has been written. Having 
this information, kernel can try to avoid writing it again when situation is 
resolved. With your interface, kernel will not learn about the fact that 
most of the data was actually successfully written - it will only know that 
the request has failed and must be repeated.

I am not saying the current backup kernel can/will take advantage of it. But 
this is a difference between having a bit of information or not having it 
and such difference will determine how useful the interface is.


PS In alternative A3 I speak only about "write bytes" service. Do you think 
the same simplification should be applied to "read bytes" one? That is, the 
service will either read as much bytes as requested or fail. But then how 
should it react if end of stream was encountered before all data could be read?

Ingo Strüwing wrote:
> Hi Rafal,
> Rafal Somla, 23.10.2009 10:35:
> ...
>> Ingo Strüwing wrote:
> ...
>>> Rafal Somla, 22.10.2009 13:04:
> ...
>>>> Ingo Strüwing wrote:
> ...
>>>>> "Amount of data that has been written"
> ...
>>> So you suggest asynchronous operation again as a use case.
>> Yes, because as I wrote I think it is quite possible that storage module
>> will drive some unusual physical storage device. Is this not a valid
>> argument?
> It is quite possible that a storage module will drive some unusual
> physical storage device. That's a fact. But how is it a valid argument?
> An argument for what? That we need asynchronous operation? In that case
> I don't agree.
> I may lack imagination. What could the possible options be for the
> backup kernel to act on an "amount of data that has been written" less
> than the requested amount?
> 1. Loop. Repeat the write with the remaining data. If the backup storage
> module is allowed to return with less data than requested, and it
> decided to do so for some reason before, it will probably return
> immediately again with zero bytes written. We'd run in an infinite loop
> until the "unusual physical storage device" returned to service. We
> could add a sleep to avoid 100% CPU usage. But how long to sleep? Too
> short and we consume too much resources on busy waiting. Too long and we
>  lose time as the storage could be come available before we end sleeping.

We can limit the number of re-tries or how long we try to write the data. We 
can adjust the sleep amount depending on how the situation evolves.

> 2. Use the time and do something else in between? Our buffer is full. We
> need to get rid of it before doing more data collection. Allocate more
> buffers and fill them? If we really want to implement double buffering,
> we would want to do that as a general solution with two or more threads.
> What else could a backup kernel do with the available time?

Maybe we wiu
> 3. (Add your ideas here)
> My current standpoint is that there is no reasonable way to profit from
> an "amount of data that has been written" information returned by the
> storage module. Hence I request from the storage module that it does its
> best to write the provided data unless it has to give up. If I want
> asynchronism, I'd use threads. Or implement a full asynchronous
> interface, which allows the kernel to wait until the storage signals
> "ready for more data".
> ...
>> Well, I'm still not convinced that my specification, as it is now, is
>> ridiculous :)
> I hope, you did not seriously read my comments as saying this. (I hope
> the smiley transports this.) I oppose against the accentuation of one
> possible status return in the specification. I mentioned other status'
> that could as well be mentioned explicitly. I said that could could even
> come up with more ridiculous examples.
> The worst you could deduce from it is that I might find the mention of
> "acknowledgement" in the high level specification ridiculous. But please
> don't extend this to the whole specification.
> But even for this piece of the HLS I wont use "ridiculous". What I think
> about it is more like "ugly" or such. But as we are going towards
> dropping the compression-selection service anyway, this isn't relevant
> any more.
>> I don't know what is your impression, but mine is that although this
>> discussion is lengthy, we still clarify many things and understand
>> better each ones position.
> Agree. And it is perhaps good to spend much time in the specification.
> At least better than noticing problems during coding only.
>>   You also give some interesting suggestions I
>> was not thinking about, like possibility to abort operation by storage
>> module or interface for getting progress info from storage modules.
> Yes, but this was to show that we don't try to cover all information
> that could be transported over the interface. I do not suggest to
> include these in the specification now. In contrast, I strive for
> getting rid of the asynchronous option to make the interface even
> simpler. I don't agree that this removes flexibility. I prefer to
> specify asynchronism differently, and only later, if required at all.
> Regards
> Ingo
Re: WL#4056: Pluggable storage modules for backupRafal Somla13 Oct
  • Re: WL#4056: Pluggable storage modules for backupIngo Strüwing15 Oct
    • Re: WL#4056: Pluggable storage modules for backupRafal Somla22 Oct
      • Re: WL#4056: Pluggable storage modules for backupIngo Strüwing22 Oct
        • RE: WL#4056: Pluggable storage modules for backupAndreas Almroth22 Oct
        • Re: WL#4056: Pluggable storage modules for backupRafal Somla23 Oct
          • Re: WL#4056: Pluggable storage modules for backupIngo Strüwing23 Oct
            • Re: WL#4056: Pluggable storage modules for backupRafal Somla3 Nov