download problem (Bug ?)

I implemented an SSH server on an embedded device. SFTP works fine for a lot of clients, uploading works with WinSCP, but my SW gets in to trouble when I try to upload a file to WinSCP that is larger than the servers maximum size of the data transport packet.

I receive following notification from WinSCP:
"Received incomplete data packet before end of file."

After this, downloading is aborted.

I believe the RFC states that the data packets do not need to be filled completely.
You can only be sure that a file is complete when the server sends an EOF status reply.

So, is this a bug in WinSCP ?
Should I 'try' and fix this in my server ?
(Embedded things are very restricted on memory threads,... so this could be a though one)

Regards Erik.

I believe the RFC states that the data packets do not need to be filled completely.
You can only be sure that a file is complete when the server sends an EOF status reply.

Quote from draft-ietf-secsh-filexfer-04.txt:

For normal disk files, it is normally guaranteed that this will read the specified number of bytes, or up to end of file. However, if the read length is very long, the server may truncate it if it doesn't support packets of that length. See General Packet Format (Section 3).

So yes, you are allowed not to send what I request. Until version 3.4 WinSCP has not required it. I have includes this limitation whan I have implemented read/write queuing to improve performance. I send up to for read/write requests, without waiting for the response. So I count on fact that the server read exactly for I have asked. I believe that the same think does even other SFTP clients, but I may be wrong. Try the latest version of PSFTP (development snapshot) for example, or Filezilla. Maybe I'm wrong and their implementation is better than mine. I can imagine that it is possible to do queuing even without the limitation. But it would make things very complicated.

I can imagine that it is possible to do queuing even without the limitation.

It is possible, but it would indeed complicate your code if you would have to examine the replies and then need to launch new reads to 'fill the gaps'.
I can see your point. I will try to improve my server (if my time schedule allows for this...)

On the other hand, one should try and stick to the RFC...

Thanks for the quick response.

It is possible, but it would indeed complicate your code if you would have to examine the replies and then need to launch new reads to 'fill the gaps'.

And consequently, I lose the performace benefit.

On the other hand, you are right with "sticking to RFC" :-)

martin wrote:

And consequently, I lose the performace benefit.

I've got my server working with WinSCP now, but... I noticed that you are using FTP window sizes of 4096 bytes. If you want to crank performance up even more, why not use the full 32KB that is forseen in the RFC ?
I'm not recommending this however, because this would break my fix :( I just increased my buffer size slightly so no real code fix, but because I dont have a lot of memory, this will have to do.
(The real fix will require even more memory)

Something else that could be interesting... If you take the 'maximum packet size' that is provided when opening a channel as your maximum packet size for SFTP (minus the header overhead), this could fix my problem for good. I guess large servers will have large buffers defined and smaller (like mine) wil have smaller buffers ??? So performance will be 'lower' for 'small' servers, but who cares, these are an exception...

A possible fix ?

Regards Erik

Thanks for your ideas. I appreciate it :-)