[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Multiple concurrent SFTP file downloads


There is a JIRA ticket about concurrent downloads. Use the vote in
JIRA to tell us that you like that.

On Tue, Jun 19, 2018 at 7:30 PM, Alexander Barrow <barrowaw@xxxxxxxxx> wrote:
> I am developing an application that downloads large files from a
> remote SFTP server over a bandwidth limited network. Download speeds
> are bottle-necked, making it highly desirable to be able to download
> multiple files concurrently. From what I've gathered, this is not a
> feature supported out of the box by the standard SFTP component.
> A solution I engineered to work around this was to use a route defined
> something like as follows:
> from( "sftp:user@host?download=false..." )
> .threads( 5 ).maxQueueSize( 0 )
> .pollEnrich().simple(
> "sftp:user@host?localWorkDirectory=.inbound&include=$simple{file:name}&disconnect=true..."
> )
> .to( "file:downloads" );
> This uses the route's consumer to supply a "list" of files to
> download, but delegates the actual downloading to the poll-enrich EIP
> across multiple threads. The issue with this approach is that it uses
> multiple connections and repeatedly connects and disconnects from the
> remote server, which the SFTP server owners dislike.
> Is there another existing component or feature that would support the
> multiple concurrent downloads use case, or is this a gap in Camel's
> feature list?

Claus Ibsen
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2