Home » Extra libraries, Code snippets, applications etc. » U++ users applications in progress and useful code snippets, including reference examples! » Using CoFor with SSH package to parallelize sftp file downloads. (Example code)
Using CoFor with SSH package to parallelize sftp file downloads. [message #58802] |
Sun, 04 September 2022 22:34 |
Oblivion
Messages: 1093 Registered: August 2007
|
Senior Contributor |
|
|
Core/SSH package has some nice multithreading abilities which can be sometimes overlooked..
The code snippet below demonstrates a way to combine the MT capabilities of SSH package with Upp::CoFor (loop parallelization function):
1) It reads the content of a remote directory
2) Downloads multiple files with certain attributes in parallel, using the CoFor loop.
#include <Core/Core.h>
#include <Core/SSH/SSH.h>
using namespace Upp;
constexpr const char *path = "/pub/example/";
constexpr const int MAXDOWNLOAD = 4;
CONSOLE_APP_MAIN
{
StdLogSetup(LOG_COUT|LOG_FILE);
// Ssh::Trace();
SshSession session;
if(!session.Timeout(10000).Compression().Connect("demo:password@test.rebex.net:22")) {
RLOG(session.GetErrorDesc());
return;
}
SFtp browser(session);
SFtp::DirList ls;
// Get a remote dir listing.
if(!browser.ListDir(path, ls)) {
RLOG(browser.GetErrorDesc());
return;
}
// Filter the dir list.
auto files = FilterRange(ls, [](const SFtp::DirEntry& e) { return e.IsFile() && e.GetSize() <= 65536; });
// Download the files, using worker threads in parallel.
CoFor(min(files.GetCount(), MAXDOWNLOAD), [&](int i){
const SFtp::DirEntry& e = files[i];
String fpath = AppendFileName(path, e.GetName());
RLOG("Downloading " << fpath);
SFtp sftp(session);
String file = sftp.LoadFile(fpath);
if(sftp.IsError())
RLOG(Format("Worker #%d: %s", sftp.GetId(), sftp.GetErrorDesc()));
else
RLOG("File " << e.GetName() << " is successfully downloaded.");
});
}
Best regards,
Oblivion
Github page: https://github.com/ismail-yilmaz
upp-components: https://github.com/ismail-yilmaz/upp-components
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
[Updated on: Sun, 04 September 2022 22:49] Report message to a moderator
|
|
|
Goto Forum:
Current Time: Sat May 04 16:46:38 CEST 2024
Total time taken to generate the page: 0.02754 seconds
|