SSH package for U++ -------------------- SSH package is a feautre-rich, flexible yet very easy to use libssh2 wrapper for Ultimate++. It supports both console and GUI-based applications on POSIX-compliant operating systems and MS Windows (tm). Currently it is in beta (version 1) stage. Classes: -------------------- - Base (core) class -> Ssh - Ssh session -----> SshSession - Sftp subsystem -----> SFtp - Ssh channel -----> SshChannel - Scp channel -----> Scp - Exec channel -----> SshExec - Real-time interactive shell -----> SshShell - X11 forwarding -----> SshShell (as operation mode) - Tcp/IP and port forwarding -----> SshTunnel - Known hosts manager -> SshHosts Features and Highlights: -------------------- - Ssh-derived classes have pick semantics, based on RAII principle, support RTTI, and allow polymorphism (i.e. different classes can be stored in the same arrays, etc.) through a common interface. - Uses U++ memory allocators (Native allocators is also a compile-time option) - Uses OpenSSL by default. - Supports time-constrained, blocking and non-blocking operation modes. - Supoorts multithreaded file transfers and remote command execution (exec), using worker threads. - Supports 3rd-party network proxies. (Such as NetProxy) - Supports known hosts verification mechanism. - Supports password, public key, host-based, and keyboard-interactive authentication methods. - Supports ssh-agents. - Supports real-time interactive command line (shell) interface with both console and GUI integration (works on Windows and Posix-compliant OS'es) - Supports multiple X11 connection forwarding. - Supports Tcp/IP and port forwarding. - Supports detailed (full) debug logging. Todo: -------------------- - Add more high level methods. - Refactor Ssh (core) class. - Improve documentation. Reference examples: ------------------- - SFtpGet: Demonstrates basic SFtp file download in blocking mode. - SFtpGetNB: Demonstrates basic SFtp file download in non-blocking mode. - SFtpGetMT: Demonstrates basic SFtp file download, using a worker thread. - SFtpGUI: Demonstrates a basic SFtp browser with GUI (with upload, download, mkdir, rename, delete commands). - SFtpMultiGetMT: Demonstrates SFtp dir listing and concurrent file transfers, using worker threads. - SFtpConsumerGet: Demonstrates the usage of a consumer function for SFtp download in blocking mode. - SFtpConsumerGetMT: Demonstrates the usage of a consumer function for SFtp download, using a worker thread. - SshExec: Demonstrates basic Ssh remote command execution in blocking mode. - SshExecNB: Demonstrates basic Ssh remote command execution in non-blocking mode. - SshExecMT: Demonstrates basic Ssh remote command execution, using a worker thread. - SshKeyboardAuth: Demonstrates basic Ssh interactive (challenge-response) authentication method in blocking mode. - SshLoggingExample: Demonstrates the logging capabilities of SSH package. - SshOverTor: Demonstrates a basic Ssh connection over TOR, using a third-party network proxy adapter. - SshPolymorphismNB: Demonstrates the polymorphism capability of SSH channels in non-blocking mode. - SshShell: Demonstrates a basic, real-time SSH shell console in blocking mode. - SshShellNB: Demonstrates a basic, real-time SSH shell console in non-blocking mode. - SshShellX11: Demonstrates a basic, real-time SSH shell console with multiple X11 forwarding. - SshShellGUI: Demonstrates the basic GUI integration of SshShell class with multiple X11 forwarding, in non-blocking mode. - SshTunnelExample: Demonstrates the basic SSH tunneling (as tunnel/server) in blocking mode.
#include <Core/Core.h> #include <SSH/SSH.h> using namespace Upp; CONSOLE_APP_MAIN { // This example demonstrates (in non-blocking mode): // 1) Reading the size of a file. // 2) Reading the content of a file (into a String). // 3) Reading the content of a directory as XML. int CMD_GET = 0, CMD_LIST = 0, CMD_SIZE = 0; const char *file = "/readme.txt"; SFtp::DirList ls; Ssh::Trace(); SshSession session; if(session.Timeout(30000).Connect("test.rebex.net", 22, "demo", "password")) { Array<SFtp> sftps; for(int i = 0; i < 3; i++) { auto& sftp = sftps.Add(new SFtp(session)); sftp.NonBlocking(); switch(i) { case 0: sftp.Get(file); CMD_GET = sftp.GetId(); break; case 1: sftp.GetSize(file); CMD_SIZE = sftp.GetId(); break; case 2: sftp.ListDir("/pub/example/", ls); CMD_LIST = sftp.GetId(); break; default: NEVER(); } } while(!sftps.IsEmpty()) { for(int i = 0 ; i < sftps.GetCount(); i++) { SocketWaitEvent we; auto& sftp = sftps[i]; sftp.AddTo(we); we.Wait(10); if(!sftp.Do()) { if(sftp.IsError()) Cerr() << sftp.GetErrorDesc() << '\n'; else { if(sftp.GetId() == CMD_GET) { Cout() << sftp.GetResult(); } else if(sftp.GetId() == CMD_SIZE) { Cout() << Format("Size of %s is %d bytes\n", file, sftp.GetResult()); } else if(sftp.GetId() == CMD_LIST) { for(auto& e : ls) Cout() << e.ToXml() << '\n'; } } sftps.Remove(i); break; } } } } else Cerr() << session.GetErrorDesc(); }
- SFtpGet - SFtpGetNB - SFtpGetMT - SshExec - SshExecNB - SshExecMT
- GetCurrentDir() - GetParentDir()
SshKeyboardAuth: Demonstrates SSH keyboard interactive (challange/response) authentication method. SFtpMultiGet: Demonstrates download of multiple files simultaneously in SFTP non-blocking mode (non MT).
bool SshSession::Connect(const String& url) -------------------------------------------- Connects to a SSH2 server specified by the url. Returns true on success. Syntax of the URL is as follows: [ssh|scp|sftp|exec]://[user:password@]host[:port]
The FileZilla engine 'uses' more protocols (storj) :
Ok, a simple example is alway usefull to see that it Works. I think that a more complex example als FileZilla is not difficult to migrate to Ultimate++.
I think that SendCommand has other meaning then the SendCommand of the FileZilla engine. FileZilla sends commands to the loaded executable Fzftp.exe. Fzftp.exe is derived from Putty project.
Ok. You implement what other (platforms) already have.
I don't know how to go to TOR implementation. Implement Libssh2 routines to the TOR implementation. I have not studie. Maybe put packets on TOR-socket?
#include <Core/Core.h> #include <SSH/SSH.h> #include <NetProxy/NetProxy.h> using namespace Upp; CONSOLE_APP_MAIN { // This example requires a running TOR daemon. // Change below strings to your preferred values. const char* ssh_host = "dummysshhostname"; const char* ssh_user = "dummysshusername"; const char* ssh_pass = "dummysshpassword"; int ssh_port = 22; StdLogSetup(LOG_FILE|LOG_COUT); Ssh::Trace(); NetProxy::Trace(); SshSession session; session.WhenProxy = [=, &session] { return NetProxy(session.GetSocket(), "127.0.0.1", 9050) .Timeout(30000) .Socks5() .Auth("none", "none") .Connect(ssh_host, ssh_port); }; if(session.Timeout(60000).Connect(ssh_host, ssh_port, ssh_user, ssh_pass)) { LOG("Successfully connected to " << ssh_host << " (over TOR)"); } else LOG("Ssh connection via TOR failed. " << session.GetErrorDesc()); }
There are a lot of info en Queue programs. I am looking First at librdkafka, maybe there are better ones.
- SshSession: It is now possible to use encryption keys loaded into memory. (Load keys from mem.) - SshSession: Host based authentication method is added. - Ssh: TraceVerbose() method is added. This method allows full-level logging (redirection) of libsssh2 diagnostic messages. - Documentation updated.
- SshOverTor: Demonstrates a basic SSH connection over TOR (Requires NetProxy package and a TOR daemon) - SshLoggingExample: Demostrates logging capabilities of SSH pakcage.
#include <Core/Core.h> #include <SSH/SSH.h> using namespace Upp; // To activate verbose logging, set the LIBSSH2TRACE flag. // (e.g. via TheIDE->main configuration settings) CONSOLE_APP_MAIN { StdLogSetup(LOG_COUT | LOG_FILE); // Ssh::Trace(); Ssh::TraceVerbose( // LIBSSH2_TRACE_SOCKET | LIBSSH2_TRACE_KEX | LIBSSH2_TRACE_AUTH | LIBSSH2_TRACE_CONN | // LIBSSH2_TRACE_SCP | // LIBSSH2_TRACE_SFTP | // LIBSSH2_TRACE_PUBLICKEY | LIBSSH2_TRACE_ERROR ); SshSession session; auto b = session.Timeout(30000).Connect("demo:password@test.rebex.net:22"); LOG((b ? "Successfully connected to SSH2 server." : session.GetErrorDesc() << '\n')); }
2018-01-19: Alpha version 2. SshChannel reworked. It is now more flexible, and more analogous to a tcp socket. Scp class gained a new Put method (and its corresponding operator overload). NEW: SShShell is added to the package. It allows GUI integration and has a "console mode" that supports both POSIX consoles and Windows command prompt. NEW: SshTunnel class is added to the package. It allows TCP/IP and port forwarding over SSH protocol. Various bug fixes, and improvements.
- All SSH components have a very simple, easy to use, and uniform interface that supports time-constrained blocking, non-blocking operation modes and multithreading, - They all work on Windows (tested on 7 & 10) and POSIX-compliant operating systems, and compile on both GCC/MingGW, and MSC. - Shell component can work simultaneously with multiple X11 forwarding (per-shell), which is AFAIK a very rare feature among the libssh2 wrappers out there. - And all this can be achieved writing very little code! (e.g. SshX11Shell has 10 LOCs for the actual code of X11-enabled full console, and SshShellGUI has 156 LOCs which are mostly usual U++ GUI setup)
- Blocking and non-blocking behaviour is now very similar to TcpSocket's. - IsBlocking(), IsWorking() methods are added. - WhenDo replaced with WhenWait, and WaitStep() method is added
Remove progress gate parameters in the getters and putters, and replace them with a single WhenProgress gate. - Add WhenContent: Consumer function for incoming data transfers. Change the Async (multithreaded) getters and putters: - They will have progress gates with three-parameteres (id, done, total), instead of two. This proved more useful. - They will use a URL similar to the one I use in the new version of FTP package. - Additional variants for async functions, which will use One<Stream> and picks input data (for outgoing transfers). - Update libssh2, as it has seen some activitiy, and gained new ciphers.
2018-04-15: Consumer function support added to SFtp and SshChannel classes. GetWaitStep() method is added to Ssh class. Multithreaded methods rewritten.
#include <Core/Core.h> #include <SSH/SSH.h> using namespace Upp; CONSOLE_APP_MAIN { StdLogSetup(LOG_COUT|LOG_FILE); // Ssh::Trace(); const char *file = "/pub/example/readme.txt"; String data; SshSession session; if(session.Timeout(30000).Connect("demo:password@test.rebex.net:22")) { auto sftp = session.CreateSFtp(); sftp.WhenContent = [&data](const void *buf, int len) { data.Cat(static_cast<const char*>(buf), len); }; sftp.Get(file); LOG((!sftp.IsError() ? data : sftp.GetErrorDesc())); } else LOG(session.GetErrorDesc()); }
static AsyncWork<void> SFtp::AsyncConsumerGet(SshSession& session, const String& path, Event<int64, const void*, int> consumer) static AsyncWork<void> Scp::AsyncConsumerGet(SshSession& session, const String& path, Event<int64, const void*, int> consumer)
- SFtpConsumerGetMT: Demonstrates the usage of a consumer function for SFtp download, using a worker thread. - SshPolymorphismNB: Demonstrates the polymorphism capability of SSH channels in non-blocking mode.
2018-05-04: Ssh::GetWaitEvents() fixed. SshTunnel::Validate() fixed.
SshTunnelExample: Demonstrates the basic SSH tunneling (as tunnel/server) in blocking mode.
#include <Core/Core.h> #include <SSH/SSH.h> using namespace Upp; // This example requires upp/reference/SocketServer and upp/reference/SocketClient examples. // SocketClient: Set the port number to 3215. // // |SocketClient (client)|<---> |SshTunnelExample (tunnel/server)| <---> |SocketClient (server)| bool SocketSendRecv(String& packet) { TcpSocket s; if(!s.Connect("127.0.0.1", 3214)) { LOG("SocketSend(): " << s.GetErrorDesc()); return false; } if(!s.PutAll(packet + '\n')) return false; packet = s.GetLine(); return !packet.IsEmpty(); } void StartTunnel(SshSession& session) { SshTunnel listener(session); if(!listener.Listen(3215, 5)) { LOG("StartTunnel(): " << listener.GetErrorDesc()); return; } LOG("SSH tunnel (server mode): Waiting for the requests to be tunneled..."); for(;;) { SshTunnel tunnel(session); if(!tunnel.Accept(listener)) { LOG("StartTunnel(): " << tunnel.GetErrorDesc()); return; } auto data = tunnel.GetLine(); LOG("Tunneled Request: " << data); if(!data.IsEmpty() && SocketSendRecv(data)) { LOG("Tunneled Response: " << data); tunnel.Put(data + '\n'); } } } CONSOLE_APP_MAIN { StdLogSetup(LOG_FILE | LOG_COUT); // Ssh::Trace(); SshSession session; if(session.Timeout(30000).Connect("username:password@localhost:22")) { StartTunnel(session.Timeout(Null)); return; } LOG(session.GetErrorDesc()); }
2018-06-15: Critical fix: SshChannel and SFtp data read and write methods fixed: The recently introduced socket wait mechanism was causing a constant I/O blocking. This is now fixed. Cosmetics & cleanup. 2018-06-06: SshShell: Console loop fixed.
SSH looks pretty reasonable. Can I move to Core/SSH?
Put(Stream& in, const String& path);
I would expect path to be the first argument here... Do you insist on this order?
1) Implementation document (giving an overview of the package and its implementation details). 2) MT server (listener) mode for SshTunnel. 3) Refactoring of Ssh (core) class.
Hello Mirek,
Quote:
SSH looks pretty reasonable. Can I move to Core/SSH?
Of course!
Quote:Put(Stream& in, const String& path);
I would expect path to be the first argument here... Do you insist on this order?
That's because I use "[source], [destination]..." order. I'd prefer it stay that way but I don't insist on it. It can be changed.
Instead of:
auto sftp = session.CreateSFtp();
I would like to have
SFtp sftp(session);
More important: I thing I would remove whole bunch of Gets and Puts.
Interestingly, it looks like the most important Gets / Puts are missing there
int Get(SFtpHandle h, void *ptr, int size);
bool Put(SFtpHandle h, const void *ptr, int size);
... then add SFtpStream instead. With methods provided in SFtp, it should be easy to do, and it would mostly remove "source/destination controversy".
Hello Mirek,
Ok then, I was going to update the package, but I'll delay the next update, and first come up with the changes you've asked.
Note that it is now part of U++, so please it is preferable that any changes you do are done in U++ svn. You should have write rights there.
SFtpHandle* -> SFtpHandle - if something is "HANDLE", it should not be a pointer to handle.
Interestingly, it looks like the most important Gets / Puts are missing there
int Get(SFtpHandle h, void *ptr, int size);
bool Put(SFtpHandle h, const void *ptr, int size);
int Get(SFtpObject* obj, void* buffer, int size); String Get(const String& path, int size, int64 offset = 0); String GetAll(const String& path); int Put(SFtpObject* obj, const void* buffer, int size); int Put(const String& path, const String& data, int size, dword flags, long mode); int Put(const String& path, const String& data, int size, int64 offset = 0); bool PutAll(const String& path, const String& data);
bool GetStream(const String& path, Stream& out); bool PutStream(const String& path, Stream& in); // And other variants...
dword Ssh::GetWaitEvents() { ssh->events = 0; if(ssh->socket && ssh->session) ssh->events = libssh2_session_block_directions(ssh->session); return !!(ssh->events & LIBSSH2_SESSION_BLOCK_INBOUND) * WAIT_READ + !!(ssh->events & LIBSSH2_SESSION_BLOCK_OUTBOUND) * WAIT_WRITE; }
I undestand the appeal, but I have to say that the result is sort of confusing. E.g. I have implemented int Get(SFtpObject* obj, void* buffer, int size); but it is clearly incompatible with this sort of async operations.
So I guess for now, I will try to pretend that those complex Cmd / ComplexCmd "nonblocking" operations are not there, p
All those NULL handles, implicit results etc make me uneasy.
In related news, I have also fixed GetWaitEvents
I am also thinking that perhaps SFtp should be (derived from) SshSession. I think it is unlikely that sharing SshSession for several protocols is all that important.
Those methods are the parts that I am not happy with, either
I have a plan and a test code to "fix" that ugliness, but actual refactoring will come later.
Did it make any difference? Because IIRC values of those upp enums and the defines of libssh2 are the same.
Is there any real-world example where you would need this kind of nonblocking behaviour? All I can came up is some code that communicates with thousands of ssh servers at once. Looks very unlikely to me....
int SFtp::Get(SFtpHandle handle, void *ptr_, int size0) { int done = 0; // <- Can't be used in non-blocking mode. char *ptr = (char *)ptr_; Cmd(SFTP_START, [=, &done]() mutable { int size = size0; if(OpCode() == SFTP_START) { if(FStat(HANDLE(handle), *sftp->finfo, false) <= 0) OpCode() = SFTP_GET; // <- This is for higher-level api. should be removed. else return false; } while(size) { int rc = libssh2_sftp_read(HANDLE(handle), ptr, min(size, ssh->chunk_size)); if(rc < 0) { if(!WouldBlock(rc)) SetError(rc); return false; } else { if(rc == 0) break; size -= rc; done += rc; if(WhenProgress(done, size0)) SetError(-1, "Read aborted."); ssh->start_time = msecs(); } } LLOG(Format("%d of %d bytes successfully read.", done, size0)); return true; }); return done; }
Are they documented to be the same?
I use it in an app and for a limited number of ssh channels (usually 10-20).
But frankly, that code remains before the the CoWork improvements and the arrival of AsyncWork.
Nowadays in most such cases I use the async methods.
I have a new proposal: What if I get rid of queue mechanism and rewrite the package with only blocking mode and optional async transfer methods(using AsyncWork, and naming them agein SFtp::Asyncxxx)?
It won't take more than a week for me to come up with a working SSH package and the existing public API wont change much (only the NB helpers will be gone).
Besides its SC will be lot cleaner.
As to your Get implementation:
I haven't tested this yet, but it shouldn't work in non-blocking mode. (because the execution will be deferred (Get will immediately return) and there is a local variable ("done") )
bool SshChannel::Lock() { if(*lock == 0) { LLOG("Channel serialization lock acquired."); *lock = ssh->oid; } return *lock == ssh->oid; }
Sure, as I said that was the point where I decided that fully non-blocking mode is "blocking" this kind of interface.
int SFtp::Read(SFtpHandle handle, Event<const void*, int>&& consumer, int size) // Data read engine. { int sz = min(size, ssh->chunksize) Buffer<char> buffer(sz); int rc = libssh2_sftp_read(HANDLE(handle), buffer, sz); if(!WouldBlock(rc) && rc < 0) SetError(rc); if(rc > 0) { consumer(buffer, rc); sftp->done += rc; if(WhenProgress(sftp->done, size)) SetError(-1, "Read aborted."); ssh->start_time = msecs(); } return rc; } int SFtp::Get(SFtpHandle* handle, void* buffer, int size) { Clear(); Cmd(SFTP_GET, [=]() mutable { int rc = Read( handle, [=](const void* p, int sz) { if(!buffer) SetError(-1, "Invalid pointer to read buffer"); memcpy((char*)(buffer + sftp->done), (char*)p, sz); }, size ); if(rc >= 0) sftp->value = sftp->done; return rc == 0 || sftp->done == size; }); return sftp->done; }
I think there is a race condition here - two threads can obtain this lock simultaneously. Now I am not sure whether is this supposed to be MT safe, but if not, why atomic, right?
I am also pretty ambivalent about all those static AsyncWork methods. I think these are better left to client code. Especially if we abandon the non-blocking mode.
Hello Mirek,
Quote:
Sure, as I said that was the point where I decided that fully non-blocking mode is "blocking" this kind of interface.
I'm sorry but I really don't understand this one.
If you find it reasonable, I'll rewrite the SSH package with only blocking mode in mind (but with neceasary thread safety, and add its components gradually).
But as I already wrote in my previous messages, let us not change the "single session-multiple channels model" and not exclude any components.
Is this OK?
"Pseudoblocking" - we still would like to have WhenWait and Abort (afaik it is now named "Cancel", but TcpSocket is using Abort, so maybe it should have the same name). WhenWait and Abort will allow GUI around it.... (That said, I think each channel should have its own WhenWait).
Well, if you are about to have such Get in the interface, you expect it to be usable with multiple streams at the same time. Storing into single ftp->done is no go then. Really, let us wait for co_await and do it right then
Absolutely, except I might want to work on it a bit too Smile So please at least check what happens in svn.
WhenWait and Abort will allow GUI around it.... (That said, I think each channel should have its own WhenWait).
In reality, I do not think that there is a lot of new things to develop. This is mostly simplifying and removing.... Probaly Cmd will get simplified, ComplexCmd probably can be removed.
I have today developed SFtpStream
Yep. You see, an ssh shell is a complex environment. You cannot initalize multiple shells, and/or exec channels at once (I don't want to go into details here). This is a limitation of the libssh2.
- Ssh, SshSession, and SFtp classes are refactored. - Non-blocking mode and multithreaded methods are removed. - The package now uses a "pseudo-blocking" technique, similar to TcpSocket's. - Accordingly, the Ssh::Cmd and Ssh::ComplexCmd methods are removed in favor of Ssh::Run and Ssh::Do() commands. Run() command is the new command execution encgine. It'll hopefully take care of the possible thread safety problems. Do() method is protected by a single static mutex, and should be thread-safe. Multithreaded execution should be possible. - Abort mechanism refactored. - Sftp::Read and SFtp::Write methods are further simplified. - SFtp::WhenProgress method is removed in favour of GetDone() method (as the esisting WhenProgress method is now pretty useless). - SshChannel and its derivatives are currently disabled. They will be re-added gradually.
I would like to learn a bit about this. Can you give me some link(s)?
Hello Mirek,
Before I commit the latest update to the code, I'd like to know if adding a "SFtpFileSystemInfo()" (non static) would be ok? It'll be an example of remote file system representation. I think FileSel, or any other file browser can benefit this. Also I can write topic docs for FileSystemInfo() as I find it a useful tool (unless it is depreceated, of course).
Best regards,
Oblivion
#include <CtrlLib/CtrlLib.h> #include <Core/SSH/SSH.h> using namespace Upp; GUI_APP_MAIN { StdLogSetup(LOG_FILE); // Ssh::Trace(); SshSession session; if(session.Timeout(30000).Connect("demo:password@test.rebex.net:22")) { SFtp sftp(session); SFtpFileSystemInfo fsi(sftp); FileSel fs; fs.Filesystem(fsi); if(fs.BaseDir("/").ExecuteOpen()) { String file = fs.Get(); Progress pi(nullptr, file); sftp.WhenProgress = [=, &pi] (int64 done, int64 total){ pi.SetText(Format(t_("%1:s of %2:s is transferred"), FormatFileSize(done), FormatFileSize(total))); return pi.SetCanceled(int(done), int(total)); }; pi.Title(t_("Downloading ") << GetFileName(file)); pi.Create(); String s = sftp.LoadFile(file); if(sftp.IsError()) ErrorOK(DeQtf(sftp.GetErrorDesc())); } } else ErrorOK(DeQtf(session.GetErrorDesc())); }
SFtpFileSystemInfo helper class is a FileSystemInfo adapter for the SFtp objects. It allows SFtp objects to be used with FileSystemInfo class that provides uniform access to folder hierarchies in a file-system agnostic ("transparent") way.