U++ Forum - RDF feed
type: channel
description: Main place for discussions about U++ cross platform development framework.
items: SFTP or full SSH2 support for U++?
items: Re: SFTP or full SSH2 support for U++?
items: Re: SSH2 wrapper for U++
items: SSH package for U++ (alpha version)

SFTP or full SSH2 support for U++?
type: item
description: Hello, Fırst, I wish you all a happy new year! Some time ago when I uploaded my FTPS implementation to the bazaar section, Daniel (Unodgs) asked me If I have any plans to implement an SFTP class. And I'd said yes. Now I am here to deliver my promise. But before I upload the package to the bazaar section, I have to make a choice and would like to hear your opinion on the subject. First of all, I decided not to implement the SSH2 protocol from the scratch. I could've, but it would be a) tedious and b) wasting my time. So I decided searching for alternatives. The best alternative that I could have come up with was wrapping an existing, multiplatform (Linux and Windows, at least) library. I found out that libssh2 was the way to go. It was plain C, easy to wrap up and had both blocking and non-blocking modes of operation available. I finally implemented an SFTP class for U++ using libssh2, which has the below features: 1) Allows blocking and non-blocking operations. Namely, it can work in both synchronous and asynchronous modes. I use HttpRequest and NetworkProxy class' asynchronous design here too, since that design proved very effective. 2) Has OPTİONAL network proxy (HTTP_CONNECT, SOCKS4/4a/5) support, using NetworkProxy class. 3) Takes advantage of U++ streams in file upload and download operations, and uses gates for progress tracking when needed. 4) Supports both MD5 and SHA 1 methods. 5) Has SFtp::DirEntry class for easy parsing of directory entries. In short, currently we have a sync/async SFTP wrapper that can authenticate using "public key" method and/or Username/password combination, and perform Open/Close/Read/Write/List/Rename/Delete operations on remote file system entries. Also its API is very similar to my FTPS class, and its design derives from NetworkProxy and HttpRequest classes and follows the U++ coding style. Now, the class is abstracted nicely so that I can either let it be a standaloune, robust SFtp class, or take it one step further and wrap up t…
creator: Oblivion
date: 2016-01-10T11:58:48-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi, sounds good. One thing this check is the license. We generally prefer BSD licensed stuff. As for what to add, SFTP is great, but adding SSH support would be just awesome. Do no be afraid about allocators, see e.g. Core/SSL/InitExit.icpp, I bet things will be pretty similar with SFTP. Mirek
creator: mirek
date: 2016-01-10T22:02:37-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello Mirek, Thanks for the tip, I'll look into Core/SSL/InitExit.icpp Ok then, next week I'll upload the Sftp, and then go for full SSH2 support in general, incrementally adding methods and classes to the package. (There are and will be some design decisions and trade-offs, though.) As to the license, I'd considered that before chosing libssh2. It has a revised BSD license (http://www.libssh2.org/license.html) Hope this is ok. Regards, Oblivion
creator: Oblivion
date: 2016-01-11T08:10:29-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Oblivion wrote on Mon, 11 January 2016 09:10 As to the license, I'd considered that before chosing libssh2. It has a revised BSD license (http://www.libssh2.org/license.html) Hope this is ok. Perfect!
creator: mirek
date: 2016-01-11T08:47:23-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello, Just to give you all an update: U++ wrapper for libssh2 is in good shape. SSHSession class: implemened. I borrowed the U++ memory allocation code (re/alloc, free) from Core/SSL package and after a slight modification glued it to the SSH package. (It is working!) Sftp class: Implemented all the necessary commands (seek/seek64/statvfs not yet implemented). I don't have access to Windows right now, so the code is currently tested only on an up-to-date arch linux installation (with a KDE/Plasma5 desktop). Below is a screenshot of a very simple (and easy to write) sftp downloader example with 10 concurrent downloads, demonstrating non-blocking/async operation capabilities. Below is the actual code responsible for 10 concurrent downloads, demonstrating sftp basic async api (Some parameters are hard coded. I was being lazy.) // Async jobs. struct Job { SFtp sftp; FileOut file; String path; int index; int cmd; }; enum Command { OPEN, READ, CLOSE, FINISH }; const char *sftp_server = "demo.wftpserver.com"; const char *sftp_user = "demo-user"; const char *sftp_pass = "demo-user"; const char *remote_file = "/download/F11_wallpaper_06_1600x1200.jpg"; void SFtpExample::Download() { // Initialize and fill an array of Job(s) for(int i = 0; i < 10; i++) { Job& job = jobs.Add(); job.sftp.User(sftp_user, sftp_pass); job.sftp.StartConnect(sftp_server, 2222); job.sftp.WhenDo = THISBACK(UpdateGui); job.file.Open(Format("/home/testuser/picture-%d.jpg", i)); job.cmd = OPEN; job.index = i; } // Actual loop: connects to the server and downloads a file in a concurrent way. while(jobs.GetCount()) { for(int i = 0; i < jobs.GetCount(); i++) { Job& job = jobs[i]; SocketWaitEvent e; e.Add(job.sftp.GetSocket()); e.Wait(10); job.sftp.Do(); if(!job.sftp.InProgress()) { if(job.sftp.IsSuccess()) { switch(job.cmd) { case OPEN: job.sftp.StartOpen(remote_file, SSH::READ); job.path = remote_file; job.cmd = READ; continue; case READ: job.sftp.StartGet(job.file, THISBACK2(DownloadProgress, job.index, …
creator: Oblivion
date: 2016-01-26T20:02:12-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Oblivion wrote on Tue, 26 January 2016 21:02 Hello, Just to give you all an update: U++ wrapper for libssh2 is in good shape. SSHSession class: implemened. I borrowed the U++ memory allocation code (re/alloc, free) from Core/SSL package and after a slight modification glued it to the SSH package. (It is working!) Sftp class: Implemented all the necessary commands (seek/seek64/statvfs not yet implemented). I don't have access to Windows right now, so the code is currently tested only on an up-to-date arch linux installation (with a KDE/Plasma5 desktop). Below is a screenshot of a very simple (and easy to write) sftp downloader example with 10 concurrent downloads, demonstrating non-blocking/async operation capabilities. Below is the actual code responsible for 10 concurrent downloads, demonstrating sftp basic async api (Some parameters are hard coded. I was being lazy.) // Async jobs. struct Job { SFtp sftp; FileOut file; String path; int index; int cmd; }; enum Command { OPEN, READ, CLOSE, FINISH }; const char *sftp_server = "demo.wftpserver.com"; const char *sftp_user = "demo-user"; const char *sftp_pass = "demo-user"; const char *remote_file = "/download/F11_wallpaper_06_1600x1200.jpg"; void SFtpExample::Download() { // Initialize and fill an array of Job(s) for(int i = 0; i < 10; i++) { Job& job = jobs.Add(); job.sftp.User(sftp_user, sftp_pass); job.sftp.StartConnect(sftp_server, 2222); job.sftp.WhenDo = THISBACK(UpdateGui); job.file.Open(Format("/home/testuser/picture-%d.jpg", i)); job.cmd = OPEN; job.index = i; } // Actual loop: connects to the server and downloads a file in a concurrent way. while(jobs.GetCount()) { for(int i = 0; i < jobs.GetCount(); i++) { Job& job = jobs[i]; SocketWaitEvent e; e.Add(job.sftp.GetSocket()); e.Wait(10); job.sftp.Do(); if(!job.sftp.InProgress()) { if(job.sftp.IsSuccess()) { switch(job.cmd) { case OPEN: job.sftp.StartOpen(remote_file, SSH::READ); job.path = remote_file; job.cmd = READ; continue; case READ: job.sftp.StartGet(job.…
creator: mirek
date: 2016-01-31T08:26:25-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello Mirek, Quote: I was thinking that it would be a cool demonstration (and a very nice PR CodeProject article) to create actual EDITOR of text files over FTP/FTPS/SFTP in "list of files in the left" style. Not sure I will have enough time for that, but it would be fun. Writing an FTP(S) & SFTP editor wouldn't be too difficult. We can use Any container and dynamically create both. Plus, directory enty parsers of the both classes are very similar. It would be easy to achive some abstract API when writing the editor. What worries me is that it'll end up in a somewhat complex code, and that might be bad for PR. Instead, we can write a UI skeleton and build two separate editors, one utilizing FTP(s), and the other SFTP. In this way, people can study the code(s) and the differences. While I am currently polishing SSH/SFtp package, I can start writing a basic FTPS based text editor, in the meantime. By the way, I was going to propose writing a CodeProject article for TcpSocket (I can write it in the following weeks, bu you may need to review it first ). IMO, it is a vital part of Upp/Core. Yet it is somewhat an enigma. It is well designed but poorly documented. (not talking about its API doc). I had to study HttpRequest/GuiWebCrawler to understand how to do nonblocking operations. Maybe we can clarify that. Also I would like to propose adding two methods for TcpSocket: My experiences with sockets and remote operations showed me that sending local IP over sockets is not uncommon. So, maybe it is possible to add a counterpart to GetPeerAddr() method: GetLocalAddr()? And of course EWOULDBLOCK is technically an error, but in practice we don't always treat it that way. So I believe adding a public WouldBlock() method to check if the socket would block would be helpful in writing nonblocking code. (GetError() is not very helpful when reading and studying the source code). Regards, Oblivion.
creator: Oblivion
date: 2016-01-31T23:06:24-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello guys, A major update: SSH package is re-written using the new version of AsyncQueue class which can utilize ordinary callbacks and lambdas, instead of the cumbersome VarArgs class. SSH package relies heavily on lambda functions, therefore it will require C++11. I believe it is much more elegant now, since it drastically simplifed the task and reduced the source code significantly. It is almost complete, and first release is at hand. (SFtp component, at least). Below code snippet is taken from the new version of a simple download test app that concurrently downloads a number of jpg files from a well-known public Sftp test server. As you can see, unnecessary state tracking is removed: struct Job { SFtp sftp; FileOut file; int index; }; Array<Job> jobs; //........ const char *sftp_server = "demo.wftpserver.com"; const char *sftp_user = "demo-user"; const char *sftp_password = "demo-user"; const char *remote_path = "/download"; bool SFtpDownloader::ReadDir(Ssh::DirList& ls) { // We'll get the directory listing, using a synchronous (blocking) call. SFtp browser; browser.WhenDo << [&] {ProcessEvents(); }; // GUI should be responsive... if(browser.Connect(sftp_server, 2222, sftp_user, sftp_password)) { // Let us use a convenience method which works on paths, not file descriptors. if(browser.ListDir(remote_path, ls)) return true; } Exclamation(browser.GetErrorDesc()); return false; } void SFtpDownloader::Download() { // Wait for the queue to be processed. if(!jobs.IsEmpty()) return; Ssh::DirList ls; if(!ReadDir(ls)) return; for(int i = 0, j = 0; i < ls.GetCount(); i++) { // We can work on directory entries easily. Ssh::DirEntry& e = ls[i]; const Ssh::Attrs& attrs = e.GetAttrs(); // Let us simply try to download all files with *.jpg extension to the current directory, asynchronously. // Here, thanks to AsyncQueue, asynchronous calls work in a fire-and-forget fashion. Similar to "batch processing". // E.g., any number of Sftp commands/jobs can be queued to be processed …
creator: Oblivion
date: 2016-03-06T22:39:46-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi Oblivion, This sounds interesting and might well fit my needs to automate some file transfers. Is it possible to get a hold of a copy of this SFTP client module of yours? Best regards, Tom
creator: Tom1
date: 2017-05-13T19:01:50-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Quote: Hi Oblivion, This sounds interesting and might well fit my needs to automate some file transfers. Is it possible to get a hold of a copy of this SFTP client module of yours? Best regards, Tom Hello Tom, Since you've revived this topic, let me give some information on recent developments. Some time ago I decided to go for a full SSH package. As of today, SFtp and Scp modules are complete, and Exec module is to-be-written. I'm cleaning up the codebase now. A technical preview version is scheduled, and will be available in June 10, 2017. I'll publish and further develop it for U++ with a BSD license, so of course you'll have the chance to get a copy. But I'm afraid you'll have to wait a little more. Regards, Oblivion
creator: Oblivion
date: 2017-05-14T14:43:54-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi Oblivion, Excellent! I'll stay tuned for the technical preview on this channel... If I'm not reacting to your technical preview announcement in a reasonable time, please PM me about the opportunity to test it. Best regards, Tom
creator: Tom1
date: 2017-05-14T15:46:48-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello everyone, It is now time to publish the technical preview version of SSH package for Upp. A brief status of the project: SSH package for U++ -------------------- SSH package is a libssh2 wrapper for Ultimate++. It aims to be a complete secure shell client solution. (SFtp, Scp, Exec, Terminal, X11Exec, KnownHosts) Currently it is at a technical preview stage, which means it is a work-in-progress, and subject to change. Requirements: - A C++ compiler with at least C++11 support. - libssh2 ssh library (https://www.libssh2.org/) Features and Highlights: - Tight integration with U++. - Support for both synchronous and asynchronous operations with both high level and low level methods. - Allows using U++ style memory [de/re]allocators with libssh2. - Ssh subsystems support RTTI, and share a common interface for polymorphism. Namely, it is possible to store different subsystems in the same U++ container. (e.g. SFtp, Exec, Scp, Shell can be all stored in the same Array.) Todo: - Add known hosts and ssh agents support. - Add more high level methods. - Add ssh X11 support. - Ship libssh2 library with the package. - Documentation. History: ----------------- 2017-06-28: EAGAIN is now properly handled across the subsystems. WhenWrite and WhenRead callbacks are removed in favour of parametrized gates. 2017-06-27: SFtp::StartStop: fixed. 2017-06-27: CleanUp() methods are added to the classes Ssh, SFtp, and Channel. This method utilizes JobQueue's WhenCleanup event. 2017-06-26: U++ memory [de/re]allocators added and made default. initial support for ssh shell, and terminal added. Subsystems now perform a clean up on failure to prevent possible heap leaks. 2017-06-22: Initial release. At the moment I am documenting the Package and its api. There are some important points you need to know before getting yourself familiar with the code. As I've mentioned in the above summary, SSH package is a libssh2 wrapper. But currently it does not contain libssh2 source code (which also has …
creator: Oblivion
date: 2017-06-27T23:13:27-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi Oblivion, This looks good. However, I may need to wait for the following item on your task list to be completed: - Ship libssh2 library with the package. Compiling libssh2 became a bit frustrating on my Windows system, so I decided to rather wait for libssh2 becoming part of the delivery. In my view, the libssh2 should be embedded inside your SSH package in a way that the whole package just gets built and statically linked in Windows like any other package without dependencies. Well, I guess the dependency of Core/SSL should be there to make it work. Best regards, Tom
creator: Tom1
date: 2017-06-28T12:46:22-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Quote: Re: SFTP or full SSH2 support for U++? Wed, 28 June 2017 15:46 Tom1 Hi Oblivion, This looks good. However, I may need to wait for the following item on your task list to be completed: - Ship libssh2 library with the package. Compiling libssh2 became a bit frustrating on my Windows system, so I decided to rather wait for libssh2 becoming part of the delivery. In my view, the libssh2 should be embedded inside your SSH package in a way that the whole package just gets built and statically linked in Windows like any other package without dependencies. Well, I guess the dependency of Core/SSL should be there to make it work. Best regards, Tom Hello Tom, Of course I am going to include the libssh2 package in the following days (probably next week). There's too much frustration about libssh2's windows build. And those complaints are usually right. The problem is that the libssh2 team didn't seem to provide a clear compilation guide. While I personally prefer OpenSSL, it is not mandatory for libssh2. Windows has CNG since Vista. I can make it an option too. Fortunately It is actually very easy to compile libssh2 on windows. (which makes you get even more frustrated, finding this after hours of of code-digging). In the mean time I can write a step by step guide (with screenshots) on how to compile it on Windows (using mingw64, with optional OpenSSL or WinCNG). Best regards, Oblivion.
creator: Oblivion
date: 2017-06-28T13:04:38-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi Oblivion, Thanks, but you do not need to write the libssh2 compilation guide for my sake. I'm not in that much hurry with this and I can certainly wait a little bit longer so that you can merge the libssh2 inside a U++ package. BTW: I wish this package is then compilable with MSC14 and above, since I cannot use mingw because of some special external library dependencies I have. Best regards, Tom
creator: Tom1
date: 2017-06-28T13:29:43-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: AFAIK Koldo (who had the chance to test my code first hand, and whom I should thank here!) has managed to compile it on VS 2015. IT needs slight changes (such as that some "."s in lambda functions should be substituted with "->"s), which I'll make this week end. The thing is, I do not use MSC. I guess I'll install it as a testing environment. Best regards, Oblivion
creator: Oblivion
date: 2017-06-28T13:50:27-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: OK. Keep up the good work! Best regards, Tom
creator: Tom1
date: 2017-06-28T14:04:00-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello, Technical preview version of the SSH package is updated. Now it contains the libssh2 library. And uses the OpenSSL supplied by U++. I've tested it with GCC on Linux, and Mingw64 on Windows. It is compiled successfully on both systems. I have yet to test it with MSC. Please test it. If it fails to compile, try configuring the libssh2 library, using libssh2_config.h header. That header file contains defs. Should you need to modify it, possible values can be copied from the libssh2/libssh2_config.h.in file. Maybe I should open a seperate topic on SSH package in the Bazaar section of the forums for the time being? Suggestions, patches, and cricisim is welcome. [See below post for the updated package] Best regards, Oblivion
creator: Oblivion
date: 2017-06-30T21:38:30-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi Oblivion, Thanks! Here are the quick results: I tried to compile your packages on Windows 10 Pro 32 bit with MSC14. I just copied all your packages to bazaar (locally). (Running on top of upp rev. 10804.) When compiling, I got the following errors and warnings for SSH: ----- SSH ( MSC14 MSC WIN32 ) (2 / 6) Ssh.cpp C:\upp-10804\bazaar\SSH\Ssh.cpp(28): error C2143: syntax error: missing ';' before '.' C:\upp-10804\bazaar\SSH\Ssh.cpp(28): error C2059: syntax error: '.' C:\upp-10804\bazaar\SSH\Ssh.cpp(35): error C2059: syntax error: '.' C:\upp-10804\bazaar\SSH\Ssh.cpp(46): error C2143: syntax error: missing ';' before '.' C:\upp-10804\bazaar\SSH\Ssh.cpp(46): error C2059: syntax error: '.' Channel.cpp C:\upp-10804\bazaar\SSH\Channel.cpp(134): warning C4244: '=': conversion from 'Upp::int64' to 'int', possible loss of data SFtp.cpp C:\upp-10804\bazaar\SSH\SFtp.cpp(269): warning C4244: 'argument': conversion from 'Upp::int64' to 'int', possible loss of data C:\upp-10804\bazaar\SSH\SFtp.cpp(269): warning C4244: 'argument': conversion from 'Upp::int64' to 'std::size_t', possible loss of data C:\upp-10804\bazaar\SSH\SFtp.cpp(277): warning C4554: '&': check operator precedence for possible error; use parentheses to clarify precedence c:\upp-10804\bazaar\ssh\ssh.h(165) : error C4716: 'Upp::SFtp::StartGetStat': must return a value c:\upp-10804\bazaar\ssh\ssh.h(174) : error C4716: 'Upp::SFtp::StartRealPath': must return a value c:\upp-10804\bazaar\ssh\sftp.cpp(231) : error C4716: 'Upp::SFtp::StartListDir': must return a value Exec.cpp Scp.cpp libssh2upp.c c:\upp-10804\bazaar\ssh\libssh2\libssh2_config.h(105): error C2375: 'snprintf': redefinition; different linkage c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1932): note: see declaration of 'snprintf' c:\upp-10804\bazaar\ssh\libssh2/agent.c(300): warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprintf_s instead. To disable de precation, use _CRT_SECURE_NO_WARNINGS. See on…
creator: Tom1
date: 2017-07-01T15:35:39-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Thank you very much for providing feedback, Tom! I've installed and configured the MSC to work with TheIDE. It now compiles on VS 2015. Hopefully the errors should be fixed now. Tested on: Windows: MINGW 32/64, MSC (2015) Linux: GCC Please test it. Best regards, Oblivion.
creator: Oblivion
date: 2017-07-02T15:20:41-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Dear Oblivion Thank you very much. It works again , and with libssh2 embedded! Just a couple of details (sorry for my perfectionism ): - Old scp demo does not work Cout() << "Getting file Demo.mo\n"; FileOut fout(AppendFileName(GetDesktopFolder(), "compile_run.sh")); const char *path = "compile_run.sh"; Scp scp(ssh); scp.WhenRead = [=](int64 total, int64 done) { return false; }; scp.WhenWrite = Proxy(scp.WhenRead); if(!scp.Get(fout, path)) Cout() << scp.GetErrorDesc(); fout.Close(); - SFtp::Init(), Stop(), and Ssh.WhenWait() do not exist. - Some warnings got by MSC15-64 SFtp.cpp C:\upp\bazaar\SSH\SFtp.cpp(102): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data C:\upp\bazaar\SSH\SFtp.cpp(136): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data C:\upp\bazaar\SSH\SFtp.cpp(151): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data C:\upp\bazaar\SSH\SFtp.cpp(181): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data C:\upp\bazaar\SSH\SFtp.cpp(237): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data C:\upp\bazaar\SSH\SFtp.cpp(252): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data C:\upp\bazaar\SSH\SFtp.cpp(269): waring C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data C:\upp\bazaar\SSH\SFtp.cpp(269): warning C4244: 'initializing': conversion from 'ssize_t' to 'int', possible loss of data C:\upp\bazaar\SSH\SFtp.cpp(300): warning C4244: 'initializing': conversion from 'ssize_t' to 'int', possible loss of data
creator: koldo
date: 2017-07-02T17:14:18-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi Oblivion, I tried SshAsyncRequests example. It compiled nicely on MSC14 32 bit with just a few obvious warnings from libssh2: ----- SSH ( MSC14 MSC WIN32 ) Ssh.cpp SFtp.cpp Channel.cpp Scp.cpp Exec.cpp Malloc.cpp libssh2upp.c c:\upp-10804\bazaar\ssh\libssh2/agent.c(300): warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprintf_s instead. To disable de precation, use _CRT_SECURE_NO_WARNINGS. See online help for details. c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1769): note: see declaration of 'sprintf' c:\upp-10804\bazaar\ssh\libssh2/channel.c(1157): warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprintf_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1769): note: see declaration of 'sprintf' c:\upp-10804\bazaar\ssh\libssh2/knownhost.c(957): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable de precation, use _CRT_SECURE_NO_WARNINGS. See online help for details. c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(207): note: see declaration of 'fopen' c:\upp-10804\bazaar\ssh\libssh2/knownhost.c(1181): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable d eprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(207): note: see declaration of 'fopen' c:\upp-10804\bazaar\ssh\libssh2/pem.c(257): warning C4018: '<=': signed/unsigned mismatch c:\upp-10804\bazaar\ssh\libssh2/userauth.c(541): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable dep recation, use _CRT_SECURE_NO_WARNINGS. See online help for details. c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(207): note: see declaration of '…
creator: Tom1
date: 2017-07-02T17:49:54-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello Koldo and Tom, Thank you both for testing the code, an dproviding me the crucial feedback! Quote: - SFtp::Init(), Stop(), and Ssh.WhenWait() do not exist. Yes, they are removed in favour of a better approach. I believe initialization and deinitializaton should be made automatic, analogous to RAII. If you request a resource, say a Scp channel, which should be allocated per request anyway, you simply spawn it and bind it to an ssh session. It will be automatically deallocated when the object is destroyed, as is with any C++ object: // Scp scp1; scp1.Session(session); // Scp channel is "scheduled" to be initialized. There is no need for manual deinitialization. It will be deinitialized at the end of the code block. //Or Scp scp2(session); // Scp channel is "scheduled" to be initialized. There is no need for manual deinitialization. It will be deinitialized at the end of the code block. In the same vein, you don't actually need to explicitly disconnect from the ssh server. (Of course you can, but it is by no means necessary). Ssh objects automatically attempt to disconnect when they are being destroyed. WhenWait was a member of JobQueue. But there is no need for WhenWait anymore. As you can see in the SshAsyncRequests example that I provided with the package, you can use SocketWaitEvent where it is useful. If you want to include it in a synchronous call, you can override JobQueue::Execute(). It is virtual. As for the old Scp Demo. Getters and Putters are now in U++ style. bool Get(Stream& out, const String& path, Gate<int64, int64> progress = Null); bool Put(Stream& in, const String& path, long mode, Gate<int64, int64> progress = Null); So the code needs a slight change: Cout() << "Getting file Demo.mo\n"; FileOut fout(AppendFileName(GetDesktopFolder(), "compile_run.sh")); const char *path = "compile_run.sh"; Scp scp(ssh); if(!scp.Get(fout, path, [=](int64 total, int64 done) { return false; })) // <-- Cout() << scp.GetErrorDesc(); fout.Close(); Regarding the warni…
creator: Oblivion
date: 2017-07-02T19:14:59-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: There is a problem that needs to be solved though. I've encountered this error recently. Connections to servers on localhost (127.0.0.1) fail for some unknown (yet) reason. It needs further investigation. As far as I can see, this failure doesn't happen when linking against externally (using Cmake) compiled libssh2. Probably I am missing some configuration switch (or a combination of switches?). libssh2 is a strange beast with a configure script which has over 20K LoC. It'll take some time (a week or so) to figure out what I'm missing. But of course in the meantime I'm open to suggestions, and I accept patches, etc. Best regards, Oblivion.
creator: Oblivion
date: 2017-07-03T05:55:53-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Thank you Oblivion. New scp interface works well
creator: koldo
date: 2017-07-03T06:39:21-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi Oblivion, I have successfully tested your examples on Windows 10 Professional 64-bit platform using MSC14, MSC14x64, MSC15 and MSC15x64 build methods. I guess the usual warnings on the deprecation of functions used by libssh2 could probably be suppressed by using a compiler flag associated to the SSH package. This is just a cosmetic issue though. Thank you very much for your effort on this task. Best regards, Tom
creator: Tom1
date: 2017-07-03T08:33:14-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello Koldo and Tom, It is nice to hear that the package is working for you. I have found the source of the problem that I mentioned in my last message (regarding the error where the ssh connection to servers on localhost was failing.) As it turns out, my suspicion was right. A preprocessor directive was missing: LIBSSH2_DH_GEX_NEW This preprocessor directive enables the newer diffie-hellman-group-exchange-sha1 syntax, which libssh2 relies on when compiled against recent versions of openssl. Also I fixed a critical error where the JobQueue can stuck in an infinite loop when the global timeout is set. Therefore please don't forget to update JobQueue too. You can find the updated package below. Now that the most annoying problem is solved, I can focus on completing the package (by adding knownhosts, X11 and terminal support, fixing warnings, etc.). Once the package is complete I am going to open a bazaar topic for SSH package. And if at that point Mirek and other U++ developers decide that SSH package can be official part of Upp, it has a new BSD license, and permission is hereby explicitly granted. (I'll gladly continue to maintain the package, since I'll be using it.) For the time being I'll provide a SVN link to the package, where you can always get the latest version: https://sourceforge.net/p/ultimatecomponents/svn/HEAD/tree/t runk/ Best regards, Oblivion
creator: Oblivion
date: 2017-07-03T19:51:06-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi Oblivion, For some reason the latest SSH.h had the include files for libssh2 changed from: #include "libssh2/libssh2.h" #include "libssh2/libssh2_sftp.h" #include "libssh2/libssh2_publickey.h" To: #include "libssh2.h" #include "libssh2_sftp.h" #include "libssh2_publickey.h" This effectively prevented compilation. I changed it back and now it compiles and runs again OK. Best regards, Tom
creator: Tom1
date: 2017-07-04T06:57:29-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Ah, while I was inspecting the cause for the connection error I changed the headers to test some cases. I was in a hurry, sorry for the inconvenience. I'll fix them tonight. Thanks! Best regards, Oblivion
creator: Oblivion
date: 2017-07-04T07:02:42-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: No worries, it was just a minor detail. Best regards, Tom
creator: Tom1
date: 2017-07-04T07:12:48-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello everyone, Next update to SSH package technical preview is finally here. Aside from bug fixes and cleanups, SSH package has now gained full support for three authentication methods (password, publickey, keyboard-interactive) properly. The crucial part was keyboard interaction (challenge/response) authentication method. It is implemented using a Function. Also, it is now possible to select between preferred transport methods. Last but not least, API reference docs are added for the most crucial classes. I am also writing a guide for its usage. But it's not finished yet. Agents, X11 support, and forwarding support are on their way way too! Now I'd like to ask you a question. This covers both SSH package and my FTP package. I'm planning to add to both package network proxy (HTTP/SOCKS4/4a/5) support using the new version of my NetworkProxy package (it will be renamed to NetProxy). I can think of three strategies: 1) Adding proxy support as an integral part to SSH and FTP packages. This will cause SSH and FTP packages to require NetProxy and I don't really prefer this. 2) Using a compiler flag for enabling proxy support. 3) Since NetworkProxy (and upcoming NetProxy) is a delegate class that simply takes an already initialized TcpSocket, connects it, and hands it over back to user, I can add a callback, say WhenProxy, to Ssh and Ftp classes, and hand over the socket to NetProxy there, and when the connection is established we can proceed. This way both packages won't require NetworkProxy at all, it will be completely optional. What do you think? Which strategy would be better? I appreciate suggections, bug reports, criticism, etc. Here's a digest: 2017-08-11: Initial api reference docs for Ssh, Ssh::SubSystem, SFtp, Scp, Exec, Knownhosts, are added. 2017-07-31: It is now possible to query, get, and set the possible transportation methods and exchange algorithms. Added Ssh::Method(), Ssh::Methods(), Ssh::GetMethod(), Ssh::GetMethods() methods. Ssh::Host class is from…
creator: Oblivion
date: 2017-08-12T20:49:18-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi Oblivion, Thanks for the update. Not sure if this has anything to do with the update, but I have trouble concatenating the following procedure to conditionally create a directory: ... if(session.IsSuccess()){ SFtp::DirList list; SFtp sftp(session); sftp.Clear(); sftp.StartListDir(directoryname,list); sftp.StartClose(); if(sftp.Execute()) return true; // Reading the directory successfully proves it exists sftp.Clear(); sftp.StartMakeDir(directoryname,0755); sftp.StartClose(); return sftp.Execute(); } ... The MakeDir part fails when done after ListDir. If I establish a separate SFtp instance for both ListDir and MakeDir, it works OK. Is it supposed to work that way? Best regards, Tom
creator: Tom1
date: 2017-08-14T10:41:27-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello Tom, tl;dr: I'll write a better reply and explanation, once I get back home. Also I'm writing a guide on it's usage. I can see what you are trying to achieve, but that's kinda' against the whole concept. I mean, in the snippet you provided, you are emulating synchronous calls with asynchronous methods, which is basically legitimate, but already done in the SSH packages synchronous calls : Synchrounous version may be (I don't know the code-flow): if(session.IsSuccess()){ SFtp::DirList list; SFtp sftp(session); if(!sftp.ListDir((directoryname, list)) return sftp.MakeDir(directoryname, 0755); } Asynchronous version (which really shouldn't be written this way. Just do not use this. Only to give you the idea.) : if(session.IsSuccess()){ SFtp::DirList list; SFtp sftp(session); sftp.StartListDir(directoryname, list); while(sftp.Do()); if(sftp.IsFailure()) { sftp.StartMakeDir(directoryname, 0755); while(sftp.Do()); return sftp.IsSucces(); } } You dont' need to call Clear() at all. Queue will be cleared automatically. It is supplied as a safety measure (which I'll cover it in the guide) for creating (custom) synchronous methods by the user. By the way, I am going to add high-level methods, such as IsFileExists(), IsDirectoryExists(), etc. Thanks for the feedback! Best regards, Oblivion
creator: Oblivion
date: 2017-08-14T12:26:06-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi, Thanks for your prompt reply. (Although I'm not in any hurry with this, whatsoever.) Well, I tried the async approach just to test, but I had the exact same problem with the synchronous one too .. i.e. This fails: if(session.IsSuccess()){ SFtp::DirList list; SFtp sftp(session); if(!sftp.ListDir(directoryname, list)) return sftp.MakeDir(directoryname, 0755); } But this works: if(session.IsSuccess()){ SFtp::DirList list; SFtp sftp(session); if(!sftp.ListDir(directoryname, list)){ SFtp sftp2(session); return sftp2.MakeDir(directoryname, 0755); } } The high level helpers, like you pictured in your message, would be great. Please check the naming of U++ counterparts to keep the access easy. (BTW: I think that I have seen IsOK() quite frequently in U++ instead of IsSuccess(), but I'm not sure if these are supposed to mean exactly the same thing.) Thanks and best regards, Tom
creator: Tom1
date: 2017-08-14T12:46:53-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello Tom, Quote: (BTW: I think that I have seen IsOK() quite frequently in U++ instead of IsSuccess(), but I'm not sure if these are supposed to mean exactly the same thing.) Queue model is an abstraction of Upp::HttpRequest's async model. I took Do(), IsSuccess(), IsFailure(), InProgress() from there. I think they are legitimate. And I see the source of the problem now, I'll fix it asap. Thanks! Best regards, Oblivion
creator: Oblivion
date: 2017-08-14T12:52:24-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi, I never used HttpRequest, so I'm not familiar with this. But no worries; I trust you know best what you're up to! Best regards, Tom
creator: Tom1
date: 2017-08-14T12:57:13-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello Mirek, Tom, Koldo, and everyone, It's been a while since I last updated the package. SSH package's been through a major rewrite, and I wanted to have a solid foundation to build it upon. Well, I believe it is finally here. New design hopefullt fixes the main problems of the previous one. To do this I focused on the major component of SSH package: SFtp, since it was the most complex part of it. I am here to share the results. Below package contains only Ssh and SFtp components for the time being. But please take your time to examine and test SFtp. It can do some nice tricks easily, such as peeking and poking at files, or -experimental- multithreading support (we'll return to that later.). I also included a simple testcase where you can see how SFtp class behaves. Note that, every other component (be it SCP, channel, exec) will behave in a similar way. Here is the highlights, and changes: 1) This version brings a major redesign of overall components. SSH package's classes are redesigned around a core class named SshCore. SshCore allows a uniform interface for both blocking and non-blocking modes of operation. Visible result of this new design is a single set of methods for both modes of operation for each class. Methods starting with "Start" prefix are removed in favor of a simple "NonBlocking()" switch. 2) In accordance with this behaviour, a sort of "feature parity" is successfully kept, thanks to the new design. Namely, in non-blocking mode, any result can be gathered using the new GetResult() method, which returns Value. 3) All SSH package classes now has pick semantics. 4) New design allows creation of higher-level ("complex") SFTP methods where needed. Unlike the simple SFTP methods, which work on file handles, Complex methods take care of (allocate/free) file handles internally. 5) Experimental multithreading support is added (using AsyncWork). This is still at a primitive stage. 6) Network proxy support is added. This new feature uses NetProxy (Http/Sock…
creator: Oblivion
date: 2017-10-15T21:10:37-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Thank you
creator: koldo
date: 2017-10-16T06:58:45-00:00

Re: SSH2 wrapper for U++
type: item
description: Hello, SSH package is updated. 2017-10-17: Class names finalized: Core/base class -> Ssh Session class -> SshSession SFtp class -> SFtp Channel class -> SshChannel Scp class -> Scp Exec class -> SshExec Knownhosts class -> SshHosts SshAgents -> (will be added as such) CreateSFtp(), CreateExec(), CreateScp(), CreateChannel() methods added to SshSession class. Logging further refined. Error management further refined. New version of crucial SSH components are re-added to the packaet. As you can see in the above history entry, component names are being finalized. I am satisfied with these new namings, and would like to hear your commennts. Also I added CreateSFtp(), CreateExec(), CreateScp(), CreateChannel() methods to SshSession class. Since now we have pick semantics, these methods are IMO more elegant than explicity constructing subsystem objecs. A simple use case would be as follows (below code is a part of SFtpDemo, by the way. (since Rebex test server is actually a SFtp server, with exec support, I did not change the example name): void GetFileInfoViaExec(SshSession& session) { DLOG("---- Getting directory listing of " << file << " using SshExec (blocking):"); auto exec = session.CreateExec(); DDUMP(exec(String(cmd) + file, Cout(), Cerr())); } As I announced in my previous message, SSH package recently gained a completely optional NetProxy support. (see Bazaar section https:// www.ultimatepp.org/forums/index.php?t=msg&th=10132&g oto=48812&#msg_48812 for NetProxy package) Here is the minimal example code (this is a real test code that works): CONSOLE_APP_MAIN { StdLogSetup(LOG_FILE | LOG_COUT | LOG_TIMESTAMP); // Ssh::Trace(); SshSession session; session.WhenProxy = [=](TcpSocket& sock) { NetProxy proxy(sock, proxy_host, proxy_port); return proxy.Timeout(10000) .Socks5() .Auth(proxy_username, proxy_password) .Connect(ssh_host, ssh_port); }; if(session.Connect(ssh_host, ssh_port, username, password)) { auto exec = session.CreateExec(); DDUMP(exec("ls -l /readme.txt…
creator: Oblivion
date: 2017-10-18T22:55:27-00:00

Re: SSH2 wrapper for U++
type: item
description: Hi Oblivion, I downloaded this latest version now after using the version from a few months back. In order to successfully compile the code, I had to comment out the entire contents of SFtpMT.cpp and the following lines in SFtp.h: AsyncWork<String> SFtpGet(SshSession& session, const String& path, Gate<int64, int64> progress = Null); AsyncWork<void> SFtpGet(SshSession& session, const String& source, const String& target, Gate<int64, int64> progress = Null); Additionally, I had to comment out the following in Core.cpp: DLOG("Initializing libssh2..."); and DLOG("Deinitializing libssh2..."); (There was some 0x40 character embedded in those lines that MSC did not like.) - Then there's one question: Now that the names have changed, is there a replacement for session.IsSuccess() ? (I just removed these calls in my code to see if SFTP still works. It did ) Thanks again for your work on SSH! Best regards, Tom
creator: Tom1
date: 2017-10-19T12:16:32-00:00

Re: SSH2 wrapper for U++
type: item
description: Hello Tom, Quote: I downloaded this latest version now after using the version from a few months back. In order to successfully compile the code, I had to comment out the entire contents of SFtpMT.cpp and the following lines in SFtp.h: AsyncWork<String> SFtpGet(SshSession& session, const String& path, Gate<int64, int64> progress = Null); AsyncWork<void> SFtpGet(SshSession& session, const String& source, const String& target, Gate<int64, int64> progress = Null); Ah yes, that's because I use U++ trunk (latest nightly builds). In case you didn't notice, there is now a class called AsyncWork in U++ core, which is a MT helper. Those functions use that. Quote: Then there's one question: Now that the names have changed, is there a replacement for session.IsSuccess() ? (I just removed these calls in my code to see if SFTP still works. It did Smile ) You don't need it. I got rid of the old ones. There is only IsError(). You can use the IsError() method to check success when it is necessary. Did you look into the provided SFtpDemo app? Take a look into it. It demonstartes the basic usage pattern for SFtp (same pattern is valid for other components too) If you use blocking mode, then you can simply check the return code (IsError will be still available though.) In non-blocking mode, A successful operation means there is no error. So again, you can simply check IsError(), and then get the result with GetResult() method if the invoked method returns a value. By the way, GetResult() method is needed in non-blocking mode for such methods as FileExists(), for we need to separate the state of operation (failure/success) from the return value of operation. Non-blocking: void GetFileSize(SshSession& session) { DLOG("---- Getting file size of " << file << " (non-blocking):"); auto sftp = session.CreateSFtp(); sftp.NonBlocking().GetSize(file); while(sftp.Do()) Sleep(1); if(sftp.IsError()) DDUMP(sftp.GetErrorDesc()); else DDUMP((int64) sftp.GetResult()); // In non-blocking mode, results …
creator: Oblivion
date: 2017-10-19T13:09:25-00:00

Re: SSH2 wrapper for U++
type: item
description: Hi Oblivion, Thanks for explaining the new details. I did not read the SFtpDemo this time, since I had already implemented client calls in my own software. So in effect I was just updating the SFTP code to its latest version. I should have paid more attention to the changes. Anyway, thanks for the update! I'll let you know if I run into any trouble with it. Best regards, Tom
creator: Tom1
date: 2017-10-23T06:29:10-00:00

SSH package for U++ (alpha version)
type: item
description: Hello, SSH package is updated. This version is the first "alpha" release. This version brings, - Multithreading support for SFtp, Scp, and SshExec classes. - Ssh agent authentication support. - SocketWaitEvents support for non-blocking I/O - Ssh known hosts support (re-added). - API documentation (updated) - SFtp::DirEntry class gained ToXml() method. - Keyboard (challenge/response) authentication (re-added) - Updated libssh2 - Various bugfixes I will soon add SshShell support. In fact there is already a prototype of SshShell for Posix based OSes. It can execute remote commands in real-time, and allows using editors such as nano, vim, emacs on console. It also works with GUI. I will add it once I clean its source code. However, adding shell support on windows is somewhat tricky (at least, adding a console (CLI) based version is not so trivial, for standard I/O handling is different on windows. I need to find a reliable way before I can include it in the pacakge, so it'll take some time...) I set up a GIT repo some time ago, where you can find the latest versions of my public U++ packages: https://github.com/ismail-yilmaz/upp-components/tree/master/ Core/SSH Below you can find the latest package. (I will open a Bazaar topic from the next release on, and update the package regularly.) It is tested on Linux and Windows (GCC/MINGW 7.2, MSC 14 (2017)). Suggestions, criticism, bug reports, reviews, testing is always welcome, Cheers! Oblivion
creator: Oblivion
date: 2017-11-13T20:25:12-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: LS, I use upp-win-11459.7z, vs2017 and Windows10 , I downloaded https://svn.code.sf.net/p/ultimatecomponents/svn/, I put the trunk directories in the bazaar directory. IpAddrInfo addrinfo; Error is that the variable 'addrinfo' is a type/structure in Ultimate. To solve error, is renaming variable 'addrinfo' in something else for example 'addrinfo2'.
creator: alkema_jm
date: 2017-11-14T07:03:36-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Quote: alkema_jm index.php?t=getfile&id=5432&private=0LS, I use upp-win-11459.7z, vs2017 and Windows10 , I downloaded https://svn.code.sf.net/p/ultimatecomponents/svn/, I put the trunk directories in the bazaar directory. IpAddrInfo addrinfo; Error is that the variable 'addrinfo' is a type/structure in Ultimate. To solve error, is renaming variable 'addrinfo' in something else for example 'addrinfo2'. Hello alkema_jm, Thank you for notifiying me about the issue. SVN repo is not yet updated. SSH package in SVN repo is outdated. I will update it tonight. https://github.com/ismail-yilmaz/upp-components/tree/master/ Core/SSH However, the name clash you've mentioned is already fixed in the GIT version above. Note that the new version is not entirely compatible with the old one. I suggest you update to it. It is a re-write with a much better design with a ton of bug-fixes and new features. (Transition should be fairly easy though), Ps.My previous message also includes the latest zip file which contains an example code demonstrating the basic usage of the latest SSH package. You may want to chech that too. Best regards, Oblivion
creator: Oblivion
date: 2017-11-14T07:38:32-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello Oblivion, I put the code in my C:\dev\upp\bazaar directory. I use each time fresh upp-win-11459.7z (source files), vs2017 and Windows10 See for the results the screendumps below. Greetings Jan Marco
creator: alkema_jm
date: 2017-11-14T10:57:05-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello Jan Marco, Thank you very much for feedback. Warnings aside (they are not really important here and will be addressed soon), Quick fix for SFtpDemo: You can make SFtpDemo work by changing SFtpGet to SFtp::AsyncGet (in SFtpDemo.cpp, at line 83) Other error messages: SSH, Job, NetProxy and FTP packages are 3rd party U++ libraries, and they are meant to be used in applications or other libraries. That's why you are getting linker errors about _main. Best regards, Oblivion
creator: Oblivion
date: 2017-11-14T11:41:05-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hi Oblivion, Thanks for the update. I just downloaded it and compiled it in and it still works just fine within my app. (That is for the SFTP part I'm using Thanks and best regards, Tom
creator: Tom1
date: 2017-11-14T13:27:38-00:00

Re: SFTP or full SSH2 support for U++?
type: item
description: Hello Tom, I am glad that it works for you. Since SSH package has now passed the technical-preview stage, and has a mostly stable API, I am opening a new topic on the Bazaar section and officially announcing it as a SSH supplement for Ultimate++ core library. From now on you can always find latest version, help, and SSH package-related news there: https://www.ultimatepp.org/forums/index.php?t=msg&th=101 72&start=0& Best regards, Oblivion
creator: Oblivion
date: 2017-11-14T20:51:26-00:00