Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Developing U++ » U++ Developers corner » SFTP or full SSH2 support for U++? (Discussion about implementing the SSH2 protocol using libssh2)
Re: SFTP or full SSH2 support for U++? [message #48411 is a reply to message #48410] Sun, 02 July 2017 19:14 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3354
Registered: August 2008
Senior Veteran
Dear Oblivion

Thank you very much. It works again Smile, and with libssh2 embedded!

Just a couple of details (sorry for my perfectionism Razz ):

- Old scp demo does not work
Cout() << "Getting file Demo.mo\n";
FileOut fout(AppendFileName(GetDesktopFolder(), "compile_run.sh"));
const char *path = "compile_run.sh";
Scp scp(ssh); 
scp.WhenRead  = [=](int64 total, int64 done) { return false; };
scp.WhenWrite = Proxy(scp.WhenRead);
if(!scp.Get(fout, path))
	Cout() << scp.GetErrorDesc();
fout.Close();


- SFtp::Init(), Stop(), and Ssh.WhenWait() do not exist.

- Some warnings got by MSC15-64
SFtp.cpp
C:\upp\bazaar\SSH\SFtp.cpp(102): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(136): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(151): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(181): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(237): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(252): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(269): waring C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(269): warning C4244: 'initializing': conversion from 'ssize_t' to 'int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(300): warning C4244: 'initializing': conversion from 'ssize_t' to 'int', possible loss of data


Best regards
Iñaki
Re: SFTP or full SSH2 support for U++? [message #48412 is a reply to message #48410] Sun, 02 July 2017 19:49 Go to previous messageGo to next message
Tom1
Messages: 1212
Registered: March 2007
Senior Contributor
Hi Oblivion,

I tried SshAsyncRequests example. It compiled nicely on MSC14 32 bit with just a few obvious warnings from libssh2:

----- SSH ( MSC14 MSC WIN32 )
Ssh.cpp
SFtp.cpp
Channel.cpp
Scp.cpp
Exec.cpp
Malloc.cpp
libssh2upp.c
c:\upp-10804\bazaar\ssh\libssh2/agent.c(300): warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprintf_s instead. To disable de
    precation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1769): note: see declaration of 'sprintf'
c:\upp-10804\bazaar\ssh\libssh2/channel.c(1157): warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprintf_s instead. To disable
     deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1769): note: see declaration of 'sprintf'
c:\upp-10804\bazaar\ssh\libssh2/knownhost.c(957): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable de
    precation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(207): note: see declaration of 'fopen'
c:\upp-10804\bazaar\ssh\libssh2/knownhost.c(1181): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable d
    eprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(207): note: see declaration of 'fopen'
c:\upp-10804\bazaar\ssh\libssh2/pem.c(257): warning C4018: '<=': signed/unsigned mismatch
c:\upp-10804\bazaar\ssh\libssh2/userauth.c(541): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable dep
    recation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(207): note: see declaration of 'fopen'
Creating library...
SSH: 7 file(s) built in (0:13.20), 1886 msecs / file, duration = 13547 msecs, parallelization 95%
C:/upp-10804/out/examples-bazaar/SSH/MSC14.\SSH.lib (1624178 B) created in (0:01.96)

OK. (0:13.64)



Then it executed with the following results:

Connecting to Rebex public sftp test server...
Unable to request SFTP subsystem
drwx------ 2 demo users          0 Oct 27  2015 .
drwx------ 2 demo users          0 Oct 27  2015 ..
-rw------- 1 demo users      14280 Mar 19  2007 ConsoleClient.png
-rw------- 1 demo users      15091 Mar 19  2007 ConsoleClientSmall.png
-rw------- 1 demo users      15836 Mar 19  2007 FtpDownloader.png
-rw------- 1 demo users      19156 Feb 16  2007 imap-console-client.png
-rw------- 1 demo users      36672 Mar 19  2007 KeyGenerator.png
-rw------- 1 demo users      24029 Mar 19  2007 KeyGeneratorSmall.png
-rw------- 1 demo users      16471 Feb 16  2007 mail-editor.png
-rw------- 1 demo users      35414 Feb 16  2007 mail-send-winforms.png
-rw------- 1 demo users      49011 Feb 16  2007 mime-explorer.png
-rw------- 1 demo users      58024 Mar 19  2007 pocketftp.png
-rw------- 1 demo users      20197 Mar 19  2007 pocketftpSmall.png
-rw------- 1 demo users      20472 Feb 16  2007 pop3-browser.png
-rw------- 1 demo users      11205 Feb 16  2007 pop3-console-client.png
-rw------- 1 demo users        407 Mar 23  2007 readme.txt
-rw------- 1 demo users      11546 Mar 19  2007 ResumableTransfer.png
-rw------- 1 demo users       2635 Mar 19  2007 winceclient.png
-rw------- 1 demo users       6146 Mar 19  2007 winceclientSmall.png
-rw------- 1 demo users      80000 Mar 19  2007 WinFormClient.png
-rw------- 1 demo users      17911 Mar 19  2007 WinFormClientSmall.png
Remote command processed. Return code is 0.
<--- Finished in (0:01.79), exitcode: 0 --->


I wonder if "Unable to request SFTP subsystem" was to be expected here.

Anyway, I will do more testing tomorrow. Smile

Thanks and best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48413 is a reply to message #48411] Sun, 02 July 2017 21:14 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello Koldo and Tom,

Thank you both for testing the code, an dproviding me the crucial feedback!


Quote:
- SFtp::Init(), Stop(), and Ssh.WhenWait() do not exist.


Yes, they are removed in favour of a better approach.
I believe initialization and deinitializaton should be made automatic, analogous to RAII.
If you request a resource, say a Scp channel, which should be allocated per request anyway, you simply spawn it and bind it to an ssh session.
It will be automatically deallocated when the object is destroyed, as is with any C++ object:

// 
Scp scp1;
scp1.Session(session);                     // Scp channel is "scheduled" to be initialized. There is no need for manual deinitialization. It will be deinitialized at the end of the code block.
//Or
Scp scp2(session);                         // Scp channel is "scheduled" to be initialized. There is no need for manual deinitialization. It will be deinitialized at the end of the code block.


In the same vein, you don't actually need to explicitly disconnect from the ssh server. (Of course you can, but it is by no means necessary).
Ssh objects automatically attempt to disconnect when they are being destroyed.


WhenWait was a member of JobQueue. But there is no need for WhenWait anymore.
As you can see in the SshAsyncRequests example that I provided with the package, you can use SocketWaitEvent where it is useful.
If you want to include it in a synchronous call, you can override JobQueue::Execute(). It is virtual. Smile

As for the old Scp Demo.
Getters and Putters are now in U++ style.
 bool Get(Stream& out, const String& path, Gate<int64, int64> progress = Null); 
 bool Put(Stream& in, const String& path, long mode, Gate<int64, int64> progress = Null);      


So the code needs a slight change:

Cout() << "Getting file Demo.mo\n";
FileOut fout(AppendFileName(GetDesktopFolder(), "compile_run.sh"));
const char *path = "compile_run.sh";
Scp scp(ssh); 
if(!scp.Get(fout, path, [=](int64 total, int64 done) { return false; })) // <-- 
    Cout() << scp.GetErrorDesc();
fout.Close();


Regarding the warnings, I'll try to fix them. Main problem is that Stream return int64 for file size, etc. but libssh2 functions expect size_t.


Hello Tom,

Those warnings are typical and can be suppressed. Usually they are not harmful. But I'll see what I can do.

Quote:
I wonder if "Unable to request SFTP subsystem" was to be expected here.


Yes, this is expected. It can happen time to time. But it should only be a rare case. If not, then there might be a problem with initialization.
I'll look into it.

There are two things to note here though:
1) Remote servers are not always reliable. Rebex is fine but response times of it's public test server are somewhat slow.

2) There is a "little" problem with ssh protocol and it's implementations: Ssh session works over a single socket.
Each ssh subsystem, be it sftp or scp, are actually channels on a single socket. And AFAIK there is no official way to "wait" on channels.
So, spawning more than one channel, especially when they are of different subsystems, can result in a such error.
It is also discussed in the libssh2 documents, There is a suggested solution for this by the libssh2 devs, but it is not implemented yet:

libssh2/TODO, lines 80 to 120 reads:
New Transport API
=================

THE PROBLEM

The problem in a nutshell is that when an application opens up multiple
channels over a single session, those are all using the same socket. If the
application is then using select() to wait for traffic (like any sensible app
does) and wants to act on the data when select() tells there is something to
for example read, what does an application do?

With our current API, you have to loop over all the channels and read from
them to see if they have data. This effectively makes blocking reads
impossible. If the app has many channels in a setup like this, it even becomes
slow. (The original API had the libssh2_poll_channel_read() and libssh2_poll()
to somewhat overcome this hurdle, but they too have pretty much the same
problems plus a few others.)

Traffic in the other direction is similarly limited: the app has to try
sending to all channels, even though some of them may very well not accept any
data at that point.

A SOLUTION

I suggest we introduce two new helper functions:

 libssh2_transport_read()

 - Read "a bunch" of data from the given socket and returns information to the
   app about what channels that are now readable (ie they will not block when
   read from). The function can be called over and over and it will repeatedly
   return info about what channels that are readable at that moment.

 libssh2_transport_write()

 - Returns information about what channels that are writable, in the sense
   that they have windows set from the remote side that allows data to get
   sent. Writing to one of those channels will not block. Of course, the
   underlying socket may only accept a certain amount of data, so at the first
   short return, nothing more should be attempted to get sent until select()
   (or equivalent) has been used on the master socket again.

I haven't yet figured out a sensible API for how these functions should return
that info, but if we agree on the general principles I guess we can work that
out.


For the time being, best workaround for this problem seems to be increasing the socket "wait" time and adopting a "max. retries" approach. Smile

I am in the process of writing the api docs, and a guide for SSH protocol and package, I'll discuss these problems there too.

Best regards,

Oblivion



[Updated on: Mon, 03 July 2017 08:15]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48414 is a reply to message #45813] Mon, 03 July 2017 07:55 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
There is a problem that needs to be solved though. I've encountered this error recently.

Connections to servers on localhost (127.0.0.1) fail for some unknown (yet) reason. It needs further investigation.
As far as I can see, this failure doesn't happen when linking against externally (using Cmake) compiled libssh2.
Probably I am missing some configuration switch (or a combination of switches?).
libssh2 is a strange beast with a configure script which has over 20K LoC. It'll take some time (a week or so) to figure out what I'm missing.
But of course in the meantime I'm open to suggestions, and I accept patches, etc.

Best regards,
Oblivion.


[Updated on: Mon, 03 July 2017 08:17]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48415 is a reply to message #48414] Mon, 03 July 2017 08:39 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3354
Registered: August 2008
Senior Veteran
Thank you Oblivion. New scp interface works well Smile

Best regards
Iñaki
Re: SFTP or full SSH2 support for U++? [message #48416 is a reply to message #48414] Mon, 03 July 2017 10:33 Go to previous messageGo to next message
Tom1
Messages: 1212
Registered: March 2007
Senior Contributor
Hi Oblivion,

I have successfully tested your examples on Windows 10 Professional 64-bit platform using MSC14, MSC14x64, MSC15 and MSC15x64 build methods.

I guess the usual warnings on the deprecation of functions used by libssh2 could probably be suppressed by using a compiler flag associated to the SSH package. This is just a cosmetic issue though.

Thank you very much for your effort on this task.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48435 is a reply to message #48416] Mon, 03 July 2017 21:51 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello Koldo and Tom,

It is nice to hear that the package is working for you. Smile
I have found the source of the problem that I mentioned in my last message (regarding the error where the ssh connection to servers on localhost was failing.)
As it turns out, my suspicion was right. A preprocessor directive was missing: LIBSSH2_DH_GEX_NEW
This preprocessor directive enables the newer diffie-hellman-group-exchange-sha1 syntax, which libssh2 relies on when compiled against recent versions of openssl.

Also I fixed a critical error where the JobQueue can stuck in an infinite loop when the global timeout is set. Therefore please don't forget to update JobQueue too.

You can find the updated package below.

Now that the most annoying problem is solved, I can focus on completing the package (by adding knownhosts, X11 and terminal support, fixing warnings, etc.).

Once the package is complete I am going to open a bazaar topic for SSH package.
And if at that point Mirek and other U++ developers decide that SSH package can be official part of Upp, it has a new BSD license, and permission is hereby explicitly granted.
(I'll gladly continue to maintain the package, since I'll be using it.)

For the time being I'll provide a SVN link to the package, where you can always get the latest version:

https://sourceforge.net/p/ultimatecomponents/svn/HEAD/tree/t runk/



Best regards,
Oblivion


[Updated on: Mon, 03 July 2017 22:36]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48437 is a reply to message #48435] Tue, 04 July 2017 08:57 Go to previous messageGo to next message
Tom1
Messages: 1212
Registered: March 2007
Senior Contributor
Hi Oblivion,

For some reason the latest SSH.h had the include files for libssh2 changed from:

#include "libssh2/libssh2.h"
#include "libssh2/libssh2_sftp.h"
#include "libssh2/libssh2_publickey.h"


To:

#include "libssh2.h"
#include "libssh2_sftp.h"
#include "libssh2_publickey.h"


This effectively prevented compilation. I changed it back and now it compiles and runs again OK.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48438 is a reply to message #48437] Tue, 04 July 2017 09:02 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Ah, while I was inspecting the cause for the connection error I changed the headers to test some cases.
I was in a hurry, sorry for the inconvenience. Embarassed
I'll fix them tonight. Thanks!

Best regards,
Oblivion


[Updated on: Tue, 04 July 2017 09:03]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48439 is a reply to message #48438] Tue, 04 July 2017 09:12 Go to previous messageGo to next message
Tom1
Messages: 1212
Registered: March 2007
Senior Contributor
No worries, it was just a minor detail.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48650 is a reply to message #45813] Sat, 12 August 2017 22:49 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello everyone,

Next update to SSH package technical preview is finally here.

Aside from bug fixes and cleanups, SSH package has now gained full support for three authentication methods (password, publickey, keyboard-interactive) properly.
The crucial part was keyboard interaction (challenge/response) authentication method. It is implemented using a Function.
Also, it is now possible to select between preferred transport methods.
Last but not least, API reference docs are added for the most crucial classes. Smile
I am also writing a guide for its usage. But it's not finished yet.

Agents, X11 support, and forwarding support are on their way way too!

Now I'd like to ask you a question. This covers both SSH package and my FTP package.
I'm planning to add to both package network proxy (HTTP/SOCKS4/4a/5) support using the new version of my NetworkProxy package (it will be renamed to NetProxy).

I can think of three strategies:

1) Adding proxy support as an integral part to SSH and FTP packages. This will cause SSH and FTP packages to require NetProxy and I don't really prefer this.
2) Using a compiler flag for enabling proxy support.
3) Since NetworkProxy (and upcoming NetProxy) is a delegate class that simply takes an already initialized TcpSocket, connects it, and hands it over back to user, I can add a callback, say WhenProxy, to Ssh and Ftp classes, and hand over the socket to NetProxy there, and when the connection is established we can proceed. This way both packages won't require NetworkProxy at all, it will be completely optional.

What do you think? Which strategy would be better?

I appreciate suggections, bug reports, criticism, etc.


Here's a digest:
2017-08-11:  Initial api reference docs for Ssh, Ssh::SubSystem, SFtp, Scp, Exec, Knownhosts, are added.
2017-07-31:  It is now possible to query, get, and set the possible transportation methods and exchange 
             algorithms. Added Ssh::Method(), Ssh::Methods(), Ssh::GetMethod(), Ssh::GetMethods() methods. 
             Ssh::Host class is from now on KnownHosts class.
2017-07-21:  Authentication methods (password, public key, keyboard interaction) are properly implemented.
             From now on it is possible to choose between authentication methods both on initialization,
             and on-the-fly (i.e. while logging in, using WhenAuth callback). 
             KnownHosts class is added to the package. This class provides basic known hosts support.
             It is now possible to verify and trust servers.


Best regards,
Oblivion


[Updated on: Sat, 12 August 2017 23:03]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48659 is a reply to message #48650] Mon, 14 August 2017 12:41 Go to previous messageGo to next message
Tom1
Messages: 1212
Registered: March 2007
Senior Contributor
Hi Oblivion,

Thanks for the update. Smile

Not sure if this has anything to do with the update, but I have trouble concatenating the following procedure to conditionally create a directory:
...
	if(session.IsSuccess()){
		SFtp::DirList list;
		SFtp sftp(session);
			
		sftp.Clear();
		sftp.StartListDir(directoryname,list);
		sftp.StartClose();
		if(sftp.Execute()) return true; // Reading the directory successfully proves it exists

		sftp.Clear();
		sftp.StartMakeDir(directoryname,0755);
		sftp.StartClose();
		return sftp.Execute();
	}
...


The MakeDir part fails when done after ListDir. If I establish a separate SFtp instance for both ListDir and MakeDir, it works OK. Is it supposed to work that way?

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48662 is a reply to message #48659] Mon, 14 August 2017 14:26 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello Tom,

tl;dr: I'll write a better reply and explanation, once I get back home. Also I'm writing a guide on it's usage. Smile

I can see what you are trying to achieve, but that's kinda' against the whole concept.
I mean, in the snippet you provided, you are emulating synchronous calls with asynchronous methods, which is basically legitimate, but already done in the SSH packages synchronous calls Smile :

Synchrounous version may be (I don't know the code-flow):

if(session.IsSuccess()){
		SFtp::DirList list;
		SFtp sftp(session);
		if(!sftp.ListDir((directoryname, list))
                   return sftp.MakeDir(directoryname, 0755);
               
}


Asynchronous version (which really shouldn't be written this way. Just do not use this. Only to give you the idea.) :
if(session.IsSuccess()){
		SFtp::DirList list;
		SFtp sftp(session);
		
                sftp.StartListDir(directoryname, list);
                while(sftp.Do()); 
                if(sftp.IsFailure()) {
                    sftp.StartMakeDir(directoryname, 0755);
                    while(sftp.Do());
                    return sftp.IsSucces();
                }
}


You dont' need to call Clear() at all. Queue will be cleared automatically.
It is supplied as a safety measure (which I'll cover it in the guide) for creating (custom) synchronous methods by the user.

By the way, I am going to add high-level methods, such as IsFileExists(), IsDirectoryExists(), etc. Smile

Thanks for the feedback!

Best regards,
Oblivion


[Updated on: Mon, 14 August 2017 14:46]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48663 is a reply to message #48662] Mon, 14 August 2017 14:46 Go to previous messageGo to next message
Tom1
Messages: 1212
Registered: March 2007
Senior Contributor
Hi,

Thanks for your prompt reply. (Although I'm not in any hurry with this, whatsoever.)

Well, I tried the async approach just to test, but I had the exact same problem with the synchronous one too .. i.e. This fails:
if(session.IsSuccess()){
		SFtp::DirList list;
		SFtp sftp(session);
		if(!sftp.ListDir(directoryname, list))
                   return sftp.MakeDir(directoryname, 0755);
               
}



But this works:

if(session.IsSuccess()){
		SFtp::DirList list;
		SFtp sftp(session);
		if(!sftp.ListDir(directoryname, list)){
			SFtp sftp2(session);
			return sftp2.MakeDir(directoryname, 0755);
		}
               
}


The high level helpers, like you pictured in your message, would be great. Please check the naming of U++ counterparts to keep the access easy. (BTW: I think that I have seen IsOK() quite frequently in U++ instead of IsSuccess(), but I'm not sure if these are supposed to mean exactly the same thing.)

Thanks and best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48664 is a reply to message #48663] Mon, 14 August 2017 14:52 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello Tom,

Quote:
(BTW: I think that I have seen IsOK() quite frequently in U++ instead of IsSuccess(), but I'm not sure if these are supposed to mean exactly the same thing.)


Queue model is an abstraction of Upp::HttpRequest's async model. I took Do(), IsSuccess(), IsFailure(), InProgress() from there. I think they are legitimate.


And I see the source of the problem now, I'll fix it asap.
Thanks!


Best regards,
Oblivion


[Updated on: Mon, 14 August 2017 14:58]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48665 is a reply to message #48664] Mon, 14 August 2017 14:57 Go to previous messageGo to next message
Tom1
Messages: 1212
Registered: March 2007
Senior Contributor
Hi,

I never used HttpRequest, so I'm not familiar with this.

But no worries; I trust you know best what you're up to! Smile

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48862 is a reply to message #45813] Sun, 15 October 2017 23:10 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello Mirek, Tom, Koldo, and everyone,

It's been a while since I last updated the package.

SSH package's been through a major rewrite, and I wanted to have a solid foundation to build it upon. Well, I believe it is finally here.

New design hopefullt fixes the main problems of the previous one. To do this I focused on the major component of SSH package: SFtp, since it was the most complex part of it.

I am here to share the results. Below package contains only Ssh and SFtp components for the time being. But please take your time to examine and test SFtp. It can do some

nice tricks easily, such as peeking and poking at files, or -experimental- multithreading support (we'll return to that later.).

I also included a simple testcase where you can see how SFtp class behaves. Note that, every other component (be it SCP, channel, exec) will behave in a similar way. Smile

Here is the highlights, and changes:

         1) This version brings a major redesign of overall components. SSH package's classes
             are redesigned around a core class named SshCore. SshCore allows a uniform interface
             for both blocking and non-blocking modes of operation. Visible result of this new 
             design is a single set of methods for both modes of operation for each class. Methods
             starting with "Start" prefix are removed in favor of a simple "NonBlocking()" switch.

             2) In accordance with this behaviour, a sort of "feature parity" is successfully 
                kept, thanks to the new design. Namely, in non-blocking mode, any result can be
                gathered using the new GetResult() method, which returns Value.

	     3) All SSH package classes now has pick semantics.
            
	     4) New design allows creation of higher-level ("complex") SFTP methods where needed.
                Unlike the simple SFTP methods, which work on file handles, Complex methods take 
                care of (allocate/free) file handles internally.
            
	     5) Experimental multithreading support is added (using AsyncWork). This is still
                at a primitive stage. 
	
	     6) Network proxy support is added. This new feature uses NetProxy (Http/Socks4/4a/5)
		package It works through a plugin, provided via Ssh::WhenProxy callback. And it
                is completely optional.

             7) Cancellation mechanism is added. Any operation can be cancelled at any time using
                the Cancel() method.

             8) SFtp::DirEntry class now has a ToString() method. (It will also gain an ToXml()
                method.) This method will give an output similar to Unix ls command.

             9) libssh2 configured to use OpenSSL by default (WinCNG will be made a compile-time
                option in the next release).


The code is not entirely cleaned-up, so there maybe also small cosmetic issues (for now.)

I appreciate feedbacks, patches, criticism, etc.

[ See below posts for updated package]

Cheers!

Oblivion


[Updated on: Wed, 18 October 2017 23:57]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48863 is a reply to message #48862] Mon, 16 October 2017 08:58 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3354
Registered: August 2008
Senior Veteran
Thank you Smile

Best regards
Iñaki
Re: SSH2 wrapper for U++ [message #48875 is a reply to message #45813] Thu, 19 October 2017 00:55 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

SSH package is updated.

2017-10-17:  Class names finalized:
             Core/base class  -> Ssh
             Session class    -> SshSession
             SFtp class       -> SFtp
             Channel class    -> SshChannel
             Scp class        -> Scp
             Exec class       -> SshExec
             Knownhosts class -> SshHosts
             SshAgents        -> (will be added as such)

             CreateSFtp(), CreateExec(), CreateScp(), CreateChannel() methods added to SshSession
             class. 
             Logging further refined.
             Error management further refined.


New version of crucial SSH components are re-added to the packaet.

As you can see in the above history entry, component names are being finalized. I am satisfied with these new namings, and would like to hear your commennts.

Also I added CreateSFtp(), CreateExec(), CreateScp(), CreateChannel() methods to SshSession class.

Since now we have pick semantics, these methods are IMO more elegant than explicity constructing subsystem objecs.

A simple use case would be as follows (below code is a part of SFtpDemo, by the way. (since Rebex test server is actually a SFtp server, with exec support, I did not change the example name):

void GetFileInfoViaExec(SshSession& session)
{
	DLOG("---- Getting directory listing of " << file << " using SshExec (blocking):");
	auto exec = session.CreateExec();
	DDUMP(exec(String(cmd) + file, Cout(), Cerr()));
	
}



As I announced in my previous message, SSH package recently gained a completely optional NetProxy support.
(see Bazaar section https:// www.ultimatepp.org/forums/index.php?t=msg&th=10132&g oto=48812&#msg_48812 for NetProxy package)
Here is the minimal example code (this is a real test code that works):
CONSOLE_APP_MAIN
{
    StdLogSetup(LOG_FILE | LOG_COUT | LOG_TIMESTAMP);

//  Ssh::Trace();
    SshSession session;
    session.WhenProxy = [=](TcpSocket& sock) {
        NetProxy proxy(sock, proxy_host, proxy_port);
        return proxy.Timeout(10000)
                    .Socks5()
                    .Auth(proxy_username, proxy_password)
                    .Connect(ssh_host, ssh_port);
    };
    if(session.Connect(ssh_host, ssh_port, username, password)) {
        auto exec = session.CreateExec();
        DDUMP(exec("ls -l /readme.txt", Cout(), Cerr());
    }
    else Cerr() << session.GetErrorDesc() << "\n";
}



RTTI support is also re-implemented.

Soon I'll move SSH package out of technical preview phase to beta. But before that I have to add documentation, a proper multithreading support, and missing classes (SshHost, SshAgents).

On multithreading:

The design of SSH core (base) class "ssh" allowed me to easily add an "internal" MT support using a compile-time switch (I did not yet publish that code).
But I haven't really decided to add MT that way. Using convenience functions (SFtpGet, SFtpPut, ScpGet, ScpPut, AsyncExec) seem a simpler and better managable way to go.
But I'd like to hear yout thoughts on that matter too.

I apprecieate reviews, bug reports, patches, suggestions, criticism.

Best regards,
Oblivion.





[Updated on: Thu, 19 October 2017 10:13]

Report message to a moderator

Re: SSH2 wrapper for U++ [message #48879 is a reply to message #48875] Thu, 19 October 2017 14:16 Go to previous messageGo to previous message
Tom1
Messages: 1212
Registered: March 2007
Senior Contributor
Hi Oblivion,

I downloaded this latest version now after using the version from a few months back.

In order to successfully compile the code, I had to comment out the entire contents of SFtpMT.cpp and the following lines in SFtp.h:

AsyncWork<String>   SFtpGet(SshSession& session, const String& path, Gate<int64, int64> progress = Null);
AsyncWork<void>     SFtpGet(SshSession& session, const String& source, const String& target, Gate<int64, int64> progress = Null);


Additionally, I had to comment out the following in Core.cpp:

	DLOG("Initializing libssh2...");
and
	DLOG("Deinitializing libssh2...");


(There was some 0x40 character embedded in those lines that MSC did not like.)

-

Then there's one question: Now that the names have changed, is there a replacement for session.IsSuccess() ? (I just removed these calls in my code to see if SFTP still works. It did Smile )

Thanks again for your work on SSH!

Best regards,

Tom

[Updated on: Thu, 19 October 2017 14:17]

Report message to a moderator

Previous Topic: Help needed with link errors (serversocket)
Next Topic: Ultimate++ repo and chat?
Goto Forum:
  


Current Time: Thu Mar 28 11:16:41 CET 2024

Total time taken to generate the page: 0.01669 seconds