U++ framework
Do not panic. Ask here before giving up.

Home » Developing U++ » U++ Developers corner » SFTP or full SSH2 support for U++? (Discussion about implementing the SSH2 protocol using libssh2)
SFTP or full SSH2 support for U++? [message #45813] Sun, 10 January 2016 12:58 Go to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello,

Fırst, I wish you all a happy new year!


Some time ago when I uploaded my FTPS implementation to the bazaar section, Daniel (Unodgs) asked me If I have any plans to implement an SFTP class.
And I'd said yes.

Now I am here to deliver my promise. Smile


But before I upload the package to the bazaar section, I have to make a choice and would like to hear your opinion on the subject.


First of all, I decided not to implement the SSH2 protocol from the scratch. I could've, but it would be a) tedious and b) wasting my time.
So I decided searching for alternatives.
The best alternative that I could have come up with was wrapping an existing, multiplatform (Linux and Windows, at least) library.
I found out that libssh2 was the way to go. It was plain C, easy to wrap up and had both blocking and non-blocking modes of operation available.

I finally implemented an SFTP class for U++ using libssh2, which has the below features:


1) Allows blocking and non-blocking operations. Namely, it can work in both synchronous and asynchronous modes. 
   I use HttpRequest and NetworkProxy class' asynchronous design here too, since that design proved very effective.

2) Has OPTİONAL network proxy (HTTP_CONNECT, SOCKS4/4a/5) support, using NetworkProxy class.

3) Takes advantage of U++ streams in file upload and download operations, and uses gates for progress tracking when needed.

4) Supports both MD5 and SHA 1 methods.

5) Has SFtp::DirEntry class for easy parsing of directory entries.




In short, currently we have a sync/async SFTP wrapper that can authenticate using "public key" method and/or Username/password combination, and perform Open/Close/Read/Write/List/Rename/Delete operations on remote file system entries. Also its API is very similar to my FTPS class, and its design derives from NetworkProxy and HttpRequest classes and follows the U++ coding style.

Now, the class is abstracted nicely so that I can either let it be a standaloune, robust SFtp class, or take it one step further and wrap up the whole functionality of libssh2.
Latter means adding SSHSession, SSHChannel, Scp classes to the package too.
What is your opinion?

(For example I can also easily add the SCP protocol to the package. SCP is known to perform better in some type file read/wtire operations.)


Also, since the libssh2 uses its own allocators (alloc, realloc, dealloc) but also gives the user the choice to implement his/her own, I'll definietly need help here.
I'm not experienced with U++ allocators.

Here is an actual, barebone SFtp example, demonstrating directory listing and file download, using a public SFtp test server:


#include <Core/Core.h>
#include <SFTP/SFTP.h>

using namespace Upp;

static const char* CRLF = "\r\n";

// This callback allows getting directory entries one by one.
// It is optional.
bool ListDirectory(SFtp::DirEntry& e) 
{

	
	String ls = e.GetEntry();
	if(!ls.IsEmpty())
		// When available, traditiona UNIX style directory listing can be used.
		Cout() << e.GetEntry() << CRLF;
	else {
		Cout() 	<< e.GetName()			        << CRLF
				<< e.GetUserID()		<< CRLF
				<< e.GetGroupID()    	        << CRLF
				<< e.GetSize() 			<< CRLF
				<< e.GetLastAccessed()	        << CRLF
				<< e.GetLastModified()	        << CRLF
				<< "Permissions:"		<< CRLF
				<< "R: " << e.IsReadable() 	<< ", "
				<< "W: " << e.IsWritable() 	<< ", "
				<< "X: " << e.IsExecutable()    << CRLF;
	}
	return false;
}

CONSOLE_APP_MAIN
{
	// This example demonstrates directory listing (ls) and file download operations,
	// using a public SFTP test server.
	// Note that this example uses blocking mode. It is synchronous. 
	// There is also a non-blocking, asynchhronous mode.
	
	FileOut wallpaper("/home/genericuser/Wallpapers/SFTP-download-test-wallpaper.jpg");
	SFtp::DirList dirlist; // SFtp::DirList is a vector containing SFtp:DirEnty elements.

	// Enable logging.
	SFtp::Trace();

	SFtp sftp;
	if(sftp.User("demo-user", "demo-user").Connect("demo.wftpserver.com", 2222)) {
		// Get server banner.
		Cout() << sftp.GetBanner() << CRLF;
		if(sftp.OpenDir("/download") && sftp.ListDir(dirlist, callback(ListDirectory))) {
			if(sftp.Open("/download/F11_wallpaper_06_1600x1200.jpg", SFtp::READ)) {
				if(sftp.Get(wallpaper)) {
				         Cout() << "File successfully downloaded.\r\n";	
                                 	 return;
                                }
		
			}
                }
	}
	Cout() << "Failed.\r\n" << "Reason: " << sftp.GetErrorDesc() << CRLF;
	
	// Instead we could have used the SFtpGet() convenience function.
	// int rc = SFtpGet(wallpaper, "demo-user", "demo-user", "demo-wftpserver.com", 2222, "/download/F11_wallpaper_06_1600x1200.jpg");  
	
}



If Sftp is sufficient, I'll upload it to the bazaar next weekend. If not, I'l still add it to the bazaar next weekend but implement the missing classes incrementally over time (next class to implemnt will be SCP). Smile

Regards,

Oblivion



[Updated on: Sun, 10 January 2016 13:05]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #45818 is a reply to message #45813] Sun, 10 January 2016 23:02 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 14291
Registered: November 2005
Ultimate Member
Hi,

sounds good.

One thing this check is the license. We generally prefer BSD licensed stuff.

As for what to add, SFTP is great, but adding SSH support would be just awesome.

Do no be afraid about allocators, see e.g. Core/SSL/InitExit.icpp, I bet things will be pretty similar with SFTP.

Mirek
Re: SFTP or full SSH2 support for U++? [message #45821 is a reply to message #45818] Mon, 11 January 2016 09:10 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello Mirek,

Thanks for the tip, I'll look into Core/SSL/InitExit.icpp

Ok then, next week I'll upload the Sftp, and then go for full SSH2 support in general, incrementally adding methods and classes to the package.
(There are and will be some design decisions and trade-offs, though.)

As to the license, I'd considered that before chosing libssh2. It has a revised BSD license (http://www.libssh2.org/license.html)
Hope this is ok. Smile

Regards,
Oblivion


[Updated on: Mon, 11 January 2016 09:13]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #45824 is a reply to message #45821] Mon, 11 January 2016 09:47 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 14291
Registered: November 2005
Ultimate Member
Oblivion wrote on Mon, 11 January 2016 09:10

As to the license, I'd considered that before chosing libssh2. It has a revised BSD license (http://www.libssh2.org/license.html)
Hope this is ok. Smile


Perfect! Smile
Re: SFTP or full SSH2 support for U++? [message #45923 is a reply to message #45813] Tue, 26 January 2016 21:02 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello,

Just to give you all an update:


U++ wrapper for libssh2 is in good shape.

SSHSession class: implemened. I borrowed the U++ memory allocation code (re/alloc, free) from Core/SSL package and after a slight modification glued it to the SSH package. (It is working!)

Sftp class: Implemented all the necessary commands (seek/seek64/statvfs not yet implemented).

I don't have access to Windows right now, so the code is currently tested only on an up-to-date arch linux installation (with a KDE/Plasma5 desktop).

Below is a screenshot of a very simple (and easy to write) sftp downloader example with 10 concurrent downloads, demonstrating non-blocking/async operation capabilities.

index.php?t=getfile&id=4927&private=0

Below is the actual code responsible for 10 concurrent downloads, demonstrating sftp basic async api (Some parameters are hard coded. I was being lazy.)


// Async jobs.

struct Job {
	SFtp 	sftp;
	FileOut file;
	String 	path;
	int 	index;
	int 	cmd;
};

enum Command {	OPEN, READ, CLOSE, FINISH };

const char *sftp_server = "demo.wftpserver.com";
const char *sftp_user   = "demo-user";
const char *sftp_pass   = "demo-user";
const char *remote_file = "/download/F11_wallpaper_06_1600x1200.jpg";

void SFtpExample::Download()
{
    // Initialize and fill an array of Job(s)

    for(int i = 0; i < 10; i++) {
        Job& job = jobs.Add();
        job.sftp.User(sftp_user, sftp_pass);
        job.sftp.StartConnect(sftp_server, 2222);
        job.sftp.WhenDo = THISBACK(UpdateGui);
        job.file.Open(Format("/home/testuser/picture-%d.jpg", i));
        job.cmd     = OPEN;
        job.index   = i;
    }

    // Actual loop: connects to the server and downloads a file in a concurrent way.

    while(jobs.GetCount()) {
        for(int i = 0; i < jobs.GetCount(); i++) {
            Job& job = jobs[i];
            SocketWaitEvent e;
            e.Add(job.sftp.GetSocket());
            e.Wait(10);
            job.sftp.Do();
            if(!job.sftp.InProgress()) {
                if(job.sftp.IsSuccess()) {
                    switch(job.cmd) {
                        case OPEN:
                            job.sftp.StartOpen(remote_file, SSH::READ);
                            job.path = remote_file;
                            job.cmd  = READ;
                            continue;
                        case READ:
                            job.sftp.StartGet(job.file, THISBACK2(DownloadProgress, job.index, job.path));
                            job.cmd  = CLOSE;
                            continue;
                        case CLOSE:
                            job.sftp.StartClose();
                            job.cmd  = FINISH;
                            continue;
                        case FINISH:
                            break;
                    }
                }
                else
                if(job.sftp.IsFailure())
                    list.Set(job.index, 1, DeQtf(job.sftp.GetErrorDesc()));
                jobs.Remove(i);
                list.Remove(i);
                for(int n = 0; n < jobs.GetCount(); n++)
                    jobs[n].index = n;
            }
        }
    }
}	


P.s. I delayed the upload of the package, since it still has some rough edges to iron-out.

Regards,
Oblivion


[Updated on: Tue, 26 January 2016 22:45]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #45952 is a reply to message #45923] Sun, 31 January 2016 09:26 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 14291
Registered: November 2005
Ultimate Member
Oblivion wrote on Tue, 26 January 2016 21:02
Hello,

Just to give you all an update:


U++ wrapper for libssh2 is in good shape.

SSHSession class: implemened. I borrowed the U++ memory allocation code (re/alloc, free) from Core/SSL package and after a slight modification glued it to the SSH package. (It is working!)

Sftp class: Implemented all the necessary commands (seek/seek64/statvfs not yet implemented).

I don't have access to Windows right now, so the code is currently tested only on an up-to-date arch linux installation (with a KDE/Plasma5 desktop).

Below is a screenshot of a very simple (and easy to write) sftp downloader example with 10 concurrent downloads, demonstrating non-blocking/async operation capabilities.

index.php?t=getfile&id=4927&private=0

Below is the actual code responsible for 10 concurrent downloads, demonstrating sftp basic async api (Some parameters are hard coded. I was being lazy.)


// Async jobs.

struct Job {
	SFtp 	sftp;
	FileOut file;
	String 	path;
	int 	index;
	int 	cmd;
};

enum Command {	OPEN, READ, CLOSE, FINISH };

const char *sftp_server = "demo.wftpserver.com";
const char *sftp_user   = "demo-user";
const char *sftp_pass   = "demo-user";
const char *remote_file = "/download/F11_wallpaper_06_1600x1200.jpg";

void SFtpExample::Download()
{
    // Initialize and fill an array of Job(s)

    for(int i = 0; i < 10; i++) {
        Job& job = jobs.Add();
        job.sftp.User(sftp_user, sftp_pass);
        job.sftp.StartConnect(sftp_server, 2222);
        job.sftp.WhenDo = THISBACK(UpdateGui);
        job.file.Open(Format("/home/testuser/picture-%d.jpg", i));
        job.cmd     = OPEN;
        job.index   = i;
    }

    // Actual loop: connects to the server and downloads a file in a concurrent way.

    while(jobs.GetCount()) {
        for(int i = 0; i < jobs.GetCount(); i++) {
            Job& job = jobs[i];
            SocketWaitEvent e;
            e.Add(job.sftp.GetSocket());
            e.Wait(10);
            job.sftp.Do();
            if(!job.sftp.InProgress()) {
                if(job.sftp.IsSuccess()) {
                    switch(job.cmd) {
                        case OPEN:
                            job.sftp.StartOpen(remote_file, SSH::READ);
                            job.path = remote_file;
                            job.cmd  = READ;
                            continue;
                        case READ:
                            job.sftp.StartGet(job.file, THISBACK2(DownloadProgress, job.index, job.path));
                            job.cmd  = CLOSE;
                            continue;
                        case CLOSE:
                            job.sftp.StartClose();
                            job.cmd  = FINISH;
                            continue;
                        case FINISH:
                            break;
                    }
                }
                else
                if(job.sftp.IsFailure())
                    list.Set(job.index, 1, DeQtf(job.sftp.GetErrorDesc()));
                jobs.Remove(i);
                list.Remove(i);
                for(int n = 0; n < jobs.GetCount(); n++)
                    jobs[n].index = n;
            }
        }
    }
}	


P.s. I delayed the upload of the package, since it still has some rough edges to iron-out.

Regards,
Oblivion


Cool. Yesterday I had to fix some ugly PHP application (do not ask...) on one of those hosting platforms where you only have SFTP access.

I was thinking that it would be a cool demonstration (and a very nice PR CodeProject article) to create actual EDITOR of text files over FTP/FTPS/SFTP in "list of files in the left" style. Not sure I will have enough time for that, but it would be fun.

Mirek
Re: SFTP or full SSH2 support for U++? [message #45963 is a reply to message #45952] Mon, 01 February 2016 00:06 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello Mirek,

Quote:

I was thinking that it would be a cool demonstration (and a very nice PR CodeProject article) to create actual EDITOR of text files over FTP/FTPS/SFTP in "list of files in the left" style. Not sure I will have enough time for that, but it would be fun.


Writing an FTP(S) & SFTP editor wouldn't be too difficult. We can use Any container and dynamically create both. Plus, directory enty parsers of the both classes are very similar. It would be easy to achive some abstract API when writing the editor. What worries me is that it'll end up in a somewhat complex code, and that might be bad for PR. Instead, we can write a UI skeleton and build two separate editors, one utilizing FTP(s), and the other SFTP. In this way, people can study the code(s) and the differences. While I am currently polishing SSH/SFtp package, I can start writing a basic FTPS based text editor, in the meantime.

By the way, I was going to propose writing a CodeProject article for TcpSocket (I can write it in the following weeks, bu you may need to review it first Smile ). IMO, it is a vital part of Upp/Core. Yet it is somewhat an enigma. It is well designed but poorly documented. (not talking about its API doc). I had to study HttpRequest/GuiWebCrawler to understand how to do nonblocking operations. Maybe we can clarify that.
Also I would like to propose adding two methods for TcpSocket: My experiences with sockets and remote operations showed me that sending local IP over sockets is not uncommon.
So, maybe it is possible to add a counterpart to GetPeerAddr() method: GetLocalAddr()?
And of course EWOULDBLOCK is technically an error, but in practice we don't always treat it that way.
So I believe adding a public WouldBlock() method to check if the socket would block would be helpful in writing nonblocking code. (GetError() is not very helpful when reading and studying the source code).


Regards,
Oblivion.





[Updated on: Thu, 04 February 2016 08:27]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #46099 is a reply to message #45813] Sun, 06 March 2016 23:39 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello guys,

A major update:

SSH package is re-written using the new version of AsyncQueue class which can utilize ordinary callbacks and lambdas, instead of the cumbersome VarArgs class. SSH package relies heavily on lambda functions, therefore it will require C++11. I believe it is much more elegant now, since it drastically simplifed the task and reduced the source code significantly. It is almost complete, and first release is at hand. (SFtp component, at least).

Below code snippet is taken from the new version of a simple download test app that concurrently downloads a number of jpg files from a well-known public Sftp test server. As you can see, unnecessary state tracking is removed:



struct Job {
        SFtp    sftp;
        FileOut file;
        int     index;
};

Array<Job> jobs;


//........

const char *sftp_server     = "demo.wftpserver.com";
const char *sftp_user       = "demo-user";
const char *sftp_password   = "demo-user";
const char *remote_path     = "/download";

bool SFtpDownloader::ReadDir(Ssh::DirList& ls)
{
    // We'll get the directory listing, using a synchronous (blocking) call.
  
    SFtp browser;
    browser.WhenDo << [&] {ProcessEvents(); };                                                  // GUI should be responsive...
    if(browser.Connect(sftp_server, 2222, sftp_user, sftp_password)) {
 
        // Let us use a convenience method which works on paths, not file descriptors.

        if(browser.ListDir(remote_path, ls))
            return true; 
    }
    Exclamation(browser.GetErrorDesc());
    return false;
}

void SFtpDownloader::Download()
{
    // Wait for the queue to be processed.
 
   if(!jobs.IsEmpty())
        return;
    
    Ssh::DirList ls;
    if(!ReadDir(ls))
        return;
    for(int i = 0, j = 0; i < ls.GetCount(); i++) {

        // We can work on directory entries easily.

        Ssh::DirEntry& e        = ls[i];
        const Ssh::Attrs& attrs = e.GetAttrs();

        // Let us simply try to download all files with *.jpg extension to the current directory, asynchronously.
        // Here, thanks to AsyncQueue, asynchronous calls work in a fire-and-forget fashion. Similar to "batch processing". 
        // E.g., any number of Sftp commands/jobs can be queued to be processed without intervention (till they're finished, or failed, of course). 

        if(attrs.IsFile() && GetFileExt(e.GetName()).IsEqual(".jpg")) {
            Job& job    = jobs.Add();
            job.index   = j;
            job.file.Open(Format("%s/SFtpExample-%d-%s", GetHomeDirectory(), job.index, e.GetName()));
            job.sftp.StartConnect(sftp_server, 2222, sftp_user, sftp_password);
            job.sftp.StartGet(job.file, Format("%s/%s", remote_path, e.GetName()), Ssh::READ, 0755, THISBACK2(DownloadProgress, job.index, e.GetName()));
            j++;
        }
    }
    
    //  Below loop does asynchronous job processing. 

    while(!jobs.IsEmpty()) {
        int i = 0;
        SocketWaitEvent we;
        we.Add(jobs[i].sftp.GetSocket());
        we.Wait(10);
        while(i < jobs.GetCount()) {
            SFtp& sftp = jobs[i].sftp;
            sftp.Do();
            if(!sftp.InProgress()) { 
                // For the sake of simplicity, we do not differentiate between success and failure.
                jobs.Remove(i);
                break;
            }
            ProcessEvents();
            i++;
        }
    }
}

//........



Before I release the first public [beta] version, I'll have to write its docs. It'll probably happen this week.

index.php?t=getfile&id=4957&private=0

Regards,

Oblivion.


[Updated on: Mon, 07 March 2016 08:36]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48061 is a reply to message #45813] Sat, 13 May 2017 21:01 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi Oblivion,

This sounds interesting and might well fit my needs to automate some file transfers. Is it possible to get a hold of a copy of this SFTP client module of yours?

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48072 is a reply to message #48061] Sun, 14 May 2017 16:43 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Quote:
Hi Oblivion,

This sounds interesting and might well fit my needs to automate some file transfers. Is it possible to get a hold of a copy of this SFTP client module of yours?

Best regards,

Tom


Hello Tom,

Since you've revived this topic, let me give some information on recent developments. Smile

Some time ago I decided to go for a full SSH package. As of today, SFtp and Scp modules are complete, and Exec module is to-be-written.
I'm cleaning up the codebase now. A technical preview version is scheduled, and will be available in June 10, 2017. Smile
I'll publish and further develop it for U++ with a BSD license, so of course you'll have the chance to get a copy.
But I'm afraid you'll have to wait a little more.

Regards,

Oblivion






[Updated on: Sun, 14 May 2017 16:45]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48074 is a reply to message #48072] Sun, 14 May 2017 17:46 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi Oblivion,

Excellent! I'll stay tuned for the technical preview on this channel... Smile

If I'm not reacting to your technical preview announcement in a reasonable time, please PM me about the opportunity to test it.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48383 is a reply to message #45813] Wed, 28 June 2017 01:13 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello everyone,

It is now time to publish the technical preview version of SSH package for Upp. Smile

A brief status of the project:

SSH package for U++
--------------------

SSH package is a libssh2 wrapper for Ultimate++.
It aims to be a complete secure shell client solution. (SFtp, Scp, Exec, Terminal, X11Exec, KnownHosts)
Currently it is at a technical preview stage, which means it is a work-in-progress, and subject to change.

Requirements:

- A C++ compiler with at least C++11 support.
- libssh2 ssh library (https://www.libssh2.org/)

Features and Highlights:

- Tight integration with U++.
- Support for both synchronous and asynchronous operations  with both high level and low level methods.
- Allows using U++ style memory [de/re]allocators with libssh2.
- Ssh subsystems support RTTI, and share a common interface for polymorphism. 
  Namely, it is possible to store different subsystems in the same U++ container. 
  (e.g. SFtp, Exec, Scp, Shell can be all stored in the same Array.)

Todo:

- Add known hosts and ssh agents support.
- Add more high level methods.
- Add ssh X11 support.
- Ship libssh2 library with the package.
- Documentation.

History:
-----------------

2017-06-28:  EAGAIN is now properly handled across the subsystems.
             WhenWrite and WhenRead callbacks are removed in favour of parametrized gates.
2017-06-27:  SFtp::StartStop: fixed.
2017-06-27:  CleanUp() methods are added to the classes Ssh, SFtp, and Channel.
             This method utilizes JobQueue's WhenCleanup event.
2017-06-26:  U++ memory [de/re]allocators added and made default.
             initial support for ssh shell, and terminal added.
             Subsystems now perform a clean up on failure to prevent possible heap leaks.          
2017-06-22:  Initial release.


At the moment I am documenting the Package and its api.
There are some important points you need to know before getting yourself familiar with the code.

As I've mentioned in the above summary, SSH package is a libssh2 wrapper. But currently it does not contain libssh2 source code (which also has BSD license).
On linux, your distribution probably provides the lib. On Windows you'll need cmake for express compilation.

Design:

The classes of SSH package are based on JobQueue class, a simple job-queue model (a callback queue based on Upp::Vector actually).
JobQueue is meant to work with (basically) sockets. It provides a uniform interface for async socket operations. In fact, it is an abstraction of Upp's own HttpRequest class' async interface.
From my experience, I can say that it is pretty scalable with a negligable overhead.It allows batch processing (e.g. you can queue the commands and then batch-process them later.)
However if you are going to batch process some commands, do note that the queue will treat the batch as a whole. Hence a failure in one command will halt and clear the queue. This is a security measure to prevent any misbehaviour.

Other advantage of JobQueue is that it allows writing code in a uniform coding pattern. (You can see it in SSH package clearly), in the sense that it exposes both a small set of common functions, and a basic structure.
All SSH sub/classes are based on JobQueue. Therefore they expose a uniform asynchronous api.
Also, each blocking method has its nonblocking counterpart (starting with "Start" prefix.)



Categorization:

SSH package is divided into several classes, based on system-subsystems relationship.
[
Ssh           -> Main session class (system). Provides connection and authentication mechanism. Although known Hosts support not added yet, it will be available in the following versions.

SshSubsystem  -> SshSubsystem is base of all SSH protocol based communication subsystems.
--------------
SFtp          -> Complete, but I'm planning to further enhance it with more high level commannds, once the api stabilized.
Channel       -> Mostly done.
Scp           -> Complete.
Exec          -> Complete.
Shell         -> Not added yet. Besic support is present. And will be developed durther.
Terminal      -> Not added yet. Basic support is present. And will be developed further.
X11Exec       -> Not added yet. Will be added incrementally.


Core class, as one might expect is, Ssh. Smile
It provides connection and authentication mechanisms.

All other Ssh subsystems are derived from a single base class: SshSubsystem.
This class provides common operations and connects subsystems to the Ssh session.

Also it has RTTI interface, which brings us to polymorphism:
SSH package allows a great level of polymorphism while keeping the crucial interface uniform throughout the derivative classes (thanks to JobQueue).
Hence it is possible to put, say, a Scp, Exec, Sftp, Terminal in a single Upp::Array and execute them asynchoronusly through a uniform calling pattern.

All other classes -except SFtp- are derived directly from Channel subsystem. All derivatives expose Channel commands, while has their own commands. So any Channel-based derivative use the power of Channel.

Memory Allocation:

libssh2 provides support for custom allocators. (or alternatively it can use system's memory allocators via USEMALLOC flag)
Basically I copied and adjusted the memory manager found in Core/SSL package.
It works fine with libssh2, but I'm not very happy with it (code duplication is something I don't like) Maybe there is a better way?

How To:

Package currently contains three simple examples:


SshExec: This simple example demonstrates a remote command execution via ssh2 protocol's exec channel, and the blocking api of SSH package.
SFtpDir: This simple example demonstrates a remote diretory listing via sftp channel, and the blocking api of SSH package.
SshAsyncRequests: This simple example demonstrates polymorphism with ssh subsystems, and asynchronous calls, combining a SFtp and Exec instance.

In fact, the first two example are so simple that they look like little code snippets: Smile

SshExec:
#include <Core/Core.h>
#include <SSH/SSH.h>

using namespace Upp;

// SshExec.cpp: 
// Demonstrates remote command execution on an ssh channel.

// A popular public test server:
// -----------------------------
// Host:     test.rebex.net
// User:     user
// Password: password
// Command:  ls -l

CONSOLE_APP_MAIN
{
	Ssh session;
	Cout() << "Hostname: ";
	auto host = ReadStdIn();
	Cout() << "Username: ";
	auto user = ReadStdIn();
	Cout() << "Password: ";
	auto pass = ReadStdIn();
	const int port = 22;

	if(session.Connect(host, port, user, pass)) {
		Cout() << "Command: ";
		auto cmd = ReadStdIn();
		Exec xc(session);
		auto rc = xc(cmd, Cout(), Cerr());
		if(xc.IsFailure()) Cerr() << Format("Exec failed. %s", xc.GetErrorDesc());
		else Cout() << Format("Remote program exited with code: %d\n", rc);
	}
	else Cerr() << session.GetErrorDesc() << '\n';
}


Or, SFtpDir:

#include <Core/Core.h>
#include <SSH/SSH.h>

using namespace Upp;

// SFtpDir.cpp: 
// Demonstrates remote directory listing via an sftp channel.

// A popular public test server:
// -----------------------------
// Host:     test.rebex.net
// User:     user
// Password: password
// Path:     pub/examples

CONSOLE_APP_MAIN
{
	Ssh session;
	Cout() << "Hostname: ";
	auto host = ReadStdIn();
	Cout() << "Username: ";
	auto user = ReadStdIn();
	Cout() << "Password: ";
	auto pass = ReadStdIn();
	Cout() << "Path: ";
	auto path = ReadStdIn();
	const int port = 22;

	if(session.Connect(host, port, user, pass)) {
		SFtp sftp(session);
		SFtp::DirList ls;
		if(!sftp.ListDir(path, ls)) Exit(sftp.GetError());
		for(auto& e : ls)  Cout() << e.GetEntry() << "\n";
	}
	else Cerr() << session.GetErrorDesc() << '\n';
}


That's it Smile






Please keep in mind that while ths package is reasonably stable it is still a work-in-progress. So, USE IT AT YOUR OWN RISK. Smile

Although, given my personal experience I am pretty satisfied by SSH package's api, it is by no means set in stone. It'll change when necessary.
I'll be grateful for any criticism, suggestions, reviews, patches, etc. After all, this package is for U++ and its users.

Package tested on Win7/10 (Mingw64), and latest Arch Linux and Fedora (25).

p.s. There is a leftover that must be changed, which I realized this morning. (In SshSubsystem::To() method) I will change it tonight, but in the meanwhile if you are going to test the package please consider changing the reinterpret_cast to dynamic_cast.

Cheers,
Oblivion.



[Updated on: Wed, 28 June 2017 08:28]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48384 is a reply to message #48383] Wed, 28 June 2017 14:46 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi Oblivion,

This looks good. However, I may need to wait for the following item on your task list to be completed:

- Ship libssh2 library with the package.

Compiling libssh2 became a bit frustrating on my Windows system, so I decided to rather wait for libssh2 becoming part of the delivery. In my view, the libssh2 should be embedded inside your SSH package in a way that the whole package just gets built and statically linked in Windows like any other package without dependencies. Well, I guess the dependency of Core/SSL should be there to make it work.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48385 is a reply to message #48384] Wed, 28 June 2017 15:04 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Quote:
Re: SFTP or full SSH2 support for U++? Wed, 28 June 2017 15:46
Tom1
Hi Oblivion,

This looks good. However, I may need to wait for the following item on your task list to be completed:

- Ship libssh2 library with the package.

Compiling libssh2 became a bit frustrating on my Windows system, so I decided to rather wait for libssh2 becoming part of the delivery. In my view, the libssh2 should be embedded inside your SSH package in a way that the whole package just gets built and statically linked in Windows like any other package without dependencies. Well, I guess the dependency of Core/SSL should be there to make it work.

Best regards,

Tom


Hello Tom,

Of course I am going to include the libssh2 package in the following days (probably next week). Smile
There's too much frustration about libssh2's windows build. And those complaints are usually right.
The problem is that the libssh2 team didn't seem to provide a clear compilation guide.
While I personally prefer OpenSSL, it is not mandatory for libssh2. Windows has CNG since Vista. I can make it an option too.

Fortunately It is actually very easy to compile libssh2 on windows. (which makes you get even more frustrated, finding this after hours of of code-digging).

In the mean time I can write a step by step guide (with screenshots) on how to compile it on Windows (using mingw64, with optional OpenSSL or WinCNG).

Best regards,
Oblivion.


[Updated on: Wed, 28 June 2017 15:10]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48386 is a reply to message #48385] Wed, 28 June 2017 15:29 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi Oblivion,

Thanks, but you do not need to write the libssh2 compilation guide for my sake. I'm not in that much hurry with this and I can certainly wait a little bit longer so that you can merge the libssh2 inside a U++ package.

BTW: I wish this package is then compilable with MSC14 and above, since I cannot use mingw because of some special external library dependencies I have.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48387 is a reply to message #48386] Wed, 28 June 2017 15:50 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
AFAIK Koldo (who had the chance to test my code first hand, and whom I should thank here!) has managed to compile it on VS 2015.
IT needs slight changes (such as that some "."s in lambda functions should be substituted with "->"s), which I'll make this week end.
The thing is, I do not use MSC. I guess I'll install it as a testing environment. Smile

Best regards,
Oblivion


[Updated on: Wed, 28 June 2017 15:51]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48388 is a reply to message #48387] Wed, 28 June 2017 16:04 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
OK. Keep up the good work! Smile

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48405 is a reply to message #45813] Fri, 30 June 2017 23:38 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello,

Technical preview version of the SSH package is updated. Now it contains the libssh2 library. And uses the OpenSSL supplied by U++. Smile

I've tested it with GCC on Linux, and Mingw64 on Windows. It is compiled successfully on both systems.
I have yet to test it with MSC.

Please test it.

If it fails to compile, try configuring the libssh2 library, using libssh2_config.h header.
That header file contains defs. Should you need to modify it, possible values can be copied from the libssh2/libssh2_config.h.in file.


Maybe I should open a seperate topic on SSH package in the Bazaar section of the forums for the time being?

Suggestions, patches, and cricisim is welcome.

[See below post for the updated package]

Best regards,
Oblivion


[Updated on: Sun, 02 July 2017 17:21]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48406 is a reply to message #48405] Sat, 01 July 2017 17:35 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi Oblivion,

Thanks! Here are the quick results:

I tried to compile your packages on Windows 10 Pro 32 bit with MSC14. I just copied all your packages to bazaar (locally). (Running on top of upp rev. 10804.)

When compiling, I got the following errors and warnings for SSH:

----- SSH ( MSC14 MSC WIN32 ) (2 / 6)
Ssh.cpp
C:\upp-10804\bazaar\SSH\Ssh.cpp(28): error C2143: syntax error: missing ';' before '.'
C:\upp-10804\bazaar\SSH\Ssh.cpp(28): error C2059: syntax error: '.'
C:\upp-10804\bazaar\SSH\Ssh.cpp(35): error C2059: syntax error: '.'
C:\upp-10804\bazaar\SSH\Ssh.cpp(46): error C2143: syntax error: missing ';' before '.'
C:\upp-10804\bazaar\SSH\Ssh.cpp(46): error C2059: syntax error: '.'
Channel.cpp
C:\upp-10804\bazaar\SSH\Channel.cpp(134): warning C4244: '=': conversion from 'Upp::int64' to 'int', possible loss of data
SFtp.cpp
C:\upp-10804\bazaar\SSH\SFtp.cpp(269): warning C4244: 'argument': conversion from 'Upp::int64' to 'int', possible loss of data
C:\upp-10804\bazaar\SSH\SFtp.cpp(269): warning C4244: 'argument': conversion from 'Upp::int64' to 'std::size_t', possible loss of data
C:\upp-10804\bazaar\SSH\SFtp.cpp(277): warning C4554: '&': check operator precedence for possible error; use parentheses to clarify precedence
c:\upp-10804\bazaar\ssh\ssh.h(165) : error C4716: 'Upp::SFtp::StartGetStat': must return a value
c:\upp-10804\bazaar\ssh\ssh.h(174) : error C4716: 'Upp::SFtp::StartRealPath': must return a value
c:\upp-10804\bazaar\ssh\sftp.cpp(231) : error C4716: 'Upp::SFtp::StartListDir': must return a value
Exec.cpp
Scp.cpp
libssh2upp.c
c:\upp-10804\bazaar\ssh\libssh2\libssh2_config.h(105): error C2375: 'snprintf': redefinition; different linkage
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1932): note: see declaration of 'snprintf'
c:\upp-10804\bazaar\ssh\libssh2/agent.c(300): warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprintf_s instead. To disable de
    precation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1769): note: see declaration of 'sprintf'
c:\upp-10804\bazaar\ssh\libssh2/channel.c(43): fatal error C1083: Cannot open include file: 'unistd.h': No such file or directory
Malloc.cpp
SSH: 7 file(s) built in (0:09.19), 1313 msecs / file, duration = 9515 msecs, parallelization 100%

There were errors. (1:40.53)


I have no time to look at the causes immediately, but hope these help you a bit. Smile

When I get to office, I will try on a 64-bit Windows platform.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48410 is a reply to message #48406] Sun, 02 July 2017 17:20 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Thank you very much for providing feedback, Tom!

I've installed and configured the MSC to work with TheIDE.
It now compiles on VS 2015. Hopefully the errors should be fixed now.

Tested on:

Windows: MINGW 32/64, MSC (2015)
Linux: GCC

Please test it.

Best regards,

Oblivion.


[Updated on: Sun, 02 July 2017 17:21]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48411 is a reply to message #48410] Sun, 02 July 2017 19:14 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3460
Registered: August 2008
Senior Veteran
Dear Oblivion

Thank you very much. It works again Smile, and with libssh2 embedded!

Just a couple of details (sorry for my perfectionism Razz ):

- Old scp demo does not work
Cout() << "Getting file Demo.mo\n";
FileOut fout(AppendFileName(GetDesktopFolder(), "compile_run.sh"));
const char *path = "compile_run.sh";
Scp scp(ssh); 
scp.WhenRead  = [=](int64 total, int64 done) { return false; };
scp.WhenWrite = Proxy(scp.WhenRead);
if(!scp.Get(fout, path))
	Cout() << scp.GetErrorDesc();
fout.Close();


- SFtp::Init(), Stop(), and Ssh.WhenWait() do not exist.

- Some warnings got by MSC15-64
SFtp.cpp
C:\upp\bazaar\SSH\SFtp.cpp(102): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(136): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(151): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(181): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(237): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(252): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(269): waring C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(269): warning C4244: 'initializing': conversion from 'ssize_t' to 'int', possible loss of data
C:\upp\bazaar\SSH\SFtp.cpp(300): warning C4244: 'initializing': conversion from 'ssize_t' to 'int', possible loss of data


Best regards
Iñaki
Re: SFTP or full SSH2 support for U++? [message #48412 is a reply to message #48410] Sun, 02 July 2017 19:49 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi Oblivion,

I tried SshAsyncRequests example. It compiled nicely on MSC14 32 bit with just a few obvious warnings from libssh2:

----- SSH ( MSC14 MSC WIN32 )
Ssh.cpp
SFtp.cpp
Channel.cpp
Scp.cpp
Exec.cpp
Malloc.cpp
libssh2upp.c
c:\upp-10804\bazaar\ssh\libssh2/agent.c(300): warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprintf_s instead. To disable de
    precation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1769): note: see declaration of 'sprintf'
c:\upp-10804\bazaar\ssh\libssh2/channel.c(1157): warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprintf_s instead. To disable
     deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1769): note: see declaration of 'sprintf'
c:\upp-10804\bazaar\ssh\libssh2/knownhost.c(957): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable de
    precation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(207): note: see declaration of 'fopen'
c:\upp-10804\bazaar\ssh\libssh2/knownhost.c(1181): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable d
    eprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(207): note: see declaration of 'fopen'
c:\upp-10804\bazaar\ssh\libssh2/pem.c(257): warning C4018: '<=': signed/unsigned mismatch
c:\upp-10804\bazaar\ssh\libssh2/userauth.c(541): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable dep
    recation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(207): note: see declaration of 'fopen'
Creating library...
SSH: 7 file(s) built in (0:13.20), 1886 msecs / file, duration = 13547 msecs, parallelization 95%
C:/upp-10804/out/examples-bazaar/SSH/MSC14.\SSH.lib (1624178 B) created in (0:01.96)

OK. (0:13.64)



Then it executed with the following results:

Connecting to Rebex public sftp test server...
Unable to request SFTP subsystem
drwx------ 2 demo users          0 Oct 27  2015 .
drwx------ 2 demo users          0 Oct 27  2015 ..
-rw------- 1 demo users      14280 Mar 19  2007 ConsoleClient.png
-rw------- 1 demo users      15091 Mar 19  2007 ConsoleClientSmall.png
-rw------- 1 demo users      15836 Mar 19  2007 FtpDownloader.png
-rw------- 1 demo users      19156 Feb 16  2007 imap-console-client.png
-rw------- 1 demo users      36672 Mar 19  2007 KeyGenerator.png
-rw------- 1 demo users      24029 Mar 19  2007 KeyGeneratorSmall.png
-rw------- 1 demo users      16471 Feb 16  2007 mail-editor.png
-rw------- 1 demo users      35414 Feb 16  2007 mail-send-winforms.png
-rw------- 1 demo users      49011 Feb 16  2007 mime-explorer.png
-rw------- 1 demo users      58024 Mar 19  2007 pocketftp.png
-rw------- 1 demo users      20197 Mar 19  2007 pocketftpSmall.png
-rw------- 1 demo users      20472 Feb 16  2007 pop3-browser.png
-rw------- 1 demo users      11205 Feb 16  2007 pop3-console-client.png
-rw------- 1 demo users        407 Mar 23  2007 readme.txt
-rw------- 1 demo users      11546 Mar 19  2007 ResumableTransfer.png
-rw------- 1 demo users       2635 Mar 19  2007 winceclient.png
-rw------- 1 demo users       6146 Mar 19  2007 winceclientSmall.png
-rw------- 1 demo users      80000 Mar 19  2007 WinFormClient.png
-rw------- 1 demo users      17911 Mar 19  2007 WinFormClientSmall.png
Remote command processed. Return code is 0.
<--- Finished in (0:01.79), exitcode: 0 --->


I wonder if "Unable to request SFTP subsystem" was to be expected here.

Anyway, I will do more testing tomorrow. Smile

Thanks and best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48413 is a reply to message #48411] Sun, 02 July 2017 21:14 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello Koldo and Tom,

Thank you both for testing the code, an dproviding me the crucial feedback!


Quote:
- SFtp::Init(), Stop(), and Ssh.WhenWait() do not exist.


Yes, they are removed in favour of a better approach.
I believe initialization and deinitializaton should be made automatic, analogous to RAII.
If you request a resource, say a Scp channel, which should be allocated per request anyway, you simply spawn it and bind it to an ssh session.
It will be automatically deallocated when the object is destroyed, as is with any C++ object:

// 
Scp scp1;
scp1.Session(session);                     // Scp channel is "scheduled" to be initialized. There is no need for manual deinitialization. It will be deinitialized at the end of the code block.
//Or
Scp scp2(session);                         // Scp channel is "scheduled" to be initialized. There is no need for manual deinitialization. It will be deinitialized at the end of the code block.


In the same vein, you don't actually need to explicitly disconnect from the ssh server. (Of course you can, but it is by no means necessary).
Ssh objects automatically attempt to disconnect when they are being destroyed.


WhenWait was a member of JobQueue. But there is no need for WhenWait anymore.
As you can see in the SshAsyncRequests example that I provided with the package, you can use SocketWaitEvent where it is useful.
If you want to include it in a synchronous call, you can override JobQueue::Execute(). It is virtual. Smile

As for the old Scp Demo.
Getters and Putters are now in U++ style.
 bool Get(Stream& out, const String& path, Gate<int64, int64> progress = Null); 
 bool Put(Stream& in, const String& path, long mode, Gate<int64, int64> progress = Null);      


So the code needs a slight change:

Cout() << "Getting file Demo.mo\n";
FileOut fout(AppendFileName(GetDesktopFolder(), "compile_run.sh"));
const char *path = "compile_run.sh";
Scp scp(ssh); 
if(!scp.Get(fout, path, [=](int64 total, int64 done) { return false; })) // <-- 
    Cout() << scp.GetErrorDesc();
fout.Close();


Regarding the warnings, I'll try to fix them. Main problem is that Stream return int64 for file size, etc. but libssh2 functions expect size_t.


Hello Tom,

Those warnings are typical and can be suppressed. Usually they are not harmful. But I'll see what I can do.

Quote:
I wonder if "Unable to request SFTP subsystem" was to be expected here.


Yes, this is expected. It can happen time to time. But it should only be a rare case. If not, then there might be a problem with initialization.
I'll look into it.

There are two things to note here though:
1) Remote servers are not always reliable. Rebex is fine but response times of it's public test server are somewhat slow.

2) There is a "little" problem with ssh protocol and it's implementations: Ssh session works over a single socket.
Each ssh subsystem, be it sftp or scp, are actually channels on a single socket. And AFAIK there is no official way to "wait" on channels.
So, spawning more than one channel, especially when they are of different subsystems, can result in a such error.
It is also discussed in the libssh2 documents, There is a suggested solution for this by the libssh2 devs, but it is not implemented yet:

libssh2/TODO, lines 80 to 120 reads:
New Transport API
=================

THE PROBLEM

The problem in a nutshell is that when an application opens up multiple
channels over a single session, those are all using the same socket. If the
application is then using select() to wait for traffic (like any sensible app
does) and wants to act on the data when select() tells there is something to
for example read, what does an application do?

With our current API, you have to loop over all the channels and read from
them to see if they have data. This effectively makes blocking reads
impossible. If the app has many channels in a setup like this, it even becomes
slow. (The original API had the libssh2_poll_channel_read() and libssh2_poll()
to somewhat overcome this hurdle, but they too have pretty much the same
problems plus a few others.)

Traffic in the other direction is similarly limited: the app has to try
sending to all channels, even though some of them may very well not accept any
data at that point.

A SOLUTION

I suggest we introduce two new helper functions:

 libssh2_transport_read()

 - Read "a bunch" of data from the given socket and returns information to the
   app about what channels that are now readable (ie they will not block when
   read from). The function can be called over and over and it will repeatedly
   return info about what channels that are readable at that moment.

 libssh2_transport_write()

 - Returns information about what channels that are writable, in the sense
   that they have windows set from the remote side that allows data to get
   sent. Writing to one of those channels will not block. Of course, the
   underlying socket may only accept a certain amount of data, so at the first
   short return, nothing more should be attempted to get sent until select()
   (or equivalent) has been used on the master socket again.

I haven't yet figured out a sensible API for how these functions should return
that info, but if we agree on the general principles I guess we can work that
out.


For the time being, best workaround for this problem seems to be increasing the socket "wait" time and adopting a "max. retries" approach. Smile

I am in the process of writing the api docs, and a guide for SSH protocol and package, I'll discuss these problems there too.

Best regards,

Oblivion



[Updated on: Mon, 03 July 2017 08:15]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48414 is a reply to message #45813] Mon, 03 July 2017 07:55 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
There is a problem that needs to be solved though. I've encountered this error recently.

Connections to servers on localhost (127.0.0.1) fail for some unknown (yet) reason. It needs further investigation.
As far as I can see, this failure doesn't happen when linking against externally (using Cmake) compiled libssh2.
Probably I am missing some configuration switch (or a combination of switches?).
libssh2 is a strange beast with a configure script which has over 20K LoC. It'll take some time (a week or so) to figure out what I'm missing.
But of course in the meantime I'm open to suggestions, and I accept patches, etc.

Best regards,
Oblivion.


[Updated on: Mon, 03 July 2017 08:17]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48415 is a reply to message #48414] Mon, 03 July 2017 08:39 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3460
Registered: August 2008
Senior Veteran
Thank you Oblivion. New scp interface works well Smile

Best regards
Iñaki
Re: SFTP or full SSH2 support for U++? [message #48416 is a reply to message #48414] Mon, 03 July 2017 10:33 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi Oblivion,

I have successfully tested your examples on Windows 10 Professional 64-bit platform using MSC14, MSC14x64, MSC15 and MSC15x64 build methods.

I guess the usual warnings on the deprecation of functions used by libssh2 could probably be suppressed by using a compiler flag associated to the SSH package. This is just a cosmetic issue though.

Thank you very much for your effort on this task.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48435 is a reply to message #48416] Mon, 03 July 2017 21:51 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello Koldo and Tom,

It is nice to hear that the package is working for you. Smile
I have found the source of the problem that I mentioned in my last message (regarding the error where the ssh connection to servers on localhost was failing.)
As it turns out, my suspicion was right. A preprocessor directive was missing: LIBSSH2_DH_GEX_NEW
This preprocessor directive enables the newer diffie-hellman-group-exchange-sha1 syntax, which libssh2 relies on when compiled against recent versions of openssl.

Also I fixed a critical error where the JobQueue can stuck in an infinite loop when the global timeout is set. Therefore please don't forget to update JobQueue too.

You can find the updated package below.

Now that the most annoying problem is solved, I can focus on completing the package (by adding knownhosts, X11 and terminal support, fixing warnings, etc.).

Once the package is complete I am going to open a bazaar topic for SSH package.
And if at that point Mirek and other U++ developers decide that SSH package can be official part of Upp, it has a new BSD license, and permission is hereby explicitly granted.
(I'll gladly continue to maintain the package, since I'll be using it.)

For the time being I'll provide a SVN link to the package, where you can always get the latest version:

https://sourceforge.net/p/ultimatecomponents/svn/HEAD/tree/t runk/



Best regards,
Oblivion


[Updated on: Mon, 03 July 2017 22:36]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48437 is a reply to message #48435] Tue, 04 July 2017 08:57 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi Oblivion,

For some reason the latest SSH.h had the include files for libssh2 changed from:

#include "libssh2/libssh2.h"
#include "libssh2/libssh2_sftp.h"
#include "libssh2/libssh2_publickey.h"


To:

#include "libssh2.h"
#include "libssh2_sftp.h"
#include "libssh2_publickey.h"


This effectively prevented compilation. I changed it back and now it compiles and runs again OK.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48438 is a reply to message #48437] Tue, 04 July 2017 09:02 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Ah, while I was inspecting the cause for the connection error I changed the headers to test some cases.
I was in a hurry, sorry for the inconvenience. Embarassed
I'll fix them tonight. Thanks!

Best regards,
Oblivion


[Updated on: Tue, 04 July 2017 09:03]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48439 is a reply to message #48438] Tue, 04 July 2017 09:12 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
No worries, it was just a minor detail.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48650 is a reply to message #45813] Sat, 12 August 2017 22:49 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello everyone,

Next update to SSH package technical preview is finally here.

Aside from bug fixes and cleanups, SSH package has now gained full support for three authentication methods (password, publickey, keyboard-interactive) properly.
The crucial part was keyboard interaction (challenge/response) authentication method. It is implemented using a Function.
Also, it is now possible to select between preferred transport methods.
Last but not least, API reference docs are added for the most crucial classes. Smile
I am also writing a guide for its usage. But it's not finished yet.

Agents, X11 support, and forwarding support are on their way way too!

Now I'd like to ask you a question. This covers both SSH package and my FTP package.
I'm planning to add to both package network proxy (HTTP/SOCKS4/4a/5) support using the new version of my NetworkProxy package (it will be renamed to NetProxy).

I can think of three strategies:

1) Adding proxy support as an integral part to SSH and FTP packages. This will cause SSH and FTP packages to require NetProxy and I don't really prefer this.
2) Using a compiler flag for enabling proxy support.
3) Since NetworkProxy (and upcoming NetProxy) is a delegate class that simply takes an already initialized TcpSocket, connects it, and hands it over back to user, I can add a callback, say WhenProxy, to Ssh and Ftp classes, and hand over the socket to NetProxy there, and when the connection is established we can proceed. This way both packages won't require NetworkProxy at all, it will be completely optional.

What do you think? Which strategy would be better?

I appreciate suggections, bug reports, criticism, etc.


Here's a digest:
2017-08-11:  Initial api reference docs for Ssh, Ssh::SubSystem, SFtp, Scp, Exec, Knownhosts, are added.
2017-07-31:  It is now possible to query, get, and set the possible transportation methods and exchange 
             algorithms. Added Ssh::Method(), Ssh::Methods(), Ssh::GetMethod(), Ssh::GetMethods() methods. 
             Ssh::Host class is from now on KnownHosts class.
2017-07-21:  Authentication methods (password, public key, keyboard interaction) are properly implemented.
             From now on it is possible to choose between authentication methods both on initialization,
             and on-the-fly (i.e. while logging in, using WhenAuth callback). 
             KnownHosts class is added to the package. This class provides basic known hosts support.
             It is now possible to verify and trust servers.


Best regards,
Oblivion


[Updated on: Sat, 12 August 2017 23:03]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48659 is a reply to message #48650] Mon, 14 August 2017 12:41 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi Oblivion,

Thanks for the update. Smile

Not sure if this has anything to do with the update, but I have trouble concatenating the following procedure to conditionally create a directory:
...
	if(session.IsSuccess()){
		SFtp::DirList list;
		SFtp sftp(session);
			
		sftp.Clear();
		sftp.StartListDir(directoryname,list);
		sftp.StartClose();
		if(sftp.Execute()) return true; // Reading the directory successfully proves it exists

		sftp.Clear();
		sftp.StartMakeDir(directoryname,0755);
		sftp.StartClose();
		return sftp.Execute();
	}
...


The MakeDir part fails when done after ListDir. If I establish a separate SFtp instance for both ListDir and MakeDir, it works OK. Is it supposed to work that way?

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48662 is a reply to message #48659] Mon, 14 August 2017 14:26 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello Tom,

tl;dr: I'll write a better reply and explanation, once I get back home. Also I'm writing a guide on it's usage. Smile

I can see what you are trying to achieve, but that's kinda' against the whole concept.
I mean, in the snippet you provided, you are emulating synchronous calls with asynchronous methods, which is basically legitimate, but already done in the SSH packages synchronous calls Smile :

Synchrounous version may be (I don't know the code-flow):

if(session.IsSuccess()){
		SFtp::DirList list;
		SFtp sftp(session);
		if(!sftp.ListDir((directoryname, list))
                   return sftp.MakeDir(directoryname, 0755);
               
}


Asynchronous version (which really shouldn't be written this way. Just do not use this. Only to give you the idea.) :
if(session.IsSuccess()){
		SFtp::DirList list;
		SFtp sftp(session);
		
                sftp.StartListDir(directoryname, list);
                while(sftp.Do()); 
                if(sftp.IsFailure()) {
                    sftp.StartMakeDir(directoryname, 0755);
                    while(sftp.Do());
                    return sftp.IsSucces();
                }
}


You dont' need to call Clear() at all. Queue will be cleared automatically.
It is supplied as a safety measure (which I'll cover it in the guide) for creating (custom) synchronous methods by the user.

By the way, I am going to add high-level methods, such as IsFileExists(), IsDirectoryExists(), etc. Smile

Thanks for the feedback!

Best regards,
Oblivion


[Updated on: Mon, 14 August 2017 14:46]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48663 is a reply to message #48662] Mon, 14 August 2017 14:46 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi,

Thanks for your prompt reply. (Although I'm not in any hurry with this, whatsoever.)

Well, I tried the async approach just to test, but I had the exact same problem with the synchronous one too .. i.e. This fails:
if(session.IsSuccess()){
		SFtp::DirList list;
		SFtp sftp(session);
		if(!sftp.ListDir(directoryname, list))
                   return sftp.MakeDir(directoryname, 0755);
               
}



But this works:

if(session.IsSuccess()){
		SFtp::DirList list;
		SFtp sftp(session);
		if(!sftp.ListDir(directoryname, list)){
			SFtp sftp2(session);
			return sftp2.MakeDir(directoryname, 0755);
		}
               
}


The high level helpers, like you pictured in your message, would be great. Please check the naming of U++ counterparts to keep the access easy. (BTW: I think that I have seen IsOK() quite frequently in U++ instead of IsSuccess(), but I'm not sure if these are supposed to mean exactly the same thing.)

Thanks and best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48664 is a reply to message #48663] Mon, 14 August 2017 14:52 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello Tom,

Quote:
(BTW: I think that I have seen IsOK() quite frequently in U++ instead of IsSuccess(), but I'm not sure if these are supposed to mean exactly the same thing.)


Queue model is an abstraction of Upp::HttpRequest's async model. I took Do(), IsSuccess(), IsFailure(), InProgress() from there. I think they are legitimate.


And I see the source of the problem now, I'll fix it asap.
Thanks!


Best regards,
Oblivion


[Updated on: Mon, 14 August 2017 14:58]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48665 is a reply to message #48664] Mon, 14 August 2017 14:57 Go to previous messageGo to next message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi,

I never used HttpRequest, so I'm not familiar with this.

But no worries; I trust you know best what you're up to! Smile

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48862 is a reply to message #45813] Sun, 15 October 2017 23:10 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello Mirek, Tom, Koldo, and everyone,

It's been a while since I last updated the package.

SSH package's been through a major rewrite, and I wanted to have a solid foundation to build it upon. Well, I believe it is finally here.

New design hopefullt fixes the main problems of the previous one. To do this I focused on the major component of SSH package: SFtp, since it was the most complex part of it.

I am here to share the results. Below package contains only Ssh and SFtp components for the time being. But please take your time to examine and test SFtp. It can do some

nice tricks easily, such as peeking and poking at files, or -experimental- multithreading support (we'll return to that later.).

I also included a simple testcase where you can see how SFtp class behaves. Note that, every other component (be it SCP, channel, exec) will behave in a similar way. Smile

Here is the highlights, and changes:

         1) This version brings a major redesign of overall components. SSH package's classes
             are redesigned around a core class named SshCore. SshCore allows a uniform interface
             for both blocking and non-blocking modes of operation. Visible result of this new 
             design is a single set of methods for both modes of operation for each class. Methods
             starting with "Start" prefix are removed in favor of a simple "NonBlocking()" switch.

             2) In accordance with this behaviour, a sort of "feature parity" is successfully 
                kept, thanks to the new design. Namely, in non-blocking mode, any result can be
                gathered using the new GetResult() method, which returns Value.

	     3) All SSH package classes now has pick semantics.
            
	     4) New design allows creation of higher-level ("complex") SFTP methods where needed.
                Unlike the simple SFTP methods, which work on file handles, Complex methods take 
                care of (allocate/free) file handles internally.
            
	     5) Experimental multithreading support is added (using AsyncWork). This is still
                at a primitive stage. 
	
	     6) Network proxy support is added. This new feature uses NetProxy (Http/Socks4/4a/5)
		package It works through a plugin, provided via Ssh::WhenProxy callback. And it
                is completely optional.

             7) Cancellation mechanism is added. Any operation can be cancelled at any time using
                the Cancel() method.

             8) SFtp::DirEntry class now has a ToString() method. (It will also gain an ToXml()
                method.) This method will give an output similar to Unix ls command.

             9) libssh2 configured to use OpenSSL by default (WinCNG will be made a compile-time
                option in the next release).


The code is not entirely cleaned-up, so there maybe also small cosmetic issues (for now.)

I appreciate feedbacks, patches, criticism, etc.

[ See below posts for updated package]

Cheers!

Oblivion


[Updated on: Wed, 18 October 2017 23:57]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48863 is a reply to message #48862] Mon, 16 October 2017 08:58 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3460
Registered: August 2008
Senior Veteran
Thank you Smile

Best regards
Iñaki
Re: SSH2 wrapper for U++ [message #48875 is a reply to message #45813] Thu, 19 October 2017 00:55 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1267
Registered: August 2007
Senior Contributor
Hello,

SSH package is updated.

2017-10-17:  Class names finalized:
             Core/base class  -> Ssh
             Session class    -> SshSession
             SFtp class       -> SFtp
             Channel class    -> SshChannel
             Scp class        -> Scp
             Exec class       -> SshExec
             Knownhosts class -> SshHosts
             SshAgents        -> (will be added as such)

             CreateSFtp(), CreateExec(), CreateScp(), CreateChannel() methods added to SshSession
             class. 
             Logging further refined.
             Error management further refined.


New version of crucial SSH components are re-added to the packaet.

As you can see in the above history entry, component names are being finalized. I am satisfied with these new namings, and would like to hear your commennts.

Also I added CreateSFtp(), CreateExec(), CreateScp(), CreateChannel() methods to SshSession class.

Since now we have pick semantics, these methods are IMO more elegant than explicity constructing subsystem objecs.

A simple use case would be as follows (below code is a part of SFtpDemo, by the way. (since Rebex test server is actually a SFtp server, with exec support, I did not change the example name):

void GetFileInfoViaExec(SshSession& session)
{
	DLOG("---- Getting directory listing of " << file << " using SshExec (blocking):");
	auto exec = session.CreateExec();
	DDUMP(exec(String(cmd) + file, Cout(), Cerr()));
	
}



As I announced in my previous message, SSH package recently gained a completely optional NetProxy support.
(see Bazaar section https:// www.ultimatepp.org/forums/index.php?t=msg&th=10132&g oto=48812&#msg_48812 for NetProxy package)
Here is the minimal example code (this is a real test code that works):
CONSOLE_APP_MAIN
{
    StdLogSetup(LOG_FILE | LOG_COUT | LOG_TIMESTAMP);

//  Ssh::Trace();
    SshSession session;
    session.WhenProxy = [=](TcpSocket& sock) {
        NetProxy proxy(sock, proxy_host, proxy_port);
        return proxy.Timeout(10000)
                    .Socks5()
                    .Auth(proxy_username, proxy_password)
                    .Connect(ssh_host, ssh_port);
    };
    if(session.Connect(ssh_host, ssh_port, username, password)) {
        auto exec = session.CreateExec();
        DDUMP(exec("ls -l /readme.txt", Cout(), Cerr());
    }
    else Cerr() << session.GetErrorDesc() << "\n";
}



RTTI support is also re-implemented.

Soon I'll move SSH package out of technical preview phase to beta. But before that I have to add documentation, a proper multithreading support, and missing classes (SshHost, SshAgents).

On multithreading:

The design of SSH core (base) class "ssh" allowed me to easily add an "internal" MT support using a compile-time switch (I did not yet publish that code).
But I haven't really decided to add MT that way. Using convenience functions (SFtpGet, SFtpPut, ScpGet, ScpPut, AsyncExec) seem a simpler and better managable way to go.
But I'd like to hear yout thoughts on that matter too.

I apprecieate reviews, bug reports, patches, suggestions, criticism.

Best regards,
Oblivion.





[Updated on: Thu, 19 October 2017 10:13]

Report message to a moderator

Re: SSH2 wrapper for U++ [message #48879 is a reply to message #48875] Thu, 19 October 2017 14:16 Go to previous messageGo to previous message
Tom1
Messages: 1319
Registered: March 2007
Ultimate Contributor
Hi Oblivion,

I downloaded this latest version now after using the version from a few months back.

In order to successfully compile the code, I had to comment out the entire contents of SFtpMT.cpp and the following lines in SFtp.h:

AsyncWork<String>   SFtpGet(SshSession& session, const String& path, Gate<int64, int64> progress = Null);
AsyncWork<void>     SFtpGet(SshSession& session, const String& source, const String& target, Gate<int64, int64> progress = Null);


Additionally, I had to comment out the following in Core.cpp:

	DLOG("Initializing libssh2...");
and
	DLOG("Deinitializing libssh2...");


(There was some 0x40 character embedded in those lines that MSC did not like.)

-

Then there's one question: Now that the names have changed, is there a replacement for session.IsSuccess() ? (I just removed these calls in my code to see if SFTP still works. It did Smile )

Thanks again for your work on SSH!

Best regards,

Tom

[Updated on: Thu, 19 October 2017 14:17]

Report message to a moderator

Previous Topic: Help needed with link errors (serversocket)
Next Topic: Ultimate++ repo and chat?
Goto Forum:
  


Current Time: Thu Jun 11 07:51:18 GMT+2 2026

Total time taken to generate the page: 0.01902 seconds