Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Developing U++ » U++ Developers corner » SFTP or full SSH2 support for U++? (Discussion about implementing the SSH2 protocol using libssh2)
SFTP or full SSH2 support for U++? [message #45813] Sun, 10 January 2016 12:58 Go to next message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
Hello,

Fırst, I wish you all a happy new year!


Some time ago when I uploaded my FTPS implementation to the bazaar section, Daniel (Unodgs) asked me If I have any plans to implement an SFTP class.
And I'd said yes.

Now I am here to deliver my promise. Smile


But before I upload the package to the bazaar section, I have to make a choice and would like to hear your opinion on the subject.


First of all, I decided not to implement the SSH2 protocol from the scratch. I could've, but it would be a) tedious and b) wasting my time.
So I decided searching for alternatives.
The best alternative that I could have come up with was wrapping an existing, multiplatform (Linux and Windows, at least) library.
I found out that libssh2 was the way to go. It was plain C, easy to wrap up and had both blocking and non-blocking modes of operation available.

I finally implemented an SFTP class for U++ using libssh2, which has the below features:


1) Allows blocking and non-blocking operations. Namely, it can work in both synchronous and asynchronous modes. 
   I use HttpRequest and NetworkProxy class' asynchronous design here too, since that design proved very effective.

2) Has OPTİONAL network proxy (HTTP_CONNECT, SOCKS4/4a/5) support, using NetworkProxy class.

3) Takes advantage of U++ streams in file upload and download operations, and uses gates for progress tracking when needed.

4) Supports both MD5 and SHA 1 methods.

5) Has SFtp::DirEntry class for easy parsing of directory entries.




In short, currently we have a sync/async SFTP wrapper that can authenticate using "public key" method and/or Username/password combination, and perform Open/Close/Read/Write/List/Rename/Delete operations on remote file system entries. Also its API is very similar to my FTPS class, and its design derives from NetworkProxy and HttpRequest classes and follows the U++ coding style.

Now, the class is abstracted nicely so that I can either let it be a standaloune, robust SFtp class, or take it one step further and wrap up the whole functionality of libssh2.
Latter means adding SSHSession, SSHChannel, Scp classes to the package too.
What is your opinion?

(For example I can also easily add the SCP protocol to the package. SCP is known to perform better in some type file read/wtire operations.)


Also, since the libssh2 uses its own allocators (alloc, realloc, dealloc) but also gives the user the choice to implement his/her own, I'll definietly need help here.
I'm not experienced with U++ allocators.

Here is an actual, barebone SFtp example, demonstrating directory listing and file download, using a public SFtp test server:


#include <Core/Core.h>
#include <SFTP/SFTP.h>

using namespace Upp;

static const char* CRLF = "\r\n";

// This callback allows getting directory entries one by one.
// It is optional.
bool ListDirectory(SFtp::DirEntry& e) 
{

	
	String ls = e.GetEntry();
	if(!ls.IsEmpty())
		// When available, traditiona UNIX style directory listing can be used.
		Cout() << e.GetEntry() << CRLF;
	else {
		Cout() 	<< e.GetName()			        << CRLF
				<< e.GetUserID()		<< CRLF
				<< e.GetGroupID()    	        << CRLF
				<< e.GetSize() 			<< CRLF
				<< e.GetLastAccessed()	        << CRLF
				<< e.GetLastModified()	        << CRLF
				<< "Permissions:"		<< CRLF
				<< "R: " << e.IsReadable() 	<< ", "
				<< "W: " << e.IsWritable() 	<< ", "
				<< "X: " << e.IsExecutable()    << CRLF;
	}
	return false;
}

CONSOLE_APP_MAIN
{
	// This example demonstrates directory listing (ls) and file download operations,
	// using a public SFTP test server.
	// Note that this example uses blocking mode. It is synchronous. 
	// There is also a non-blocking, asynchhronous mode.
	
	FileOut wallpaper("/home/genericuser/Wallpapers/SFTP-download-test-wallpaper.jpg");
	SFtp::DirList dirlist; // SFtp::DirList is a vector containing SFtp:DirEnty elements.

	// Enable logging.
	SFtp::Trace();

	SFtp sftp;
	if(sftp.User("demo-user", "demo-user").Connect("demo.wftpserver.com", 2222)) {
		// Get server banner.
		Cout() << sftp.GetBanner() << CRLF;
		if(sftp.OpenDir("/download") && sftp.ListDir(dirlist, callback(ListDirectory))) {
			if(sftp.Open("/download/F11_wallpaper_06_1600x1200.jpg", SFtp::READ)) {
				if(sftp.Get(wallpaper)) {
				         Cout() << "File successfully downloaded.\r\n";	
                                 	 return;
                                }
		
			}
                }
	}
	Cout() << "Failed.\r\n" << "Reason: " << sftp.GetErrorDesc() << CRLF;
	
	// Instead we could have used the SFtpGet() convenience function.
	// int rc = SFtpGet(wallpaper, "demo-user", "demo-user", "demo-wftpserver.com", 2222, "/download/F11_wallpaper_06_1600x1200.jpg");  
	
}



If Sftp is sufficient, I'll upload it to the bazaar next weekend. If not, I'l still add it to the bazaar next weekend but implement the missing classes incrementally over time (next class to implemnt will be SCP). Smile

Regards,

Oblivion



[Updated on: Sun, 10 January 2016 13:05]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #45818 is a reply to message #45813] Sun, 10 January 2016 23:02 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12527
Registered: November 2005
Ultimate Member
Hi,

sounds good.

One thing this check is the license. We generally prefer BSD licensed stuff.

As for what to add, SFTP is great, but adding SSH support would be just awesome.

Do no be afraid about allocators, see e.g. Core/SSL/InitExit.icpp, I bet things will be pretty similar with SFTP.

Mirek
Re: SFTP or full SSH2 support for U++? [message #45821 is a reply to message #45818] Mon, 11 January 2016 09:10 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
Hello Mirek,

Thanks for the tip, I'll look into Core/SSL/InitExit.icpp

Ok then, next week I'll upload the Sftp, and then go for full SSH2 support in general, incrementally adding methods and classes to the package.
(There are and will be some design decisions and trade-offs, though.)

As to the license, I'd considered that before chosing libssh2. It has a revised BSD license (http://www.libssh2.org/license.html)
Hope this is ok. Smile

Regards,
Oblivion


[Updated on: Mon, 11 January 2016 09:13]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #45824 is a reply to message #45821] Mon, 11 January 2016 09:47 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12527
Registered: November 2005
Ultimate Member
Oblivion wrote on Mon, 11 January 2016 09:10

As to the license, I'd considered that before chosing libssh2. It has a revised BSD license (http://www.libssh2.org/license.html)
Hope this is ok. Smile


Perfect! Smile
Re: SFTP or full SSH2 support for U++? [message #45923 is a reply to message #45813] Tue, 26 January 2016 21:02 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
Hello,

Just to give you all an update:


U++ wrapper for libssh2 is in good shape.

SSHSession class: implemened. I borrowed the U++ memory allocation code (re/alloc, free) from Core/SSL package and after a slight modification glued it to the SSH package. (It is working!)

Sftp class: Implemented all the necessary commands (seek/seek64/statvfs not yet implemented).

I don't have access to Windows right now, so the code is currently tested only on an up-to-date arch linux installation (with a KDE/Plasma5 desktop).

Below is a screenshot of a very simple (and easy to write) sftp downloader example with 10 concurrent downloads, demonstrating non-blocking/async operation capabilities.

index.php?t=getfile&id=4927&private=0

Below is the actual code responsible for 10 concurrent downloads, demonstrating sftp basic async api (Some parameters are hard coded. I was being lazy.)


// Async jobs.

struct Job {
	SFtp 	sftp;
	FileOut file;
	String 	path;
	int 	index;
	int 	cmd;
};

enum Command {	OPEN, READ, CLOSE, FINISH };

const char *sftp_server = "demo.wftpserver.com";
const char *sftp_user   = "demo-user";
const char *sftp_pass   = "demo-user";
const char *remote_file = "/download/F11_wallpaper_06_1600x1200.jpg";

void SFtpExample::Download()
{
    // Initialize and fill an array of Job(s)

    for(int i = 0; i < 10; i++) {
        Job& job = jobs.Add();
        job.sftp.User(sftp_user, sftp_pass);
        job.sftp.StartConnect(sftp_server, 2222);
        job.sftp.WhenDo = THISBACK(UpdateGui);
        job.file.Open(Format("/home/testuser/picture-%d.jpg", i));
        job.cmd     = OPEN;
        job.index   = i;
    }

    // Actual loop: connects to the server and downloads a file in a concurrent way.

    while(jobs.GetCount()) {
        for(int i = 0; i < jobs.GetCount(); i++) {
            Job& job = jobs[i];
            SocketWaitEvent e;
            e.Add(job.sftp.GetSocket());
            e.Wait(10);
            job.sftp.Do();
            if(!job.sftp.InProgress()) {
                if(job.sftp.IsSuccess()) {
                    switch(job.cmd) {
                        case OPEN:
                            job.sftp.StartOpen(remote_file, SSH::READ);
                            job.path = remote_file;
                            job.cmd  = READ;
                            continue;
                        case READ:
                            job.sftp.StartGet(job.file, THISBACK2(DownloadProgress, job.index, job.path));
                            job.cmd  = CLOSE;
                            continue;
                        case CLOSE:
                            job.sftp.StartClose();
                            job.cmd  = FINISH;
                            continue;
                        case FINISH:
                            break;
                    }
                }
                else
                if(job.sftp.IsFailure())
                    list.Set(job.index, 1, DeQtf(job.sftp.GetErrorDesc()));
                jobs.Remove(i);
                list.Remove(i);
                for(int n = 0; n < jobs.GetCount(); n++)
                    jobs[n].index = n;
            }
        }
    }
}	


P.s. I delayed the upload of the package, since it still has some rough edges to iron-out.

Regards,
Oblivion


[Updated on: Tue, 26 January 2016 22:45]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #45952 is a reply to message #45923] Sun, 31 January 2016 09:26 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12527
Registered: November 2005
Ultimate Member
Oblivion wrote on Tue, 26 January 2016 21:02
Hello,

Just to give you all an update:


U++ wrapper for libssh2 is in good shape.

SSHSession class: implemened. I borrowed the U++ memory allocation code (re/alloc, free) from Core/SSL package and after a slight modification glued it to the SSH package. (It is working!)

Sftp class: Implemented all the necessary commands (seek/seek64/statvfs not yet implemented).

I don't have access to Windows right now, so the code is currently tested only on an up-to-date arch linux installation (with a KDE/Plasma5 desktop).

Below is a screenshot of a very simple (and easy to write) sftp downloader example with 10 concurrent downloads, demonstrating non-blocking/async operation capabilities.

index.php?t=getfile&id=4927&private=0

Below is the actual code responsible for 10 concurrent downloads, demonstrating sftp basic async api (Some parameters are hard coded. I was being lazy.)


// Async jobs.

struct Job {
	SFtp 	sftp;
	FileOut file;
	String 	path;
	int 	index;
	int 	cmd;
};

enum Command {	OPEN, READ, CLOSE, FINISH };

const char *sftp_server = "demo.wftpserver.com";
const char *sftp_user   = "demo-user";
const char *sftp_pass   = "demo-user";
const char *remote_file = "/download/F11_wallpaper_06_1600x1200.jpg";

void SFtpExample::Download()
{
    // Initialize and fill an array of Job(s)

    for(int i = 0; i < 10; i++) {
        Job& job = jobs.Add();
        job.sftp.User(sftp_user, sftp_pass);
        job.sftp.StartConnect(sftp_server, 2222);
        job.sftp.WhenDo = THISBACK(UpdateGui);
        job.file.Open(Format("/home/testuser/picture-%d.jpg", i));
        job.cmd     = OPEN;
        job.index   = i;
    }

    // Actual loop: connects to the server and downloads a file in a concurrent way.

    while(jobs.GetCount()) {
        for(int i = 0; i < jobs.GetCount(); i++) {
            Job& job = jobs[i];
            SocketWaitEvent e;
            e.Add(job.sftp.GetSocket());
            e.Wait(10);
            job.sftp.Do();
            if(!job.sftp.InProgress()) {
                if(job.sftp.IsSuccess()) {
                    switch(job.cmd) {
                        case OPEN:
                            job.sftp.StartOpen(remote_file, SSH::READ);
                            job.path = remote_file;
                            job.cmd  = READ;
                            continue;
                        case READ:
                            job.sftp.StartGet(job.file, THISBACK2(DownloadProgress, job.index, job.path));
                            job.cmd  = CLOSE;
                            continue;
                        case CLOSE:
                            job.sftp.StartClose();
                            job.cmd  = FINISH;
                            continue;
                        case FINISH:
                            break;
                    }
                }
                else
                if(job.sftp.IsFailure())
                    list.Set(job.index, 1, DeQtf(job.sftp.GetErrorDesc()));
                jobs.Remove(i);
                list.Remove(i);
                for(int n = 0; n < jobs.GetCount(); n++)
                    jobs[n].index = n;
            }
        }
    }
}	


P.s. I delayed the upload of the package, since it still has some rough edges to iron-out.

Regards,
Oblivion


Cool. Yesterday I had to fix some ugly PHP application (do not ask...) on one of those hosting platforms where you only have SFTP access.

I was thinking that it would be a cool demonstration (and a very nice PR CodeProject article) to create actual EDITOR of text files over FTP/FTPS/SFTP in "list of files in the left" style. Not sure I will have enough time for that, but it would be fun.

Mirek
Re: SFTP or full SSH2 support for U++? [message #45963 is a reply to message #45952] Mon, 01 February 2016 00:06 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
Hello Mirek,

Quote:

I was thinking that it would be a cool demonstration (and a very nice PR CodeProject article) to create actual EDITOR of text files over FTP/FTPS/SFTP in "list of files in the left" style. Not sure I will have enough time for that, but it would be fun.


Writing an FTP(S) & SFTP editor wouldn't be too difficult. We can use Any container and dynamically create both. Plus, directory enty parsers of the both classes are very similar. It would be easy to achive some abstract API when writing the editor. What worries me is that it'll end up in a somewhat complex code, and that might be bad for PR. Instead, we can write a UI skeleton and build two separate editors, one utilizing FTP(s), and the other SFTP. In this way, people can study the code(s) and the differences. While I am currently polishing SSH/SFtp package, I can start writing a basic FTPS based text editor, in the meantime.

By the way, I was going to propose writing a CodeProject article for TcpSocket (I can write it in the following weeks, bu you may need to review it first Smile ). IMO, it is a vital part of Upp/Core. Yet it is somewhat an enigma. It is well designed but poorly documented. (not talking about its API doc). I had to study HttpRequest/GuiWebCrawler to understand how to do nonblocking operations. Maybe we can clarify that.
Also I would like to propose adding two methods for TcpSocket: My experiences with sockets and remote operations showed me that sending local IP over sockets is not uncommon.
So, maybe it is possible to add a counterpart to GetPeerAddr() method: GetLocalAddr()?
And of course EWOULDBLOCK is technically an error, but in practice we don't always treat it that way.
So I believe adding a public WouldBlock() method to check if the socket would block would be helpful in writing nonblocking code. (GetError() is not very helpful when reading and studying the source code).


Regards,
Oblivion.





[Updated on: Thu, 04 February 2016 08:27]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #46099 is a reply to message #45813] Sun, 06 March 2016 23:39 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
Hello guys,

A major update:

SSH package is re-written using the new version of AsyncQueue class which can utilize ordinary callbacks and lambdas, instead of the cumbersome VarArgs class. SSH package relies heavily on lambda functions, therefore it will require C++11. I believe it is much more elegant now, since it drastically simplifed the task and reduced the source code significantly. It is almost complete, and first release is at hand. (SFtp component, at least).

Below code snippet is taken from the new version of a simple download test app that concurrently downloads a number of jpg files from a well-known public Sftp test server. As you can see, unnecessary state tracking is removed:



struct Job {
        SFtp    sftp;
        FileOut file;
        int     index;
};

Array<Job> jobs;


//........

const char *sftp_server     = "demo.wftpserver.com";
const char *sftp_user       = "demo-user";
const char *sftp_password   = "demo-user";
const char *remote_path     = "/download";

bool SFtpDownloader::ReadDir(Ssh::DirList& ls)
{
    // We'll get the directory listing, using a synchronous (blocking) call.
  
    SFtp browser;
    browser.WhenDo << [&] {ProcessEvents(); };                                                  // GUI should be responsive...
    if(browser.Connect(sftp_server, 2222, sftp_user, sftp_password)) {
 
        // Let us use a convenience method which works on paths, not file descriptors.

        if(browser.ListDir(remote_path, ls))
            return true; 
    }
    Exclamation(browser.GetErrorDesc());
    return false;
}

void SFtpDownloader::Download()
{
    // Wait for the queue to be processed.
 
   if(!jobs.IsEmpty())
        return;
    
    Ssh::DirList ls;
    if(!ReadDir(ls))
        return;
    for(int i = 0, j = 0; i < ls.GetCount(); i++) {

        // We can work on directory entries easily.

        Ssh::DirEntry& e        = ls[i];
        const Ssh::Attrs& attrs = e.GetAttrs();

        // Let us simply try to download all files with *.jpg extension to the current directory, asynchronously.
        // Here, thanks to AsyncQueue, asynchronous calls work in a fire-and-forget fashion. Similar to "batch processing". 
        // E.g., any number of Sftp commands/jobs can be queued to be processed without intervention (till they're finished, or failed, of course). 

        if(attrs.IsFile() && GetFileExt(e.GetName()).IsEqual(".jpg")) {
            Job& job    = jobs.Add();
            job.index   = j;
            job.file.Open(Format("%s/SFtpExample-%d-%s", GetHomeDirectory(), job.index, e.GetName()));
            job.sftp.StartConnect(sftp_server, 2222, sftp_user, sftp_password);
            job.sftp.StartGet(job.file, Format("%s/%s", remote_path, e.GetName()), Ssh::READ, 0755, THISBACK2(DownloadProgress, job.index, e.GetName()));
            j++;
        }
    }
    
    //  Below loop does asynchronous job processing. 

    while(!jobs.IsEmpty()) {
        int i = 0;
        SocketWaitEvent we;
        we.Add(jobs[i].sftp.GetSocket());
        we.Wait(10);
        while(i < jobs.GetCount()) {
            SFtp& sftp = jobs[i].sftp;
            sftp.Do();
            if(!sftp.InProgress()) { 
                // For the sake of simplicity, we do not differentiate between success and failure.
                jobs.Remove(i);
                break;
            }
            ProcessEvents();
            i++;
        }
    }
}

//........



Before I release the first public [beta] version, I'll have to write its docs. It'll probably happen this week.

index.php?t=getfile&id=4957&private=0

Regards,

Oblivion.


[Updated on: Mon, 07 March 2016 08:36]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48061 is a reply to message #45813] Sat, 13 May 2017 21:01 Go to previous messageGo to next message
Tom1
Messages: 794
Registered: March 2007
Contributor
Hi Oblivion,

This sounds interesting and might well fit my needs to automate some file transfers. Is it possible to get a hold of a copy of this SFTP client module of yours?

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48072 is a reply to message #48061] Sun, 14 May 2017 16:43 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
Quote:
Hi Oblivion,

This sounds interesting and might well fit my needs to automate some file transfers. Is it possible to get a hold of a copy of this SFTP client module of yours?

Best regards,

Tom


Hello Tom,

Since you've revived this topic, let me give some information on recent developments. Smile

Some time ago I decided to go for a full SSH package. As of today, SFtp and Scp modules are complete, and Exec module is to-be-written.
I'm cleaning up the codebase now. A technical preview version is scheduled, and will be available in June 10, 2017. Smile
I'll publish and further develop it for U++ with a BSD license, so of course you'll have the chance to get a copy.
But I'm afraid you'll have to wait a little more.

Regards,

Oblivion






[Updated on: Sun, 14 May 2017 16:45]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48074 is a reply to message #48072] Sun, 14 May 2017 17:46 Go to previous messageGo to next message
Tom1
Messages: 794
Registered: March 2007
Contributor
Hi Oblivion,

Excellent! I'll stay tuned for the technical preview on this channel... Smile

If I'm not reacting to your technical preview announcement in a reasonable time, please PM me about the opportunity to test it.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48383 is a reply to message #45813] Wed, 28 June 2017 01:13 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
Hello everyone,

It is now time to publish the technical preview version of SSH package for Upp. Smile

A brief status of the project:

SSH package for U++
--------------------

SSH package is a libssh2 wrapper for Ultimate++.
It aims to be a complete secure shell client solution. (SFtp, Scp, Exec, Terminal, X11Exec, KnownHosts)
Currently it is at a technical preview stage, which means it is a work-in-progress, and subject to change.

Requirements:

- A C++ compiler with at least C++11 support.
- libssh2 ssh library (https://www.libssh2.org/)

Features and Highlights:

- Tight integration with U++.
- Support for both synchronous and asynchronous operations  with both high level and low level methods.
- Allows using U++ style memory [de/re]allocators with libssh2.
- Ssh subsystems support RTTI, and share a common interface for polymorphism. 
  Namely, it is possible to store different subsystems in the same U++ container. 
  (e.g. SFtp, Exec, Scp, Shell can be all stored in the same Array.)

Todo:

- Add known hosts and ssh agents support.
- Add more high level methods.
- Add ssh X11 support.
- Ship libssh2 library with the package.
- Documentation.

History:
-----------------

2017-06-28:  EAGAIN is now properly handled across the subsystems.
             WhenWrite and WhenRead callbacks are removed in favour of parametrized gates.
2017-06-27:  SFtp::StartStop: fixed.
2017-06-27:  CleanUp() methods are added to the classes Ssh, SFtp, and Channel.
             This method utilizes JobQueue's WhenCleanup event.
2017-06-26:  U++ memory [de/re]allocators added and made default.
             initial support for ssh shell, and terminal added.
             Subsystems now perform a clean up on failure to prevent possible heap leaks.          
2017-06-22:  Initial release.


At the moment I am documenting the Package and its api.
There are some important points you need to know before getting yourself familiar with the code.

As I've mentioned in the above summary, SSH package is a libssh2 wrapper. But currently it does not contain libssh2 source code (which also has BSD license).
On linux, your distribution probably provides the lib. On Windows you'll need cmake for express compilation.

Design:

The classes of SSH package are based on JobQueue class, a simple job-queue model (a callback queue based on Upp::Vector actually).
JobQueue is meant to work with (basically) sockets. It provides a uniform interface for async socket operations. In fact, it is an abstraction of Upp's own HttpRequest class' async interface.
From my experience, I can say that it is pretty scalable with a negligable overhead.It allows batch processing (e.g. you can queue the commands and then batch-process them later.)
However if you are going to batch process some commands, do note that the queue will treat the batch as a whole. Hence a failure in one command will halt and clear the queue. This is a security measure to prevent any misbehaviour.

Other advantage of JobQueue is that it allows writing code in a uniform coding pattern. (You can see it in SSH package clearly), in the sense that it exposes both a small set of common functions, and a basic structure.
All SSH sub/classes are based on JobQueue. Therefore they expose a uniform asynchronous api.
Also, each blocking method has its nonblocking counterpart (starting with "Start" prefix.)



Categorization:

SSH package is divided into several classes, based on system-subsystems relationship.
[
Ssh           -> Main session class (system). Provides connection and authentication mechanism. Although known Hosts support not added yet, it will be available in the following versions.

SshSubsystem  -> SshSubsystem is base of all SSH protocol based communication subsystems.
--------------
SFtp          -> Complete, but I'm planning to further enhance it with more high level commannds, once the api stabilized.
Channel       -> Mostly done.
Scp           -> Complete.
Exec          -> Complete.
Shell         -> Not added yet. Besic support is present. And will be developed durther.
Terminal      -> Not added yet. Basic support is present. And will be developed further.
X11Exec       -> Not added yet. Will be added incrementally.


Core class, as one might expect is, Ssh. Smile
It provides connection and authentication mechanisms.

All other Ssh subsystems are derived from a single base class: SshSubsystem.
This class provides common operations and connects subsystems to the Ssh session.

Also it has RTTI interface, which brings us to polymorphism:
SSH package allows a great level of polymorphism while keeping the crucial interface uniform throughout the derivative classes (thanks to JobQueue).
Hence it is possible to put, say, a Scp, Exec, Sftp, Terminal in a single Upp::Array and execute them asynchoronusly through a uniform calling pattern.

All other classes -except SFtp- are derived directly from Channel subsystem. All derivatives expose Channel commands, while has their own commands. So any Channel-based derivative use the power of Channel.

Memory Allocation:

libssh2 provides support for custom allocators. (or alternatively it can use system's memory allocators via USEMALLOC flag)
Basically I copied and adjusted the memory manager found in Core/SSL package.
It works fine with libssh2, but I'm not very happy with it (code duplication is something I don't like) Maybe there is a better way?

How To:

Package currently contains three simple examples:


SshExec: This simple example demonstrates a remote command execution via ssh2 protocol's exec channel, and the blocking api of SSH package.
SFtpDir: This simple example demonstrates a remote diretory listing via sftp channel, and the blocking api of SSH package.
SshAsyncRequests: This simple example demonstrates polymorphism with ssh subsystems, and asynchronous calls, combining a SFtp and Exec instance.

In fact, the first two example are so simple that they look like little code snippets: Smile

SshExec:
#include <Core/Core.h>
#include <SSH/SSH.h>

using namespace Upp;

// SshExec.cpp: 
// Demonstrates remote command execution on an ssh channel.

// A popular public test server:
// -----------------------------
// Host:     test.rebex.net
// User:     user
// Password: password
// Command:  ls -l

CONSOLE_APP_MAIN
{
	Ssh session;
	Cout() << "Hostname: ";
	auto host = ReadStdIn();
	Cout() << "Username: ";
	auto user = ReadStdIn();
	Cout() << "Password: ";
	auto pass = ReadStdIn();
	const int port = 22;

	if(session.Connect(host, port, user, pass)) {
		Cout() << "Command: ";
		auto cmd = ReadStdIn();
		Exec xc(session);
		auto rc = xc(cmd, Cout(), Cerr());
		if(xc.IsFailure()) Cerr() << Format("Exec failed. %s", xc.GetErrorDesc());
		else Cout() << Format("Remote program exited with code: %d\n", rc);
	}
	else Cerr() << session.GetErrorDesc() << '\n';
}


Or, SFtpDir:

#include <Core/Core.h>
#include <SSH/SSH.h>

using namespace Upp;

// SFtpDir.cpp: 
// Demonstrates remote directory listing via an sftp channel.

// A popular public test server:
// -----------------------------
// Host:     test.rebex.net
// User:     user
// Password: password
// Path:     pub/examples

CONSOLE_APP_MAIN
{
	Ssh session;
	Cout() << "Hostname: ";
	auto host = ReadStdIn();
	Cout() << "Username: ";
	auto user = ReadStdIn();
	Cout() << "Password: ";
	auto pass = ReadStdIn();
	Cout() << "Path: ";
	auto path = ReadStdIn();
	const int port = 22;

	if(session.Connect(host, port, user, pass)) {
		SFtp sftp(session);
		SFtp::DirList ls;
		if(!sftp.ListDir(path, ls)) Exit(sftp.GetError());
		for(auto& e : ls)  Cout() << e.GetEntry() << "\n";
	}
	else Cerr() << session.GetErrorDesc() << '\n';
}


That's it Smile






Please keep in mind that while ths package is reasonably stable it is still a work-in-progress. So, USE IT AT YOUR OWN RISK. Smile

Although, given my personal experience I am pretty satisfied by SSH package's api, it is by no means set in stone. It'll change when necessary.
I'll be grateful for any criticism, suggestions, reviews, patches, etc. After all, this package is for U++ and its users.

Package tested on Win7/10 (Mingw64), and latest Arch Linux and Fedora (25).

p.s. There is a leftover that must be changed, which I realized this morning. (In SshSubsystem::To() method) I will change it tonight, but in the meanwhile if you are going to test the package please consider changing the reinterpret_cast to dynamic_cast.

Cheers,
Oblivion.



[Updated on: Wed, 28 June 2017 08:28]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48384 is a reply to message #48383] Wed, 28 June 2017 14:46 Go to previous messageGo to next message
Tom1
Messages: 794
Registered: March 2007
Contributor
Hi Oblivion,

This looks good. However, I may need to wait for the following item on your task list to be completed:

- Ship libssh2 library with the package.

Compiling libssh2 became a bit frustrating on my Windows system, so I decided to rather wait for libssh2 becoming part of the delivery. In my view, the libssh2 should be embedded inside your SSH package in a way that the whole package just gets built and statically linked in Windows like any other package without dependencies. Well, I guess the dependency of Core/SSL should be there to make it work.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48385 is a reply to message #48384] Wed, 28 June 2017 15:04 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
Quote:
Re: SFTP or full SSH2 support for U++? Wed, 28 June 2017 15:46
Tom1
Hi Oblivion,

This looks good. However, I may need to wait for the following item on your task list to be completed:

- Ship libssh2 library with the package.

Compiling libssh2 became a bit frustrating on my Windows system, so I decided to rather wait for libssh2 becoming part of the delivery. In my view, the libssh2 should be embedded inside your SSH package in a way that the whole package just gets built and statically linked in Windows like any other package without dependencies. Well, I guess the dependency of Core/SSL should be there to make it work.

Best regards,

Tom


Hello Tom,

Of course I am going to include the libssh2 package in the following days (probably next week). Smile
There's too much frustration about libssh2's windows build. And those complaints are usually right.
The problem is that the libssh2 team didn't seem to provide a clear compilation guide.
While I personally prefer OpenSSL, it is not mandatory for libssh2. Windows has CNG since Vista. I can make it an option too.

Fortunately It is actually very easy to compile libssh2 on windows. (which makes you get even more frustrated, finding this after hours of of code-digging).

In the mean time I can write a step by step guide (with screenshots) on how to compile it on Windows (using mingw64, with optional OpenSSL or WinCNG).

Best regards,
Oblivion.


[Updated on: Wed, 28 June 2017 15:10]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48386 is a reply to message #48385] Wed, 28 June 2017 15:29 Go to previous messageGo to next message
Tom1
Messages: 794
Registered: March 2007
Contributor
Hi Oblivion,

Thanks, but you do not need to write the libssh2 compilation guide for my sake. I'm not in that much hurry with this and I can certainly wait a little bit longer so that you can merge the libssh2 inside a U++ package.

BTW: I wish this package is then compilable with MSC14 and above, since I cannot use mingw because of some special external library dependencies I have.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48387 is a reply to message #48386] Wed, 28 June 2017 15:50 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
AFAIK Koldo (who had the chance to test my code first hand, and whom I should thank here!) has managed to compile it on VS 2015.
IT needs slight changes (such as that some "."s in lambda functions should be substituted with "->"s), which I'll make this week end.
The thing is, I do not use MSC. I guess I'll install it as a testing environment. Smile

Best regards,
Oblivion


[Updated on: Wed, 28 June 2017 15:51]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48388 is a reply to message #48387] Wed, 28 June 2017 16:04 Go to previous messageGo to next message
Tom1
Messages: 794
Registered: March 2007
Contributor
OK. Keep up the good work! Smile

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48405 is a reply to message #45813] Fri, 30 June 2017 23:38 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
Hello,

Technical preview version of the SSH package is updated. Now it contains the libssh2 library. And uses the OpenSSL supplied by U++. Smile

I've tested it with GCC on Linux, and Mingw64 on Windows. It is compiled successfully on both systems.
I have yet to test it with MSC.

Please test it.

If it fails to compile, try configuring the libssh2 library, using libssh2_config.h header.
That header file contains defs. Should you need to modify it, possible values can be copied from the libssh2/libssh2_config.h.in file.


Maybe I should open a seperate topic on SSH package in the Bazaar section of the forums for the time being?

Suggestions, patches, and cricisim is welcome.

[See below post for the updated package]

Best regards,
Oblivion


[Updated on: Sun, 02 July 2017 17:21]

Report message to a moderator

Re: SFTP or full SSH2 support for U++? [message #48406 is a reply to message #48405] Sat, 01 July 2017 17:35 Go to previous messageGo to next message
Tom1
Messages: 794
Registered: March 2007
Contributor
Hi Oblivion,

Thanks! Here are the quick results:

I tried to compile your packages on Windows 10 Pro 32 bit with MSC14. I just copied all your packages to bazaar (locally). (Running on top of upp rev. 10804.)

When compiling, I got the following errors and warnings for SSH:

----- SSH ( MSC14 MSC WIN32 ) (2 / 6)
Ssh.cpp
C:\upp-10804\bazaar\SSH\Ssh.cpp(28): error C2143: syntax error: missing ';' before '.'
C:\upp-10804\bazaar\SSH\Ssh.cpp(28): error C2059: syntax error: '.'
C:\upp-10804\bazaar\SSH\Ssh.cpp(35): error C2059: syntax error: '.'
C:\upp-10804\bazaar\SSH\Ssh.cpp(46): error C2143: syntax error: missing ';' before '.'
C:\upp-10804\bazaar\SSH\Ssh.cpp(46): error C2059: syntax error: '.'
Channel.cpp
C:\upp-10804\bazaar\SSH\Channel.cpp(134): warning C4244: '=': conversion from 'Upp::int64' to 'int', possible loss of data
SFtp.cpp
C:\upp-10804\bazaar\SSH\SFtp.cpp(269): warning C4244: 'argument': conversion from 'Upp::int64' to 'int', possible loss of data
C:\upp-10804\bazaar\SSH\SFtp.cpp(269): warning C4244: 'argument': conversion from 'Upp::int64' to 'std::size_t', possible loss of data
C:\upp-10804\bazaar\SSH\SFtp.cpp(277): warning C4554: '&': check operator precedence for possible error; use parentheses to clarify precedence
c:\upp-10804\bazaar\ssh\ssh.h(165) : error C4716: 'Upp::SFtp::StartGetStat': must return a value
c:\upp-10804\bazaar\ssh\ssh.h(174) : error C4716: 'Upp::SFtp::StartRealPath': must return a value
c:\upp-10804\bazaar\ssh\sftp.cpp(231) : error C4716: 'Upp::SFtp::StartListDir': must return a value
Exec.cpp
Scp.cpp
libssh2upp.c
c:\upp-10804\bazaar\ssh\libssh2\libssh2_config.h(105): error C2375: 'snprintf': redefinition; different linkage
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1932): note: see declaration of 'snprintf'
c:\upp-10804\bazaar\ssh\libssh2/agent.c(300): warning C4996: 'sprintf': This function or variable may be unsafe. Consider using sprintf_s instead. To disable de
    precation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
c:/program files/windows kits/10/include/10.0.14393.0/ucrt\stdio.h(1769): note: see declaration of 'sprintf'
c:\upp-10804\bazaar\ssh\libssh2/channel.c(43): fatal error C1083: Cannot open include file: 'unistd.h': No such file or directory
Malloc.cpp
SSH: 7 file(s) built in (0:09.19), 1313 msecs / file, duration = 9515 msecs, parallelization 100%

There were errors. (1:40.53)


I have no time to look at the causes immediately, but hope these help you a bit. Smile

When I get to office, I will try on a 64-bit Windows platform.

Best regards,

Tom
Re: SFTP or full SSH2 support for U++? [message #48410 is a reply to message #48406] Sun, 02 July 2017 17:20 Go to previous messageGo to previous message
Oblivion is currently offline  Oblivion
Messages: 665
Registered: August 2007
Location: Turkey
Contributor
Thank you very much for providing feedback, Tom!

I've installed and configured the MSC to work with TheIDE.
It now compiles on VS 2015. Hopefully the errors should be fixed now.

Tested on:

Windows: MINGW 32/64, MSC (2015)
Linux: GCC

Please test it.

Best regards,

Oblivion.


[Updated on: Sun, 02 July 2017 17:21]

Report message to a moderator

Previous Topic: Help needed with link errors (serversocket)
Next Topic: Ultimate++ repo and chat?
Goto Forum:
  


Current Time: Fri Jun 05 02:55:47 CEST 2020

Total time taken to generate the page: 0.05152 seconds