Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » U++ Library support » U++ MT-multithreading and servers » SSH package for U++ (A feature-rich ilbssh2 wrapper for Ultimate++)
Re: SSH package for U++ [message #50131 is a reply to message #50119] Fri, 03 August 2018 17:34 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
Hello Mirek,



Quote:
Instead of:

auto sftp = session.CreateSFtp();

I would like to have

SFtp sftp(session);


This is already possible.
It's just that I preferred the method versions over the constructor in the examples.


Quote:

More important: I thing I would remove whole bunch of Gets and Puts.

Interestingly, it looks like the most important Gets / Puts are missing there

int Get(SFtpHandle h, void *ptr, int size);
bool Put(SFtpHandle h, const void *ptr, int size);


Ok, since the next version of the SSH package might end up to be the official ssh plugin for U++, I can change the data I/O api at this point.
I'll re-write the gets and puts. They'll be more aligned with the TcpSocket's.


Quote:

... then add SFtpStream instead. With methods provided in SFtp, it should be easy to do, and it would mostly remove "source/destination controversy".


This is a good and interesting idea. Smile
I'll see what I can do.


Ok then, I was going to update the package, but I'll delay the next update, and first come up with the changes you've asked.

Best regards,
Oblivion








Re: SSH package for U++ [message #50133 is a reply to message #50131] Fri, 03 August 2018 18:03 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 11991
Registered: November 2005
Ultimate Member
Oblivion wrote on Fri, 03 August 2018 17:34
Hello Mirek,
Ok then, I was going to update the package, but I'll delay the next update, and first come up with the changes you've asked.


Note that it is now part of U++, so please it is preferable that any changes you do are done in U++ svn. You should have write rights there.

(OTOH, if you just want to continue in GIT, I will manage.. Smile

Mirek
Re: SSH package for U++ [message #50134 is a reply to message #50133] Fri, 03 August 2018 18:44 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
Quote:

Note that it is now part of U++, so please it is preferable that any changes you do are done in U++ svn. You should have write rights there.


Ah, OK.
I'm on a vacation now. I didn't have the time to check the svn repository. I'll start committing the changes within next week.

Best regards,
Oblivion


Re: SSH package for U++ [message #50145 is a reply to message #50133] Tue, 07 August 2018 23:32 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
Hello Mirek,

I'll start committing the changes in a couple of days. But before I start I'd like to clear some points and ask for your opinion on some points.

Quote:

SFtpHandle* -> SFtpHandle - if something is "HANDLE", it should not be a pointer to handle.


The thing is, libbsh2 itself returns a pointer to a handle (SFtpHandle is actually an alias for LIBSSH2_SFTP_HANDLE)
I'm reluctant to change that pointer into a real handle as it'll make the things slightly more complex.
However, if we must change the naming I would suggest we use "SFtpObject" instead, since we are dealing with remote file system objects here.

Quote:

Interestingly, it looks like the most important Gets / Puts are missing there

int Get(SFtpHandle h, void *ptr, int size);
bool Put(SFtpHandle h, const void *ptr, int size);


Easy to implement. I'll propose:

int         Get(SFtpObject* obj, void* buffer, int size);
String      Get(const String& path, int size, int64 offset = 0);
String      GetAll(const String& path);

int         Put(SFtpObject* obj, const void* buffer, int size);
int         Put(const String& path, const String& data, int size, dword flags, long mode);
int         Put(const String& path, const String& data, int size, int64 offset = 0);
bool        PutAll(const String& path, const String& data);


As for the stream versions of Gets and Puts:
Stream versions come very handy when transferring files > 2GB, and processing the I/O immediately (e.g when filtering).

I've thought about adding SFtpStream variants(eg. SFtpFileIn, SFtpFileOut, etc.) to the package, as you suggessted.
But in the end I concluded that they'll only add another layer of abstraction which can be confusing (as you know, SSH package already has a plenty of classes). Thus I propose to simply rename them and change their parameter order.

E.g.

bool GetStream(const String& path, Stream& out);
bool PutStream(const String& path, Stream& in);
// And other variants...



About the reference examples:

My idea is to create a reference -console- example for each class (SFtp, Scp, SshExec, SshShell, SshTunnel), and have them each contain separate and basic blocking, non-blocking, and multitihreaded code as functions that can be selected/compiled using preprocessor directives.(defs,e.g. BLOCKING, NONBLOCKING, MULTITHREADED). And use these examples as tutorial in Topic++.

What do you think?


Best regards,
Oblivion


[Updated on: Wed, 08 August 2018 10:51]

Report message to a moderator

Re: SSH package for U++ [message #50146 is a reply to message #50145] Wed, 08 August 2018 10:56 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 11991
Registered: November 2005
Ultimate Member
Well, I have already did SFtpHandle* -> SFtpHandle with no ill effects.

Now I am trying to deal with "Cmd" and async operations.

I undestand the appeal, but I have to say that the result is sort of confusing. E.g. I have implemented

int Get(SFtpObject* obj, void* buffer, int size);

but it is clearly incompatible with this sort of async operations. All those NULL handles, implicit results etc make me uneasy.

Meanwhile, I have started to checking coming coroutines support (probably C++20) - on the first look, these look like "queue done right".

So I guess for now, I will try to pretend that those complex Cmd / ComplexCmd "nonblocking" operations are not there, probably putting assert to make that sure for methods like "Get" and hope to do all that right with co_wait / co_return in the future.

In related news, I have also fixed GetWaitEvents

dword Ssh::GetWaitEvents()
{
	ssh->events = 0;
	if(ssh->socket && ssh->session)
		ssh->events = libssh2_session_block_directions(ssh->session);
	return !!(ssh->events & LIBSSH2_SESSION_BLOCK_INBOUND) * WAIT_READ +
	       !!(ssh->events & LIBSSH2_SESSION_BLOCK_OUTBOUND) * WAIT_WRITE;
}



I am also thinking that perhaps SFtp should be (derived from) SshSession. I think it is unlikely that sharing SshSession for several protocols is all that important.

[Updated on: Wed, 08 August 2018 11:03]

Report message to a moderator

Re: SSH package for U++ [message #50147 is a reply to message #50146] Wed, 08 August 2018 11:22 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
I undestand the appeal, but I have to say that the result is sort of confusing. E.g. I have implemented

int Get(SFtpObject* obj, void* buffer, int size);

but it is clearly incompatible with this sort of async operations.


Well, I don't plan that method to be async (MT). IMO Async methods (that have SFtp:Asyncxxx prefix) should either use streams or consumer function (Event<const void*, int>), which will have the same effect anyway.

Asynchronous (MT) operations on ssh is somewhat complicated. I had to make some compromises for the sake of simplicity and maintainabilily, but first I have to see tha changes you've made to the code, then I'll write a summary of its design and explain those "odd" choices (it's a complex beast but works fine.)


So I guess for now, I will try to pretend that those complex Cmd / ComplexCmd "nonblocking" operations are not there, p


Those methods are the parts that I am not happy with, either Smile
I have a plan and a test code to "fix" that ugliness, but actual refactoring will come later.

All those NULL handles, implicit results etc make me uneasy.


Most of them are, I believe, taken care of and contained. Implicit results are there, but shouldn't pose any real danger (Not that Ic can see of, at least). They can, and hopefully will, be reduced (another TODO for me,connected with Cmd/ComplexCmd pair).
In related news, I have also fixed GetWaitEvents


Did it make any difference? Because IIRC values of those upp enums and the defines of libssh2 are the same.

Quote:

I am also thinking that perhaps SFtp should be (derived from) SshSession. I think it is unlikely that sharing SshSession for several protocols is all that important.


I repectfully disagree. Servers usually limit the number of sessions. (e.g. the servers I work with allows only five sessions from the same user.) OTOH, there is no channel limit. (they are only limited with bandwidth). Also this way we will give the developers flexibility. For example,If I have the ability to use Scp to transfer files while browsing and manipulating file system objects with SFtp, I'll prefer Scp for transfers, since it is relatively faster.



Also I have a plan to override libssh2 raw data read and write methods (it allows it) and use TcpSocket directly, and move Wait() there.



Best regards,
Oblivion.


[Updated on: Wed, 08 August 2018 11:45]

Report message to a moderator

Re: SSH package for U++ [message #50149 is a reply to message #50147] Wed, 08 August 2018 13:28 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 11991
Registered: November 2005
Ultimate Member
Oblivion wrote on Wed, 08 August 2018 11:22

Those methods are the parts that I am not happy with, either Smile
I have a plan and a test code to "fix" that ugliness, but actual refactoring will come later.


Is there any real-world example where you would need this kind of nonblocking behaviour?

All I can came up is some code that communicates with thousands of ssh servers at once. Looks very unlikely to me....

Quote:

Did it make any difference? Because IIRC values of those upp enums and the defines of libssh2 are the same.


Are they documented to be the same?

Mirek
Re: SSH package for U++ [message #50150 is a reply to message #50149] Wed, 08 August 2018 14:16 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
Is there any real-world example where you would need this kind of nonblocking behaviour?

All I can came up is some code that communicates with thousands of ssh servers at once. Looks very unlikely to me....


I use it in an app and for a limited number of ssh channels (usually 10-20).
But frankly, that code remains before the the CoWork improvements and the arrival of AsyncWork.
Nowadays in most such cases I use the async methods.



A short technical info:

- Ssh-based classes use a single callable target queue (FIFO), relying on deferred execution.

- Call to the queue elements are protected by a single static mutex (using INTERLOCKED).

- This allows easy maintenance but has a negative impact on the asynchronicity (it is not really possible to achieve %100 asynchronous operations with the design of libssh2 anyway. Yet the performance gains can be up to %30 in MT mode)

- Queue is populated by two interrelated methods: Cmd/ComplexCmd.
Cmd: Adds a single ananymous function to the callable target queue. In blocking mode it executes the function immediately. In non-blocking mode it defers the execution.
ComplexCmd: Firs and foremos,t this method acts as a nest for other Cmds. and other nests. This way it is possible to execute command chains in a consistent way (as if they are a single CMD).


Certainly you are not comfortable with the current implementation. (Besides, I haven't tested it yet bu as far as I can see your Get() implementation in svn won't work in non-blocking mode)

I have a new proposal: What if I get rid of queue mechanism and rewrite the package with only blocking mode and optional async transfer methods(using AsyncWork, and naming them agein SFtp::Asyncxxx)?
It won't take more than a week for me to come up with a working SSH package and the existing public API wont change much (only the NB helpers will be gone).
Besides its SC will be lot cleaner.

In fact, I had a working prototype of this (was using my Job class) I ditched in favour of the existing version.


As to your Get implementation:

int SFtp::Get(SFtpHandle handle, void *ptr_, int size0)
{
	int done = 0;                             // <- Can't be used in non-blocking mode.
	char *ptr = (char *)ptr_;
	Cmd(SFTP_START, [=, &done]() mutable {    
		int size = size0;
		if(OpCode() == SFTP_START) {
			if(FStat(HANDLE(handle), *sftp->finfo, false) <= 0) OpCode() = SFTP_GET; // <- This is for higher-level api. should be removed.
			else return false;
		}
		while(size) {
			int rc = libssh2_sftp_read(HANDLE(handle), ptr, min(size, ssh->chunk_size));
			if(rc < 0) {
				if(!WouldBlock(rc))
					SetError(rc);
				return false;
			}
			else {
				if(rc == 0)
					break;
				size -= rc;
				done += rc;
				if(WhenProgress(done, size0))
					SetError(-1, "Read aborted.");
				ssh->start_time = msecs();
			}
		}
		LLOG(Format("%d of %d bytes successfully read.", done, size0));
		return true;
	});
	return done;
}



I haven't tested this yet, but it shouldn't work in non-blocking mode. (because the execution will be deferred (Get will immediately return) and there is a local variable ("done") )

Quote:

Are they documented to be the same?


Nope, its my fault.
Best regards,
Oblivion.



[Updated on: Wed, 08 August 2018 21:48]

Report message to a moderator

Re: SSH package for U++ [message #50152 is a reply to message #50150] Thu, 09 August 2018 11:54 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 11991
Registered: November 2005
Ultimate Member
Quote:

I use it in an app and for a limited number of ssh channels (usually 10-20).
But frankly, that code remains before the the CoWork improvements and the arrival of AsyncWork.
Nowadays in most such cases I use the async methods.


OK, no need to have fully async mode for this....

Quote:

I have a new proposal: What if I get rid of queue mechanism and rewrite the package with only blocking mode and optional async transfer methods(using AsyncWork, and naming them agein SFtp::Asyncxxx)?
It won't take more than a week for me to come up with a working SSH package and the existing public API wont change much (only the NB helpers will be gone).
Besides its SC will be lot cleaner.


Definitely.

Quote:

As to your Get implementation:

I haven't tested this yet, but it shouldn't work in non-blocking mode. (because the execution will be deferred (Get will immediately return) and there is a local variable ("done") )


Sure, as I said that was the point where I decided that fully non-blocking mode is "blocking" this kind of interface.

BTW, digging deeper into the code

bool SshChannel::Lock()
{
	if(*lock == 0) {
		LLOG("Channel serialization lock acquired.");
		*lock = ssh->oid;
	}
	return *lock == ssh->oid;
}


I think there is a race condition here - two threads can obtain this lock simultaneously. Now I am not sure whether is this supposed to be MT safe, but if not, why atomic, right?

Mirek

[Updated on: Thu, 09 August 2018 11:54]

Report message to a moderator

Re: SSH package for U++ [message #50153 is a reply to message #50152] Thu, 09 August 2018 12:00 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 11991
Registered: November 2005
Ultimate Member
I am also pretty ambivalent about all those static AsyncWork methods. I think these are better left to client code. Especially if we abandon the non-blocking mode.

Mirek
Re: SSH package for U++ [message #50156 is a reply to message #50152] Thu, 09 August 2018 16:24 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
Hello Mirek,


Quote:

Sure, as I said that was the point where I decided that fully non-blocking mode is "blocking" this kind of interface.


I'm sorry but I really don't understand this one.
Here is the snippet of the working version of the same method (bot in blocking and non-blocking mode) from the now-cancelled-update:

int SFtp::Read(SFtpHandle handle, Event<const void*, int>&& consumer, int size) // Data read engine.
{
        int sz = min(size, ssh->chunksize) 
 	Buffer<char> buffer(sz);
	int rc = libssh2_sftp_read(HANDLE(handle), buffer, sz);
	if(!WouldBlock(rc) && rc < 0)
		SetError(rc);
	if(rc > 0) {
		consumer(buffer, rc);
		sftp->done += rc;
                if(WhenProgress(sftp->done, size))
			SetError(-1, "Read aborted.");
		ssh->start_time = msecs();
	}
	return rc;
}

int SFtp::Get(SFtpHandle* handle, void* buffer, int size)
{
	
        Clear();
 	Cmd(SFTP_GET, [=]() mutable {
		int rc = Read(
			handle,
			[=](const void* p, int sz)
			{
					if(!buffer)
						SetError(-1, "Invalid pointer to read buffer");
				memcpy((char*)(buffer + sftp->done), (char*)p, sz);
			},
			size
		);
		if(rc >= 0)
			sftp->value = sftp->done;
		return rc == 0 || sftp->done == size;
	});
	return sftp->done;
}



This works as expected. Note that for any kind of get method all you have to do is simply wrap the "engine" to suit your needs.



Quote:

I think there is a race condition here - two threads can obtain this lock simultaneously. Now I am not sure whether is this supposed to be MT safe, but if not, why atomic, right?


Yep. You see, an ssh shell is a complex environment. You cannot initalize multiple shells, and/or exec channels at once (I don't want to go into details here). This is a limitation of the libssh2. Their initialization have to be in some way serialized. These so-called Lock/Unlock methods handle that serialization when multiple shells or execs were started while maintaining a single threaded non-blocking and/or multithreaded asynchronous initalization. This is what makes multiple shells or exec channels at once (and the SshShellGUI example, for that matter) possible. I added the lock for NB and started to convert it into a thread-safe version. But the idea was yet to be finalized. In the end (in the now-cancelled-update) I decided to go with a RWMutex instead.


Quote:

I am also pretty ambivalent about all those static AsyncWork methods. I think these are better left to client code. Especially if we abandon the non-blocking mode.


Ok, this is not something I would object. Once the new SSH package is done I'll write a small complementary package with a handful of MT convenience functions only and maintain it in my git repo.

To sum up:

If you find it reasonable, I'll rewrite the SSH package with only blocking mode in mind (but with neceasary thread safety, and add its components gradually).

But as I already wrote in my previous messages, let us not change the "single session-multiple channels model" and not exclude any components.
When you start working in environments that rely on ssh ecosystem, as I do, you eventually end up working with ssh tunnesl, exec, and shells.
Besides the current shell implementation we have is AFAIK one of its kind. Smile I don't know any other open source shell component that does what Upp::SshShell does.
Hence it has educational and advertorial value. I mean, If you look up on stackoverflow or other relevant sites you'll see that even a barebone practical ssh shell implementation with libssh2, and also libssh is a mystery to people, let alone a shell that can have running multiple instances at the same time and that even works under Windows dumb-console. We can use this to promote U++, by writing a tutorial, demonstrating the shell in, say, codeproject.


Is this OK?

If so, I'll start writing the new code and commit the changed package within next week.

Best regards,
Oblivion






[Updated on: Thu, 09 August 2018 17:03]

Report message to a moderator

Re: SSH package for U++ [message #50157 is a reply to message #50156] Thu, 09 August 2018 16:49 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 11991
Registered: November 2005
Ultimate Member
Oblivion wrote on Thu, 09 August 2018 16:24
Hello Mirek,


Quote:

Sure, as I said that was the point where I decided that fully non-blocking mode is "blocking" this kind of interface.


I'm sorry but I really don't understand this one.



Well, if you are about to have such Get in the interface, you expect it to be usable with multiple streams at the same time. Storing into single ftp->done is no go then.

Really, let us wait for co_await and do it right then Smile

Quote:

If you find it reasonable, I'll rewrite the SSH package with only blocking mode in mind (but with neceasary thread safety, and add its components gradually).


"Pseudoblocking" - we still would like to have WhenWait and Abort (afaik it is now named "Cancel", but TcpSocket is using Abort, so maybe it should have the same name). WhenWait and Abort will allow GUI around it.... (That said, I think each channel should have its own WhenWait).

In reality, I do not think that there is a lot of new things to develop. This is mostly simplifying and removing.... Probaly Cmd will get simplified, ComplexCmd probably can be removed.

I have today developed SFtpStream (based on new blocking Get/Put) and tried to add a "new schema" SaveFile / LoadFile. It is a good feeling to load file over sftp line by line Smile Anyway, but supporting the standard stream, most of those specialised methods become unncessarry IMO.

Quote:

But as I already wrote in my previous messages, let us not change the "single session-multiple channels model" and not exclude any components.


Yes, after you have explained the reason, I agree. (Well, maybe it could have "private Ssh session" and both modes, but probably not worth it)

Quote:

Is this OK?


Absolutely, except I might want to work on it a bit too Smile So please at least check what happens in svn.

Mirek
Re: SSH package for U++ [message #50158 is a reply to message #50157] Thu, 09 August 2018 17:03 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
"Pseudoblocking" - we still would like to have WhenWait and Abort (afaik it is now named "Cancel", but TcpSocket is using Abort, so maybe it should have the same name). WhenWait and Abort will allow GUI around it.... (That said, I think each channel should have its own WhenWait).


Of course, otherwise we cant even have optional MT or even multiple channels. Smile It will be blocking from the user-POV (similar to TcpSocket).


Well, if you are about to have such Get in the interface, you expect it to be usable with multiple streams at the same time. Storing into single ftp->done is no go then.
Really, let us wait for co_await and do it right then


Sure, let's wait. Smile But I really don't understand the problem. Can you elaborate a little mode if you don't mind?
In an ssh system you don't use a single sftp channel for multiple/concurrent transfers. Channels are disposable. This is even encouraged. So each byte counter (done) is used for a single transfer. Then it gets cleared. We do multiple Sftp transfers with multiple sftp sessions.


Quote:

Absolutely, except I might want to work on it a bit too Smile So please at least check what happens in svn.


Of course, It's taking some time on my side because I'm thinking on the new design Smile

Best regards,
Oblivion


[Updated on: Thu, 09 August 2018 17:24]

Report message to a moderator

Re: SSH package for U++ [message #50159 is a reply to message #50157] Thu, 09 August 2018 17:59 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
WhenWait and Abort will allow GUI around it.... (That said, I think each channel should have its own WhenWait).


In the previous version, we had WhenWait for each object. But in the end I found it unnecessary since we have a single socket.
Besides, I was planning to use TcpSocket for the next iteration of SSH package. It is possible to use TcpSocket directly with libssh2 (libssh2 have definable callbacks for raw socket read/write functions. I was planning direcly use TcpSocket's Get and Put and move "Wait()" into those callbacks. In the current version each Ssh object has its own Wait method, which is somewhat inefficient.)


Quote:

In reality, I do not think that there is a lot of new things to develop. This is mostly simplifying and removing.... Probaly Cmd will get simplified, ComplexCmd probably can be removed.


I agree. In the blocking prototyle I'd ditched in favour of the current version I was using Cmd again, but only to execute the command immediately (no queueing).
It was the same except no queeue. This way we can also preserve the thread safety and error reporting through exceptions (in a single method) we'll need.
I'm sure that design will allow us to have a easliy restructurable base when the new co_await arrives. (Proof: this version is an improvisation on that design.)

Quote:

I have today developed SFtpStream


Great! Since we have no non-blocking mode, I have no objections left to this addition. It'll work as expected.

I'll better get started. Smile

Best regards,
Oblivion.


[Updated on: Thu, 09 August 2018 18:01]

Report message to a moderator

Re: SSH package for U++ [message #50160 is a reply to message #50156] Thu, 09 August 2018 18:35 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 11991
Registered: November 2005
Ultimate Member
Oblivion wrote on Thu, 09 August 2018 16:24

Yep. You see, an ssh shell is a complex environment. You cannot initalize multiple shells, and/or exec channels at once (I don't want to go into details here). This is a limitation of the libssh2.


I would like to learn a bit about this. Can you give me some link(s)?

Mirek
Re: SSH package for U++ [message #50174 is a reply to message #50160] Sun, 12 August 2018 12:51 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
Hello Mirek,

I've commited the first party of updates.

A summary:


- Ssh, SshSession, and SFtp classes are refactored.

- Non-blocking mode and multithreaded methods are removed.

- The package now uses a "pseudo-blocking" technique, similar to TcpSocket's.

- Accordingly, the Ssh::Cmd and Ssh::ComplexCmd methods are removed in favor of Ssh::Run and Ssh::Do() commands.
  Run() command is the new command execution encgine. It'll hopefully take care of the possible thread safety problems.
  Do() method is protected by a single static mutex, and should be thread-safe. Multithreaded execution should be possible.

- Abort mechanism refactored.

- Sftp::Read and SFtp::Write methods are further simplified.

- SFtp::WhenProgress method is removed in favour of GetDone() method (as the esisting WhenProgress method is now pretty useless).

- SshChannel and its derivatives are currently disabled. They will be re-added gradually.


I believe the new code can be easily maintained and changed in the future. (if we plan to take advantage of coroutines, etc.)

As for the SFtpStream and its variants: They are quite handy indeed. Thanks!


Quote:

I would like to learn a bit about this. Can you give me some link(s)?


Sorry, I should've phrase my words better: SSH and its relevant protocols are just fine. The problem I mentioned arises with implementations (on server or client side).
As you know, ssh is inherently an asynchronous communication protocol relying on channel multiplexing. So both servers and clients need to keep track of every package they sent and received.
In some cases where multiple channels are requested at the same time (be it in non-blocking-ST or in MT) channel requests fail with "Failed waiting for channel success" message.
The reason seems to be that SSH_CHANNEL_SUCCESS message for some channels are either somehow get lost in the process or not delivered in due time.
(Yet we get a valid handle for the requested channel from libssh2. So while we got a valid channel handle in the end, the subsequent calls would still fail.)
This is possibly a result of a race condition (at implementaion level).

I wrote that locking/unlocking mechanism as a workaround for this issue. Basically it functioned as a crude bu effective "WouldBlock()".
It allowed us to maintain the non-blocking behaviour while sequentially initalizing the requested channels.

Best regards,
Oblivion


[Updated on: Sun, 12 August 2018 12:58]

Report message to a moderator

Re: SSH package for U++ [message #50226 is a reply to message #50160] Thu, 30 August 2018 07:27 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
Hello Mirek,

Before I commit the latest update to the code, I'd like to know if adding a "SFtpFileSystemInfo()" (non static) would be ok? It'll be an example of remote file system representation. I think FileSel, or any other file browser can benefit this. Also I can write topic docs for FileSystemInfo() as I find it a useful tool (unless it is depreceated, of course).

Best regards,
Oblivion


[Updated on: Thu, 30 August 2018 07:28]

Report message to a moderator

Re: SSH package for U++ [message #50227 is a reply to message #50226] Thu, 30 August 2018 09:04 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 11991
Registered: November 2005
Ultimate Member
Oblivion wrote on Thu, 30 August 2018 07:27
Hello Mirek,

Before I commit the latest update to the code, I'd like to know if adding a "SFtpFileSystemInfo()" (non static) would be ok? It'll be an example of remote file system representation. I think FileSel, or any other file browser can benefit this. Also I can write topic docs for FileSystemInfo() as I find it a useful tool (unless it is depreceated, of course).

Best regards,
Oblivion


At this point I see no problem.
Re: SSH package for U++ [message #50228 is a reply to message #50227] Fri, 31 August 2018 00:12 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
Hello Mirek,

I've committed the changes. Package now includes a SFtpFileSystemInfo class as an experimental feature. It has its rough edges, and is to be refined, but it works. Smile

Below code is a SFtpGet to test the experimental class:

#include <CtrlLib/CtrlLib.h>
#include <Core/SSH/SSH.h>

using namespace Upp;

GUI_APP_MAIN
{
	StdLogSetup(LOG_FILE);
//	Ssh::Trace();

	SshSession session;
	if(session.Timeout(30000).Connect("demo:password@test.rebex.net:22")) {
		SFtp sftp(session);
		SFtpFileSystemInfo fsi(sftp);
		FileSel fs;
		fs.Filesystem(fsi);
		if(fs.BaseDir("/").ExecuteOpen()) {
			String file = fs.Get();
			Progress pi(nullptr, file);
			sftp.WhenProgress = [=, &pi] (int64 done, int64 total){
				pi.SetText(Format(t_("%1:s of %2:s is transferred"),
				FormatFileSize(done),
				FormatFileSize(total)));
				return pi.SetCanceled(int(done), int(total));
			};
			pi.Title(t_("Downloading ") << GetFileName(file));
			pi.Create();
			String s = sftp.LoadFile(file);
			if(sftp.IsError())
				ErrorOK(DeQtf(sftp.GetErrorDesc()));
		}
	}
	else
		ErrorOK(DeQtf(session.GetErrorDesc()));
}


Screenshot:

index.php?t=getfile&id=5650&private=0

It's not a 100% integration (at least, not yet), but it can be very useful.

Best regards,
Oblivion


Re: SSH package for U++ [message #50246 is a reply to message #50227] Sun, 02 September 2018 21:59 Go to previous messageGo to previous message
Oblivion is currently offline  Oblivion
Messages: 574
Registered: August 2007
Location: Turkey
Contributor
Hello Mirek,

The first official version of SSH package is almost done. Currently I'm stress-testing it (mostly for possible multithreading issues).
In the meantime, I rewrote the SFtpGui example and renamed it as SFtpBrowser.

I was going to commit it via SVN for review but it is somewhat lengthy for a "reference" example. (420 LOCs in cpp, mostly gui building)
It think it would be better included in uppsrc/examples. And as I don't have SVN access to examples directory, I'm uploading it here.

This example demonstrates:

- Both password and public key authenticationa.
- SSH gui integration.
- Using a slave sftp channel for file D/Us.
- Browsing an SFtp server.
- Basic SFtp commands (download, uploadr, rename, delete, mkdir).

I didn't include multithreading. I also wrote a multithreaded version of the browser with a download queue. Although it is a cool example, I'm not sure if it'll fit nicely in reference or examples section. So it'll be available at my git repo, along with the new version of GUI integrated ssh shell example.

Also, I'll improve the SFtpFileSystemInfo. It works well, but while I was studying the FileSel code, I've noticed that FileList/Load() function (FileSel.cpp, ln: 467-540) disregards the last modification time of files.
Is this intentional?
I think it should include that information too.

Best regards,
Oblivion



[Updated on: Sun, 02 September 2018 22:40]

Report message to a moderator

Previous Topic: How do I create a loop for a window to react to volatile changes to a global variable?
Next Topic: TURTLE high cpu usage, potential security flaw, and client handling problem
Goto Forum:
  


Current Time: Tue Aug 20 02:41:00 CEST 2019

Total time taken to generate the page: 0.02192 seconds