Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Community » Newbie corner » [Solved] SFTPBrowser example build failed (SFTPBrowser example buil error: error LNK2019: unresolved external symbol __imp_CertCloseStore referenced in function winstore_close)
Re: SFTPBrowser example build failed [message #61304 is a reply to message #61301] Sun, 22 December 2024 02:30 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hi Oblivion,


I try remove SFtp::TRUNCATE as you said, it did not work, same error.

I also testing remove both SFtp::CREATE|SFtp::TRUNCATE from SFtpStream.cpp, then the behavior like below:
1. Failed if SFTP file not existing, it show error: Failed opening remote file. I think it caused by I remove SFtp::CREATE flag too.
2. I try use other SFTP software to upload same file firstly. And then U++ exe can upload file, but it would write content to existing file, double contents.

There should be have other subtle gaps if compare to OpenWrite function:
Open(path, CREATE | WRITE, IRALL | IWUSR);


I personal guess it caused by below code, so add several RLOG for debug: (test.txt file size is 47 byte)
	if(handle) {
		RLOG("Have handle, before checking GetAttrs");
		SFtpAttrs attrs;
		if(!sftp->GetAttrs(handle, attrs)) {
			RLOG("Failed get attrs");
			sftp->Close(handle);
			handle = nullptr;
			return false;
		}
		else{
			RLOG("Succes get attrs");
			RLOG("===Show filesize");
			DUMP(attrs);
			RLOG(attrs.filesize);
			RLOG((int64)attrs.filesize);
			RLOG("----");
		}
		OpenInit(mode, attrs.filesize);
	}

void BlockStream::OpenInit(dword mode, int64 _filesize) {
	

	streamsize = _filesize;
	
	RLOG(String()<<"++In OpenInit function - DEBUG: file size"<<_filesize);
	RLOG(_filesize);
	RLOG(streamsize);
	RLOG("---------------");
	


LOG does show attributes retrievedm, so the issue should be in function OpenInit(mode, attrs.filesize); And why additional session created
SSH: Session, oid: 1: Starting DNS sequence locally for deft.dell.com:22
SSH: Session, oid: 1: Successfully connected to deft.dell.com:22
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: /
SSH: SFtp, oid: 2: Directory '/' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (6 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 2: Directory '//msp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (3 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 2: Directory '//msp/config' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (171 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (2 entries)
SSH: SFtp, oid: 2: File handle freed.
dest: //msp/config/temp\test.txt, UnixPath: //msp/config/temp/test.txt
++In OpenInit function - DEBUG: file size47
47
47

---------------
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File '//msp/config/temp/test.txt' is successfully opened.
Have handle, before checking GetAttrs
SSH: SFtp, oid: 3: File attributes successfully retrieved.
Succes get attrs
===Show filesize
839918903936
839918903936
----
++In OpenInit function - DEBUG: file size839918903936
839918903936
839918903936
---------------

DEBUG: NOT READ
SSH: SFtp, oid: 3: EOF received.
SSH: SFtp, oid: 3: Stream write error.
SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (3 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
SSH: SFtp, oid: 2: File '//msp/config/temp/test.txt' is successfully deleted.
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (2 entries)
SSH: SFtp, oid: 2: File handle freed.


Questions: Why addition SSH oid created? Why file size wrong?

Anyway, I am ok with my last post to use worker.OpenWrite & worker.Put, it worked.

Attach good code here for others reference.

Regards,
Scott Huang

[Updated on: Sun, 22 December 2024 07:12]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61306 is a reply to message #61304] Sun, 22 December 2024 11:41 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

Thank you for your help. It would be better for us all if we fix the issue instead of using a workaround, if possible.

Given your latest log, it appears that the structure returns an invalid file size for the newly created file (it should be 0).
Hopefully, the below patch fixes the issue once and for all:

bool SFtpStream::Open(SFtp& sftp_, const char *filename, dword mode, int acm)
{
	if(IsOpen())
		Close();
	sftp = &sftp_;
	int iomode = mode & ~SHAREMASK;
	handle = sftp->Open(filename,
						iomode == READ
							? SFtp::READ
							: iomode == CREATE
								? SFtp::READ|SFtp::WRITE|SFtp::CREATE|SFtp::TRUNCATE
								: SFtp::READ|SFtp::WRITE,
						acm
						);
				
	if(handle) {
		SFtpAttrs attrs;
		Zero(attrs);
		if(iomode != CREATE && !sftp->GetAttrs(handle, attrs)) {
			sftp->Close(handle);
			handle = nullptr;
			return false;
		}
		OpenInit(mode, attrs.filesize); // filesize always should be 0 for create|truncate
	}
	return handle;
}



I would be grateful if you could try this patch and report back.


Also, I have updated the example (attached). It can now:

1) set chunksize
2) set waitstep
3) set compression
4) allows dropping files onto browser for uploading.


As for your other questions:


1. Please keep in mind that SFtpBrowser is a very simple example, demonstrating how to integrate SSH package into a GUI. It is not intended to be a rival for WinSCP or any other commercial app. Smile
Of course, SSH package allows you to write such applications. In fact I do have a such professional app and its performance is on par with WinScp, if not faster.
To achieve faster transfer speeds with SFtpFileBrowser, you can enable compression and increase the chunksize (preferably to 1 Mib).

2. Again, this is a simple example. However, file partitioning (parallel download of a single, huge file) is possible with SSH package, but it would be a complex app, beyond the scope of this demo. (my app does that too).


Best regards,
Oblivion





[Updated on: Sun, 22 December 2024 12:12]

Report message to a moderator

[RESOLVED] Re: SFTPBrowser example build failed [message #61307 is a reply to message #61306] Sun, 22 December 2024 13:25 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hello Oblivion,

It worked at last Smile

In short:

1. Fix LoadDir() issue, replaced by SFtp::DirList
2. Fix filesize (in SFtpStream.cpp open function) issue.
if(handle) {
		SFtpAttrs attrs;
		Zero(attrs);
		if(iomode != CREATE && !sftp->GetAttrs(handle, attrs)) {
			sftp->Close(handle);
			handle = nullptr;
			return false;
		}
		OpenInit(mode, attrs.filesize); // filesize always should be 0 for create|truncate
	}


And You also add more features:
1) set chunksize
2) set waitstep
3) set compression
4) allows dropping files onto browser for uploading.

I do learn a lot from these discussions/update code/do testings Smile
Appreciate your professional and kindly support! Thanks.

And please add parallel download/upload capability for single big file into this sample if you agree and when you free. Thanks.
Btw, before I use Golang Fyne GUI framework and its networking package, one of simple solution for above purpose is create several thread read that big file different portion and do the parallel upload/download.

Regards,
Scott Huang

[Updated on: Sun, 22 December 2024 13:34]

Report message to a moderator

Re: [RESOLVED] Re: SFTPBrowser example build failed [message #61308 is a reply to message #61307] Sun, 22 December 2024 14:55 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

Happy to help.

Here's an updated version. This version contains a switch to transfer files using Sftp or Scp. Scp transfers, if supported by the server, can be significantly faster.

If this is OK with you, I will update the upp-components package.

Then, I'll write a simple example for parallel download (file partitioning). But it will take some time, probably within this week.

Best regards,

Oblivion


[Updated on: Sun, 22 December 2024 14:57]

Report message to a moderator

Re: [RESOLVED] Re: SFTPBrowser example build failed [message #61309 is a reply to message #61308] Sun, 22 December 2024 15:54 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hello Oblivion,

Per testing, your new code also working for Upload. That SFTP server seems support SCP transfer for upload. Thanks.
Btw, do SCP mean Secure Copy Protocal, why it would be faster than normal SFTP way? thanks.

But SCP way failed for download. (SFTP way good for download if I change setting back)
So, split SCP for upload only? and download still use SFTP way, but enhanced to multiple thread when you have time? Thanks.

LOG:
SSH: Scp, oid: 10: Unable to open file //msp/config/temp/test.txt
SSH: Scp, oid: 10: Failed. Code = -28, Failed to recv file
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (5 entries)
SSH: SFtp, oid: 2: File handle freed.


Appreciate your continious enhancements!

Take your time of below. Merry Christmas!
Quote:

Then, I'll write a simple example for parallel download (file partitioning). But it will take some time, probably within this week.


Regards,
Scott Huang

[Updated on: Mon, 23 December 2024 03:48]

Report message to a moderator

Re: [RESOLVED] Re: SFTPBrowser example build failed [message #61314 is a reply to message #61309] Mon, 23 December 2024 08:50 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

Quote:
Btw, do SCP mean Secure Copy Protocal, why it would be faster than normal SFTP way?


Well, long story short: SCP is different from SFTP in that SFTP has a protocol overhead (it internally chunks and "acks" transfered chunks), so it will be slower on big file transfers.
The downside of scp is that it has no transfer/pause/continue mechanism, as it is only a data stream over SSH protocol. Difference is usually visible in huge file transfers.


SCP download error: Possibly your server not supporting downloads, and only uploads by SCP?


As for the file partitioning: I decided to integrate a dumbed-down version of it (can be selected in the settings) into the SFtpBrowser within a couple of days.



Best regards,
Oblivion







[Updated on: Mon, 23 December 2024 08:51]

Report message to a moderator

Re: [RESOLVED] Re: SFTPBrowser example build failed [message #61315 is a reply to message #61314] Mon, 23 December 2024 10:08 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hello Oblivion,

Likely. Anyway, I adjust settings to able select upload through SCP, but download still SFTP.
And several days later would be faster after you enable partitioning downloading. Thanks.

Quote:
SCP download error: Possibly your server not supporting downloads, and only uploads by SCP?


Regards,
Scott Huang
Re: [RESOLVED] Re: SFTPBrowser example build failed [message #61343 is a reply to message #61315] Sat, 28 December 2024 11:18 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

I have attached the latest SFtpBrowser example (with parallel download, namely file partitioning).

As I have noted in the code, ideally we would create cosessions only once (on login) and then use them as needed.

But that would make the example very complex. In order to reduce the complexity in this example I am using a more direct approach.

Keep in mind that it would make no sense to use SFTP parallel download feature with small files. It is useful for downloading large files (yet, in most cases a single threaded SCP tranfer will still be faster)

Also, I omitted some checks that would be better implemented in real-life situations.


It is up to you to write a better SFtpBrowser. In fact I encourage you to do so. This is what example codes are for. Smile

IIRC, this is the first time I am demonstrating the parallel download feature.

Best regards,
Oblivion


[Updated on: Sat, 28 December 2024 11:23]

Report message to a moderator

Re: [RESOLVED] Re: SFTPBrowser example build failed [message #61344 is a reply to message #61343] Sat, 28 December 2024 16:29 Go to previous message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hello Oblivion,

Appreciate you share the tips of filepart/LoadFilePart/CoLoadFile, really helpful.

It works only except one bug, we need add 'connected = true' in Connect function after successful connect to SFTP, or else, the mkdir & upload & context would not enabled. Thanks.

	if(CreateSession(session)) {
	        connected = true;//Add by Scott
		browser.Attach(new SFtp(session));
		basedir = browser->GetDefaultDir();
		Workdir(basedir);
		LoadDir();
	}
	else
		SessionError();




Regards,
Scott Huang

[Updated on: Sat, 28 December 2024 17:13]

Report message to a moderator

Previous Topic: Latest U++ PDF version user guide to describe framework/tips/examples
Next Topic: The UPP looks good -> Opportunities to evolve to better support RPA & AI
Goto Forum:
  


Current Time: Sat Apr 26 02:19:33 CEST 2025

Total time taken to generate the page: 0.01376 seconds