Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Community » Newbie corner » [Solved] SFTPBrowser example build failed (SFTPBrowser example buil error: error LNK2019: unresolved external symbol __imp_CertCloseStore referenced in function winstore_close)
[Solved] SFTPBrowser example build failed [message #61263] Wed, 18 December 2024 10:10 Go to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
I try sample SFTPBrowser since plan study the sample and then create a GUI for secure FTP later.

I try build with MSVS22x64 and CLANGx64 but both failed with below error message.

Do someone have experiences to fix this link issue? Thanks.

Build Error Message under MSVS22x64:
(): Linking has failed
(): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertCloseStore referenced in function winstore_close
(): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertFindCertificateInStore referenced in function winstore_win_advance
(): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertFreeCertificateContext referenced in function winstore_close
(): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertOpenSystemStoreW referenced in function winstore_open
(): C:\upp\out\MyApps\MSVS22x64.Gui\SFtpBrowser.exe : fatal error LNK1120: 4 unresolved externals


CLANGx64 error msg:

(): Linking has failed
(): ld.lld: error: undefined symbol: __declspec(dllimport) CertOpenSystemStoreW
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_ope n)
():
(): ld.lld: error: undefined symbol: __declspec(dllimport) CertFreeCertificateContext
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_ope n)
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_set _ctx_params)
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_clo se)
():
(): ld.lld: error: undefined symbol: __declspec(dllimport) CertFindCertificateInStore
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_set _ctx_params)
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_loa d)
():
(): ld.lld: error: undefined symbol: __declspec(dllimport) CertCloseStore
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_clo se)
(): clang-18: error: linker command failed with exit code 1 (use -v to see invocation)

20241222 Update: This issue finally solved with Oblivion's professional and kindly support. You can go through all posts to get details or direct go to last post to get summary.

[Updated on: Sun, 22 December 2024 13:41]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61264 is a reply to message #61263] Wed, 18 December 2024 10:55 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

There seems to be a change in windows SSL builds, so package configuration for SSH package needs to be adjusted. I will send a patch ASAP, but in the meantime, you can do it manually.
Open package organizer in TheIde (right click on SSH package and select Package Organizer menu item) then add "crypt32" to the libraries, as shown below:

index.php?t=getfile&id=7021&private=0


It should work.

Best regards,
Oblivion


[Updated on: Wed, 18 December 2024 10:56]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61271 is a reply to message #61264] Wed, 18 December 2024 13:54 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Thank you @Oblivion. Yes, can successful build now.

And I try run the exe, it able connect to SFTP server.
However several issues:
1. My SFTP server have some folders, but the GUI did not show up.
2. I try create a folder through exe, no response & error. I use other software to connect to my SFTP, the folder does created.
3. I try upload a small text file, such as test.txt, only 47byte, it seems error. However, it create a 0 byte test.txt file in SFTP server if I use other software to checking.
index.php?t=getfile&id=7022&private=0

4. So, I try to rebuild all, and hope the issue can be fixed since we just adjust SSH build way, the issue still same.

Below are the logs:
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: /
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
SSH: SFtp, oid: 2: Directory '//temp' is succesfully created.
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File '//test.txt' is successfully opened.
SSH: SFtp, oid: 3: File attributes successfully retrieved.
SSH: SFtp, oid: 3: EOF received.
SSH: SFtp, oid: 3: Stream write error.
SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
SSH: SFtp, oid: 2: Session deinitalized.
SSH: Session, oid: 1: Successfully disconnected from the server.
SSH: Session, oid: 1: Session handles freed.

Re: SFTPBrowser example build failed [message #61272 is a reply to message #61271] Wed, 18 December 2024 14:11 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

Quote:
Thank you @Oblivion. Yes, can successful build now.

And I try run the exe, it able connect to SFTP server.
However several issues:
1. My SFTP server have some folders, but the GUI did not show up.
2. I try create a folder through exe, no response & error. I use other software to connect to my SFTP, the folder does created.
3. I try upload a small text file, such as test.txt, only 47byte, it seems error. However, it create a 0 byte test.txt file in SFTP server if I use other software to checking.
index.php?t=getfile&id=7022&private=0

4. So, I try to rebuild all, and hope the issue can be fixed since we just adjust SSH build way, the issue still same.

Below are the logs:
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: /
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
SSH: SFtp, oid: 2: Directory '//temp' is succesfully created.
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File '//test.txt' is successfully opened.
SSH: SFtp, oid: 3: File attributes successfully retrieved.
SSH: SFtp, oid: 3: EOF received.
SSH: SFtp, oid: 3: Stream write error.
SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
SSH: SFtp, oid: 2: Session deinitalized.
SSH: Session, oid: 1: Successfully disconnected from the server.
SSH: Session, oid: 1: Session handles freed.


I use the SSH package all the time with no problem (I am also the developer and maintainer of it)

This error "usually" means you don't have read/write access in your current session:

Failed. Code = -31, SFTP Protocol Error


SSH2 servers can setup per-user read/write access differently, that may cause error. Or it may be a path error. I can't really tell at this point. Could you manually DLOG() the path strings in debug mode, so that we can examine them.


To verify this,

Could you try the below test server and see if it is working (you should be able to navigate the server and download files, you will not have write access)

Server: test.rebex.net
Port:   22
User:   demo
Password: password


Best regards,

Oblivion


[Updated on: Wed, 18 December 2024 14:17]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61273 is a reply to message #61272] Wed, 18 December 2024 15:20 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
@Oblivion, thanks for your prompt answer!

I try test the demo SFTP server which you provided, it seems work well.

However, when I switch to my SFTP server, it have the same error which I describe in last post.
I am sure my SFTP user name & password is correct and have write access right.

I guess may be dir/path error, you also mention it, so I adjust connect() function to manual set basedir as below.
session.Timeout(settings.timeout * 1000);
	if((connected = session.Connect(
		~settings.host,
		~settings.port,
		~settings.user,
		~settings.password
		))) {
		browser.Attach(new SFtp(session));
		filesystem.Mount(*browser);
		basedir = browser->GetDefaultDir();
		basedir = "msp";//add for testing
		DLOG(basedir);
		dir.SetData("msp");//add for testing
		DLOG(dir.GetData().ToString());
		DLOG(GetWorkdir());
		DLOG("Before work dir");
		Workdir(basedir);
		DLOG("After work dir, and before LoadDir");
		LoadDir();
		DLOG("After load dir");
	}
	else SessionError();
}


The list still empty, but I am able upload file or create sub folder without error. And other SFTP software does show the file & folder created too.

So, that means, loaddir() function have issue.
Not able successful list folder & files.
After add more DLOG, I am sure it caused by below:
if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false))
sortbyext ? SortByExt(list) : SortByName(list);




LOG:

SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: /
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File 'msp/test.txt' is successfully opened.

SSH: SFtp, oid: 3: File attributes successfully retrieved.
SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error ==> Scott, It caused by I try create one existing temp folder.
SSH: SFtp, oid: 2: Directory 'msp/temp2' is succesfully created.
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error ==> Scott: Should be loaddir() function caused error.
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file

I try add more DLOG:
void SFtpBrowser::LoadDir()
{
	if(browser->InProgress())
		return;
	list.Clear();
	DLOG("Before SFtpFileSystemInfo sfsi(*browser);");
	SFtpFileSystemInfo sfsi(*browser);
	DLOG("After SFtpFileSystemInfo sfsi(*browser);");
	DLOG("Before If");
	DLOG(GetWorkdir());
	if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)){
		DLOG("Before sort");
		sortbyext ? SortByExt(list) : SortByName(list);
		DLOG("After sort");
		}
	DLOG("After If");
	DLOG("Before Summary()");
	Summary();
	DLOG("After Summary(), and before Sync()");
	Sync();
	DLOG("After Sync()");
}

LOG Details:
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: / ==> Scott: Is it the root cause? I try set to known folder "MSP" instead of "/"
msp
msp
msp
Before work dir
After work dir, and before LoadDir
Before SFtpFileSystemInfo sfsi(*browser);
After SFtpFileSystemInfo sfsi(*browser);
Before If
msp ==> Scott: I try DLOG(GetWorkdir());
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file

After If
Before Summary()
After Summary(), and before Sync()
After Sync()
After load dir



Any way to fix it?

[Updated on: Wed, 18 December 2024 15:59]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61276 is a reply to message #61273] Thu, 19 December 2024 03:14 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
I try debug again, so try add list files functions in LoadDir(), that manual added function able successful list files.
So, can @Oblivion or someone else help to fix issue of
if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)){
? Thanks.


void SFtpBrowser::LoadDir()
{
	if(browser->InProgress())
		return;
	list.Clear();
	DLOG("Before SFtpFileSystemInfo sfsi(*browser);");
	SFtpFileSystemInfo sfsi(*browser);
	DLOG("After SFtpFileSystemInfo sfsi(*browser);");
	DLOG("Start list folder");
	DLOG(GetWorkdir());
	SFtp::DirList ls;
		// Get a remote dir listing.
	if(!browser->ListDir(GetWorkdir(), ls)) {
		DLOG(browser->GetErrorDesc());
		return;
	}
	// Filter the dir list.
	auto files =  FilterRange(ls, [](const SFtp::DirEntry& e) { return e.IsFile() && e.GetSize() <= 65536; });
		// Download the files, using worker threads in parallel.
	CoFor(min(files.GetCount(), 1000), [&](int i){
		DLOG(files[i]);
	});
	DLOG("Before If");		
	if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)){
	DLOG("Before sort");
		sortbyext ? SortByExt(list) : SortByName(list);
		DLOG("After sort");
		}
		
	DLOG("After If");
	DLOG("Before Summary()");
	Summary();
	DLOG("After Summary(), and before Sync()");
	Sync();
	DLOG("After Sync()");
}


LOGs:
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: /
/msp/
/msp/
/msp/
Before work dir
After work dir, and before LoadDir
Before SFtpFileSystemInfo sfsi(*browser);
After SFtpFileSystemInfo sfsi(*browser);
Start list folder
/msp/
SSH: SFtp, oid: 2: Directory '/msp/' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (4 entries)
SSH: SFtp, oid: 2: File handle freed.
-rw-r--r-- 30015 1485 411 12/19/2024 01:24:12 12/19/2024 01:24:12 MSP~some file name.csv

Before If
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
After If
Before Summary()
After Summary(), and before Sync()
After Sync()
After load dir


Re: SFTPBrowser example build failed [message #61277 is a reply to message #61276] Thu, 19 December 2024 04:15 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

I will try to fix the SftpBrowser. However, Could you try reference/SFtpFileSel example on your SSH server (see if it shows directory and file structures of your server correctly), I want to be sure that problem is in SFtpBrowser code, not elsewhere.

Best regards,
Oblivion


Re: SFTPBrowser example build failed [message #61278 is a reply to message #61277] Thu, 19 December 2024 05:41 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hello Oblivion,

I try do some minor adjustment to testing the last sample:
String  GetWorkdir() const          { 
    String workDir = dir.GetData().ToString(); 
    if(workDir == "/") return Null;
        else return workDir;
    }

Which intend set default as empty instead of "/", at least it work for 1st top layer.
(Still some bug, such as it would show as folder icon if I upload a test.txt file)

But when navigate to sub folder, it will not able correct LoadDir() again.
Then I try
DLOG(GetWorkdir());
	SFtp::DirList ls;
		// Get a remote dir listing.
	if(!browser->ListDir(GetWorkdir(), ls)) {
		DLOG(browser->GetErrorDesc());
		return;
	}
	// Filter the dir list.
	//auto files =  FilterRange(ls, [](const SFtp::DirEntry& e) { return e.IsFile() && e.GetSize() <= 65536; });	
	auto files =  FilterRange(ls, [](const SFtp::DirEntry& e) { return true; });
		// Download the files, using worker threads in parallel.
	CoFor(min(files.GetCount(), 1000), [&](int i){
		DLOG(files[i]);	});
by myself, it able list data correctly. So, may be you can update your sample to use SFtp::DirList way?

And then I try testing sfsi, since we know that portion have issue:
	DLOG("Start for sfsi.Find");	
		Array<FileSystemInfo::FileInfo> root = sfsi.Find("*");
		for(int i = 0; i < root.GetCount(); i++){
			DLOG(root[i].filename);
			//DLOG(root[i].root_style);
			//DLOG(root[i].root_desc);
		}
	DLOG("End for sfsi.Find");

It is work at 1st time when GetWordir() return Null, but failed when I select sub folder from dir dropdownlist.
Note, the SFTP server itself is Posix, since sfsi.IsPosix() return true, while my working computer is Windows.


So, I think the issue may be in Load() function in FileSel.cpp since at least work when if(dir.IsEmpty()) .
	Array<FileSystemInfo::FileInfo> ffi =
			filesystem.Find(AppendFileName(dir, filesystem.IsWin32() ? "*.*" : "*"));





And then I try reference/SFtpFileSel example per your suggestion for testing, the result as below:
index.php?t=getfile&id=7023&private=0


Re: SFTPBrowser example build failed [message #61283 is a reply to message #61278] Thu, 19 December 2024 08:11 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

Thank you for your patience and the info.

Quote:
Note, the SFTP server itself is Posix, since sfsi.IsPosix() return true, while my working computer is Windows.


That's because in most systems, SSH2 is configured to run using POSIX style file paths. Usually, unless you have a niche SSH server implementation, it should be ok.

E.g. I run OpenSSH on my windows machine and connect to windows machines with OpenSSH implementation, and the example works fine on them.

By the way, what is your ssh configuration? (not any personal info or paths, just the server's general config) I am going to see what is actually going on, and fix and improve the example but it would be nice to replicate a similar problem on my machines.

P.s: In my windows setups, the directory/file can be created or uploaded. However, the file list is not updated/refreshed after the operation. It is missing in the code. So that's one bug to fix Smile

Best regads,
Oblivion


[Updated on: Thu, 19 December 2024 09:03]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61284 is a reply to message #61283] Thu, 19 December 2024 10:00 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hello Oblivion,

Thanks for commit would make some time to fix bug. And it is not easy since not able duplicate similar env in your computer.

The SFTP server I try to connect is setup by my company IT team(A big company), I do not have enough info of it.
Should be shared to multiple people.
But does able smoothly connect and list/download/upload data through other 3rd party software, such as RealFTP, WinSCP.
I also able use PHP to manipulate that SFTP server data.

The ultimatepp SFTP function actually should be work too, such as able SFtp::DirList and so on so forth.
And it work well if connect to your demo SFTP server.

But need you help deep dive below code, such as Load() and sfsi portion, it must have something not consider well.


if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false))
            sortbyext ? SortByExt(list) : SortByName(list);


[Updated on: Thu, 19 December 2024 10:08]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61285 is a reply to message #61284] Thu, 19 December 2024 10:30 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

Sure, I'll change the directory enumaration logic (removing the sfsi, and using direct calls) and update the example for testing later today.

Best regards,
Oblivion


Re: SFTPBrowser example build failed [message #61290 is a reply to message #61284] Thu, 19 December 2024 21:47 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

I've tried to improve the example. You can find it attached.

It now reloads the directory after all actions and uses direct calls to load directory entries. See the LoadDir() method for changes and some customization possibilities.

Please check.


Best regards,
Oblivion



Re: SFTPBrowser example build failed [message #61291 is a reply to message #61290] Fri, 20 December 2024 03:07 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hi Oblivion,

Yeah, it work as expected if under DEBUG build mode after update LoadDir() as below:
Able list/download/upload file without issues.
Note: As for build output mode, I select All static and Default debug info leve = None, and add below additional line:
f.hidden = false;//add this sentence, or else summary function would not count it.

void SFtpBrowser::LoadDir()
{
	if(browser->InProgress())
		return;
	list.Clear();
	SFtp::DirList ls;
	if(browser->ListDir(GetWorkdir(), ls)) {
		for(const SFtp::DirEntry& e : ls) {
			String s = e.GetName();
			if(s == "." || s == "..") continue;
			FileList::File f; // Can be easily costumized...
			f.name = s;
			f.time = e.GetLastModified();
			f.font = StdFont();
			f.ink  = SColorText;
			f.icon = e.IsFile() ? CtrlImg::File() : CtrlImg::Dir();
			f.isdir = e.IsDirectory();
			f.length = e.GetSize();
			f.extink = SLtBlue;
                        f.hidden = false;//Scott: add this sentence, or else summary function would not count it.
			list.Add(f);
		}
		sortbyext ? SortByExt(list) : SortByName(list);
			
	}
	Summary();
	Sync();
}


But, there still have bug if under Release build mode:
which is not able upload file, always error and end with 0 byte.
LOG:
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (2 entries)
SSH: SFtp, oid: 2: File handle freed.
//Scott: I add 'RLOG(Format("dest: %s, UnixPath: %s ",dest,UnixPath(dest)));' in upload function for debug since assume may be path error.
dest: //msp/config/temp\test.txt, UnixPath: //msp/config/temp/test.txt
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File '//msp/config/temp/test.txt' is successfully opened.
SSH: SFtp, oid: 3: File attributes successfully retrieved.
SSH: SFtp, oid: 3: EOF received.
SSH: SFtp, oid: 3: Stream write error.

SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (3 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
SSH: SFtp, oid: 2: File '//msp/config/temp/test.txt' is successfully deleted.
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (2 entries)
SSH: SFtp, oid: 2: File handle freed.


So, please help on below:
1. Why debug mode able upload file without issue while release mode encounter EOF and create 0 byte file?

2. It will be show SFTP protocal error in pop up alert if intend delete a none empty folder.
(in debug log file it show:SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error)
So, suggest show frendly alert to say the folder not empty and cannot delete, instead of execute and then failed.

3. Do you know why before old code is wrong? Please better also find out root cause of sfsi or Upp::Load function too. Thanks.
if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false))
            sortbyext ? SortByExt(list) : SortByName(list);

[Updated on: Fri, 20 December 2024 04:18]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61292 is a reply to message #61291] Fri, 20 December 2024 06:05 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

In order to see what's wrong with writing a file and isolate it (Unfortunately I can't reproduce the error), could you replace the below method. Use a small file to test uploading please.

void SFtpBrowser::Transfer(int opcode, const String src, const String& dest)
{
	SFtp worker(session);
	Progress pi(this, src);

	worker.WhenProgress = [&pi](int64 done, int64 total)
	{
		pi.SetText(
			Format(
				t_("%1:s of %2:s is transferred"),
				FormatFileSize(done),
				FormatFileSize(total)
			)
		);
		return pi.SetCanceled(int(done), int(total));
	};
	pi.Create();
	switch(opcode) {
		case GET: {
			pi.Title(t_("Downloading ") << GetFileName(src));
			FileOut fout(dest);
			if(fout && !worker.LoadFile(fout, src))
				ErrorOK(DeQtf(worker.GetErrorDesc()));
			break;
		}
		case PUT: {
			pi.Title(t_("Uploading ") << GetFileName(src)); // Not for use, only for testing
			String s = LoadFile(src);
			if(auto h = worker.OpenWrite(dest); h) {
				int n = worker.Put(h, ~s, s.GetLength());
				RLOG(String() << "Uploaded: " << n << ", filesize: " << s.GetLength());
				RDUMP(worker.IsError());
				if(worker.IsError())
					ErrorOK(DeQtf(worker.GetErrorDesc()));
				worker.Close(h);
			}
			else
				RLOG("Couldn't open file: " << src << ", error: " << worker.GetErrorDesc());
			break;
		}
	}
	LoadDir();
}





Best regards,
Oblivion


Re: SFTPBrowser example build failed [message #61294 is a reply to message #61292] Fri, 20 December 2024 08:25 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hi Oblivion,

It worked without error in Release build mode with below new code:

This new method try write src into a String , and then write String to dest, so no issue of small/middle files,
but would losing visibility of progressing % during upload. Anyway to load by chunk and show progressing under new code? Thanks.

case PUT: {
			pi.Title(t_("Uploading ") << GetFileName(src)); // Not for use, only for testing
			String s = LoadFile(src);
			if(auto h = worker.OpenWrite(dest); h) {
				int n = worker.Put(h, ~s, s.GetLength());
				RLOG(String() << "Uploaded: " << n << ", filesize: " << s.GetLength());
				RDUMP(worker.IsError());
				if(worker.IsError())
					ErrorOK(DeQtf(worker.GetErrorDesc()));
				worker.Close(h);
			}
			else
				RLOG("Couldn't open file: " << src << ", error: " << worker.GetErrorDesc());
			break;
		}


old wrong code is:
case PUT: {
			pi.Title(t_("Uploading ") << GetFileName(src));
			FileIn fin(src);
			if(fin && !worker.SaveFile(dest, fin))
				ErrorOK(DeQtf(worker.GetErrorDesc()));
			break;
		}


Note: The most tricky thing is old code failed upload file when build by CLANG RELEASE & DEBUG MODE or VS2022 RELEASE MODE. Only work in VS2022 DEBUG build mode.
Do you able point out the key gap of the old code and try fix it? It is better have upload progressing bar too:) Thanks.

Or else, I have to use VS2022 DEBUG BUILD,then exe size is big. Or use your new code under RELEASE BUILD, but losing upload progressing progress.
Thanks.


Regards,
Scott Huang

[Updated on: Fri, 20 December 2024 08:41]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61295 is a reply to message #61294] Fri, 20 December 2024 08:47 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

Thanks, that's what I suspected. I will upload a patched version -functionally identical to originally intended code- with some other improvements, likely later today. Progress display will be back, it is disabled temporarily.

Best regards,
Oblivion


[Updated on: Fri, 20 December 2024 08:50]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61296 is a reply to message #61295] Fri, 20 December 2024 09:20 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hello Oblivion,

Good to know you will enhance code again to add progressing %. Thnaks.

In the meanwhile, I also try upgrade the old code and testing by myself with below update:
		case PUT: {
			pi.Title(t_("Uploading ") << GetFileName(src));
			FileIn fin(src);
			/*
			if(fin && !worker.SaveFile(dest, fin))
				ErrorOK(DeQtf(worker.GetErrorDesc()));
			break;
			*/

			SFtpFileOut fout(worker, dest);
			int64 size = fin.GetSize(), count = 0;
			int64 chunkSize = 1024 * 64;
			Buffer<byte> chunk(chunkSize, 0);
			Gate<int64, int64>      WhenProgress;
			WhenProgress(0, size);
			
				while(!fin.IsEof()) {
					RLOG(Format("size %d, count %d, chunkSize %d, min %d",  size, count, chunkSize, (int)min<int64>(size - count, chunkSize)  ));
					int n = fin.Get(chunk, (int) min<int64>(size - count, chunkSize));
					RLOG(Format("n %d",n));
					if(n > 0) {
						fout.Put(chunk, n);
						if(fout.IsError()) {
							RLOG("Stream write error. " + fin.GetErrorText() + " out stream: "+fout.GetErrorText());
							ErrorOK(DeQtf(worker.GetErrorDesc())+"Stream write error. " + fin.GetErrorText() + " out stream: "+fout.GetErrorText());
							break;
						}
						count += n;
						if(WhenProgress(count, size)) {
							break;
						}
					}
					if(fin.IsError()) {
						RLOG("Stream read error. " + fin.GetErrorText());
						ErrorOK(DeQtf(worker.GetErrorDesc())+"Stream read error. " + fin.GetErrorText());
						break;
					}
				}
				
				
			}


The LOG:

SSH: SFtp, oid: 2: File handle freed.
dest: /\test.txt, UnixPath: //test.txt
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File '//test.txt' is successfully opened.
SSH: SFtp, oid: 3: File attributes successfully retrieved.
size 47, count 0, chunkSize 65536, min 47
n 47
SSH: SFtp, oid: 3: EOF received.
Stream write error. out stream: A non-blocking socket operation could not be completed immediately.

SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 2: Directory '/' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (7 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.

May be you can checking why EOF received? Is it caused by non-blocking socket & blocking socket different if compare with your SFTP and my company's SFTP server?
Thanks.

Regards,
Scott Huang

[Updated on: Fri, 20 December 2024 09:25]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61297 is a reply to message #61296] Fri, 20 December 2024 09:45 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Quote:
May be you can checking why EOF received? Is it caused by non-blocking socket & blocking socket different if compare with your SFTP and my company's SFTP server?

I have a suspicion, so I'll try to come up with a fix, but can you tell me the the size of the files that are failed to be uploaded? Do you encounter the same problem with file size >= 64K?


Best regards,
Oblivion


Re: SFTPBrowser example build failed [message #61299 is a reply to message #61297] Fri, 20 December 2024 15:42 Go to previous messageGo to next message
Scott_Huang is currently offline  Scott_Huang
Messages: 27
Registered: December 2024
Promising Member
Hi Oblivion,

Merry Christmas!

Actually I testing different size files before last post, including 1M, all failed and create 0 size file in SFTP server.
SFtpFileOut should have some bug in case some SFTP server not similar as your demo SFTP server.

Anyway, I have upgrade code with your hint as below, it can successful upload files and also show progress %:
void SFtpBrowser::Transfer(int opcode, const String src, const String& dest)
{
	SFtp worker(session);
	Progress pi(this, src);

	worker.WhenProgress = [&pi](int64 done, int64 total) {
		pi.SetText(Format(t_("%1:s of %2:s is transferred"), FormatFileSize(done),
		                  FormatFileSize(total)));
		return pi.SetCanceled(int(done), int(total));
	};
	pi.Create();
	switch(opcode) {
	case GET: {
		pi.Title(t_("Downloading ") << GetFileName(src));
		FileOut fout(dest);
		if(fout && !worker.LoadFile(fout, src))
			ErrorOK(DeQtf(worker.GetErrorDesc()));
		break;
	}
	case PUT: {
		pi.Title(t_("Uploading ") << GetFileName(src)); // Not for use, only for testing
		String s = LoadFile(src);

		if(auto h = worker.OpenWrite(dest); h) {

			/*			int n = worker.Put(h, ~s, s.GetLength());
			            RLOG(String() << "Uploaded: " << n << ", filesize: " << s.GetLength());
			            RDUMP(worker.IsError());
			            if(worker.IsError())
			                ErrorOK(DeQtf(worker.GetErrorDesc()));
			            worker.Close(h);*/

			int64 size = s.GetLength(), count = 0;
			if(settings.log)
				RLOG(Format("File size %d", size));
			int chunkSize = 1024 * 64, thisTimeSize = 0;
			worker.WhenProgress(0, size);
			while(!(count >= size)) {
				thisTimeSize = (int)min<int64>(size - count, chunkSize);
				if(settings.log)
					RLOG(Format("======= size %d, count %d, chunkSize %d, min %d", size, count,
					            chunkSize, thisTimeSize));
				int x = worker.Put(h, (~s) + count, thisTimeSize);
				count += x;
				if(settings.log)
					RLOG(String() << "This time should be uploaded: " << thisTimeSize
					              << ", Real uploaded: " << x << ", Current Accumulated Count:"
					              << count << ", total filesize: " << size);
				// RDUMP(worker.IsError());
				if(worker.WhenProgress(count, size)) {
					break;
				}

				if(worker.IsError()) {
					ErrorOK("Upload failed : " + DeQtf(worker.GetErrorDesc()));
					break;
				}
			}

			worker.Close(h);
			break;
		}
		else if(settings.log)
			RLOG("Couldn't open file: " << src << ", error: " << worker.GetErrorDesc());

		break;
	}
	}
	LoadDir();
}


You can consider this issue fixed.
But could you help improve it when you have time:

1. Any better way if compare to above? (I am new for UPP)
2. Enhance it to real multiple threads, above actually is single upload thread, quite slow if compare to 3rd party SFTP software.

3. Why stream way failed and we have switch to one time read file into String and then chunk write into SFTP?


Regards,
Scott Huang

[Updated on: Sat, 21 December 2024 04:44]

Report message to a moderator

Re: SFTPBrowser example build failed [message #61301 is a reply to message #61299] Sat, 21 December 2024 11:11 Go to previous messageGo to previous message
Oblivion is currently offline  Oblivion
Messages: 1202
Registered: August 2007
Senior Contributor
Hello Scott,

Thank you for your patience and feedback.

I see no difference between your updated transfer code and SFtp::CopyData, which is used by SFtpFileStream derived class.
However, I suspect the server's filesystem does not support truncation, which is set by SFtpFileOut, by default, and it is NOT set by plain SFtp::OpenWrite method.
That might be the culprit, newer servers, especially flash-based ones don't support it, so the operation fails. (At least my theory).

Can you modify the SFtpStream.cpp file in SSH package and then test the original code with streams?

bool SFtpStream::Open(SFtp& sftp_, const char *filename, dword mode, int acm)
{
	if(IsOpen())
		Close();
	sftp = &sftp_;
	int iomode = mode & ~SHAREMASK;
	handle = sftp->Open(filename,
						iomode == READ
							? SFtp::READ
							: iomode == CREATE
								? SFtp::READ|SFtp::WRITE|SFtp::CREATE|SFtp::TRUNCATE // <- remove SFtp::TRUNCATE
								: SFtp::READ|SFtp::WRITE,
						acm
						);
				
	if(handle) {
		SFtpAttrs attrs;
		if(!sftp->GetAttrs(handle, attrs)) {
			sftp->Close(handle);
			handle = nullptr;
			return false;
		}
		OpenInit(mode, attrs.filesize);
	}
	return handle;
}



I will try to answer your other questions as well. Smile

Best regards,

Oblivion


[Updated on: Sat, 21 December 2024 11:12]

Report message to a moderator

Previous Topic: Latest U++ PDF version user guide to describe framework/tips/examples
Next Topic: The UPP looks good -> Opportunities to evolve to better support RPA & AI
Goto Forum:
  


Current Time: Sat Apr 26 02:22:14 CEST 2025

Total time taken to generate the page: 0.01556 seconds