Main place for discussions about U++ cross platform development framework.
[Solved] SFTPBrowser example build failed
I try sample SFTPBrowser since plan study the sample and then create a GUI for secure FTP later. I try build with MSVS22x64 and CLANGx64 but both failed with below error message. Do someone have experiences to fix this link issue? Thanks. Build Error Message under MSVS22x64: (): Linking has failed (): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertCloseStore referenced in function winstore_close (): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertFindCertificateInStore referenced in function winstore_win_advance (): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertFreeCertificateContext referenced in function winstore_close (): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertOpenSystemStoreW referenced in function winstore_open (): C:\upp\out\MyApps\MSVS22x64.Gui\SFtpBrowser.exe : fatal error LNK1120: 4 unresolved externals CLANGx64 error msg: (): Linking has failed (): ld.lld: error: undefined symbol: __declspec(dllimport) CertOpenSystemStoreW (): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_ope n) (): (): ld.lld: error: undefined symbol: __declspec(dllimport) CertFreeCertificateContext (): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_ope n) (): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_set _ctx_params) (): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_clo se) (): (): ld.lld: error: undefined symbol: __declspec(dllimport) CertFindCertificateInStore (): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_set _ctx_params) (): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_loa d) (): (): ld.lld: error: undefined symbol: __declspec(dllimport) CertCloseStore (): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_clo se) (): clang-18: error: linker command failed with exit code 1 (use -v to see invocation) 20241222 Update: This issue finally solved with Oblivion's professional and kindly support. You can go through all posts to get details or direct go to last post to get summary.
Hello Scott, There seems to be a change in windows SSL builds, so package configuration for SSH package needs to be adjusted. I will send a patch ASAP, but in the meantime, you can do it manually. Open package organizer in TheIde (right click on SSH package and select Package Organizer menu item) then add "crypt32" to the libraries, as shown below: It should work. Best regards, Oblivion
Thank you @Oblivion. Yes, can successful build now. And I try run the exe, it able connect to SFTP server. However several issues: 1. My SFTP server have some folders, but the GUI did not show up. 2. I try create a folder through exe, no response & error. I use other software to connect to my SFTP, the folder does created. 3. I try upload a small text file, such as test.txt, only 47byte, it seems error. However, it create a 0 byte test.txt file in SFTP server if I use other software to checking. 4. So, I try to rebuild all, and hope the issue can be fixed since we just adjust SSH build way, the issue still same. Below are the logs: SSH: Session, oid: 1: Using Upp's memory managers. SSH: Session, oid: 1: Session successfully initialized. SSH: Session, oid: 1: Compression is disabled. SSH: Session, oid: 1: Handshake successful. SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive] SSH: Session, oid: 1: Client succesfully authenticated. SSH: SFtp, oid: 2: Session successfully initialized. SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: / SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file SSH: SFtp, oid: 2: Directory '//temp' is succesfully created. SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file SSH: SFtp, oid: 3: Session successfully initialized. SSH: SFtp, oid: 3: File '//test.txt' is successfully opened. SSH: SFtp, oid: 3: File attributes successfully retrieved. SSH: SFtp, oid: 3: EOF received. SSH: SFtp, oid: 3: Stream write error. SSH: SFtp, oid: 3: File handle freed. SSH: SFtp, oid: 3: Session deinitalized. SSH: SFtp, oid: 2: Session deinitalized. SSH: Session, oid: 1: Successfully disconnected from the server. SSH: Session, oid: 1: Session handles freed.
Hello Scott, Quote: Thank you @Oblivion. Yes, can successful build now. And I try run the exe, it able connect to SFTP server. However several issues: 1. My SFTP server have some folders, but the GUI did not show up. 2. I try create a folder through exe, no response & error. I use other software to connect to my SFTP, the folder does created. 3. I try upload a small text file, such as test.txt, only 47byte, it seems error. However, it create a 0 byte test.txt file in SFTP server if I use other software to checking. index.php?t=getfile&id=7022&private=0 4. So, I try to rebuild all, and hope the issue can be fixed since we just adjust SSH build way, the issue still same. Below are the logs: SSH: Session, oid: 1: Using Upp's memory managers. SSH: Session, oid: 1: Session successfully initialized. SSH: Session, oid: 1: Compression is disabled. SSH: Session, oid: 1: Handshake successful. SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive] SSH: Session, oid: 1: Client succesfully authenticated. SSH: SFtp, oid: 2: Session successfully initialized. SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: / SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file SSH: SFtp, oid: 2: Directory '//temp' is succesfully created. SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file SSH: SFtp, oid: 3: Session successfully initialized. SSH: SFtp, oid: 3: File '//test.txt' is successfully opened. SSH: SFtp, oid: 3: File attributes successfully retrieved. SSH: SFtp, oid: 3: EOF received. SSH: SFtp, oid: 3: Stream write error. SSH: SFtp, oid: 3: File handle freed. SSH: SFtp, oid: 3: Session deinitalized. SSH: SFtp, oid: 2: Session deinitalized. SSH: Session, oid: 1: Successfully disconnected from the server. SSH: Session, oid: 1: Session handles freed. I use the SSH package all the time with no problem (I am also the developer and maintainer of it) This error "usually" means you don't have read/write access in your current session: Failed. Code = -31, SFTP Protocol Error SSH2 servers can setup per-user read/write access differently, that may cause error. Or it may be a path error. I can't really tell at this point. Could you manually DLOG() the path strings in debug mode, so that we can examine them. To verify this, Could you try the below test server and see if it is working (you should be able to navigate the server and download files, you will not have write access) Server: test.rebex.net Port: 22 User: demo Password: password Best regards, Oblivion
@Oblivion, thanks for your prompt answer! I try test the demo SFTP server which you provided, it seems work well. However, when I switch to my SFTP server, it have the same error which I describe in last post. I am sure my SFTP user name & password is correct and have write access right. I guess may be dir/path error, you also mention it, so I adjust connect() function to manual set basedir as below. session.Timeout(settings.timeout * 1000); if((connected = session.Connect( ~settings.host, ~settings.port, ~settings.user, ~settings.password ))) { browser.Attach(new SFtp(session)); filesystem.Mount(*browser); basedir = browser->GetDefaultDir(); basedir = "msp";//add for testing DLOG(basedir); dir.SetData("msp");//add for testing DLOG(dir.GetData().ToString()); DLOG(GetWorkdir()); DLOG("Before work dir"); Workdir(basedir); DLOG("After work dir, and before LoadDir"); LoadDir(); DLOG("After load dir"); } else SessionError(); } The list still empty, but I am able upload file or create sub folder without error. And other SFTP software does show the file & folder created too. So, that means, loaddir() function have issue. Not able successful list folder & files. After add more DLOG, I am sure it caused by below: if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)) sortbyext ? SortByExt(list) : SortByName(list); LOG: SSH: Session, oid: 1: Using Upp's memory managers. SSH: Session, oid: 1: Session successfully initialized. SSH: Session, oid: 1: Compression is disabled. SSH: Session, oid: 1: Handshake successful. SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive] SSH: Session, oid: 1: Client succesfully authenticated. SSH: SFtp, oid: 2: Session successfully initialized. SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: / SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file SSH: SFtp, oid: 3: Session successfully initialized. SSH: SFtp, oid: 3: File 'msp/test.txt' is successfully opened. SSH: SFtp, oid: 3: File attributes successfully retrieved. SSH: SFtp, oid: 3: File handle freed. SSH: SFtp, oid: 3: Session deinitalized. SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error ==> Scott, It caused by I try create one existing temp folder. SSH: SFtp, oid: 2: Directory 'msp/temp2' is succesfully created. SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error ==> Scott: Should be loaddir() function caused error. SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file I try add more DLOG: void SFtpBrowser::LoadDir() { if(browser->InProgress()) return; list.Clear(); DLOG("Before SFtpFileSystemInfo sfsi(*browser);"); SFtpFileSystemInfo sfsi(*browser); DLOG("After SFtpFileSystemInfo sfsi(*browser);"); DLOG("Before If"); DLOG(GetWorkdir()); if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)){ DLOG("Before sort"); sortbyext ? SortByExt(list) : SortByName(list); DLOG("After sort"); } DLOG("After If"); DLOG("Before Summary()"); Summary(); DLOG("After Summary(), and before Sync()"); Sync(); DLOG("After Sync()"); } LOG Details: SSH: Session, oid: 1: Using Upp's memory managers. SSH: Session, oid: 1: Session successfully initialized. SSH: Session, oid: 1: Compression is disabled. SSH: Session, oid: 1: Handshake successful. SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive] SSH: Session, oid: 1: Client succesfully authenticated. SSH: SFtp, oid: 2: Session successfully initialized. SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: / ==> Scott: Is it the root cause? I try set to known folder "MSP" instead of "/" msp msp msp Before work dir After work dir, and before LoadDir Before SFtpFileSystemInfo sfsi(*browser); After SFtpFileSystemInfo sfsi(*browser); Before If msp ==> Scott: I try DLOG(GetWorkdir()); SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file After If Before Summary() After Summary(), and before Sync() After Sync() After load dir Any way to fix it?
I try debug again, so try add list files functions in LoadDir(), that manual added function able successful list files. So, can @Oblivion or someone else help to fix issue of if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)){ ? Thanks. void SFtpBrowser::LoadDir() { if(browser->InProgress()) return; list.Clear(); DLOG("Before SFtpFileSystemInfo sfsi(*browser);"); SFtpFileSystemInfo sfsi(*browser); DLOG("After SFtpFileSystemInfo sfsi(*browser);"); DLOG("Start list folder"); DLOG(GetWorkdir()); SFtp::DirList ls; // Get a remote dir listing. if(!browser->ListDir(GetWorkdir(), ls)) { DLOG(browser->GetErrorDesc()); return; } // Filter the dir list. auto files = FilterRange(ls, [](const SFtp::DirEntry& e) { return e.IsFile() && e.GetSize() <= 65536; }); // Download the files, using worker threads in parallel. CoFor(min(files.GetCount(), 1000), [&](int i){ DLOG(files[i]); }); DLOG("Before If"); if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)){ DLOG("Before sort"); sortbyext ? SortByExt(list) : SortByName(list); DLOG("After sort"); } DLOG("After If"); DLOG("Before Summary()"); Summary(); DLOG("After Summary(), and before Sync()"); Sync(); DLOG("After Sync()"); } LOGs: SSH: Session, oid: 1: Using Upp's memory managers. SSH: Session, oid: 1: Session successfully initialized. SSH: Session, oid: 1: Compression is disabled. SSH: Session, oid: 1: Handshake successful. SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive] SSH: Session, oid: 1: Client succesfully authenticated. SSH: SFtp, oid: 2: Session successfully initialized. SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: / /msp/ /msp/ /msp/ Before work dir After work dir, and before LoadDir Before SFtpFileSystemInfo sfsi(*browser); After SFtpFileSystemInfo sfsi(*browser); Start list folder /msp/ SSH: SFtp, oid: 2: Directory '/msp/' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (4 entries) SSH: SFtp, oid: 2: File handle freed. -rw-r--r-- 30015 1485 411 12/19/2024 01:24:12 12/19/2024 01:24:12 MSP~some file name.csv Before If SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file After If Before Summary() After Summary(), and before Sync() After Sync() After load dir
Hello Scott, I will try to fix the SftpBrowser. However, Could you try reference/SFtpFileSel example on your SSH server (see if it shows directory and file structures of your server correctly), I want to be sure that problem is in SFtpBrowser code, not elsewhere. Best regards, Oblivion
Hello Oblivion, I try do some minor adjustment to testing the last sample: String GetWorkdir() const { String workDir = dir.GetData().ToString(); if(workDir == "/") return Null; else return workDir; } Which intend set default as empty instead of "/", at least it work for 1st top layer. (Still some bug, such as it would show as folder icon if I upload a test.txt file) But when navigate to sub folder, it will not able correct LoadDir() again. Then I try DLOG(GetWorkdir()); SFtp::DirList ls; // Get a remote dir listing. if(!browser->ListDir(GetWorkdir(), ls)) { DLOG(browser->GetErrorDesc()); return; } // Filter the dir list. //auto files = FilterRange(ls, [](const SFtp::DirEntry& e) { return e.IsFile() && e.GetSize() <= 65536; }); auto files = FilterRange(ls, [](const SFtp::DirEntry& e) { return true; }); // Download the files, using worker threads in parallel. CoFor(min(files.GetCount(), 1000), [&](int i){ DLOG(files[i]); }); by myself, it able list data correctly. So, may be you can update your sample to use SFtp::DirList way? And then I try testing sfsi, since we know that portion have issue: DLOG("Start for sfsi.Find"); Array<FileSystemInfo::FileInfo> root = sfsi.Find("*"); for(int i = 0; i < root.GetCount(); i++){ DLOG(root[i].filename); //DLOG(root[i].root_style); //DLOG(root[i].root_desc); } DLOG("End for sfsi.Find"); It is work at 1st time when GetWordir() return Null, but failed when I select sub folder from dir dropdownlist. Note, the SFTP server itself is Posix, since sfsi.IsPosix() return true, while my working computer is Windows. So, I think the issue may be in Load() function in FileSel.cpp since at least work when if(dir.IsEmpty()) . Array<FileSystemInfo::FileInfo> ffi = filesystem.Find(AppendFileName(dir, filesystem.IsWin32() ? "*.*" : "*")); And then I try reference/SFtpFileSel example per your suggestion for testing, the result as below:
Hello Scott, Thank you for your patience and the info. Quote: Note, the SFTP server itself is Posix, since sfsi.IsPosix() return true, while my working computer is Windows. That's because in most systems, SSH2 is configured to run using POSIX style file paths. Usually, unless you have a niche SSH server implementation, it should be ok. E.g. I run OpenSSH on my windows machine and connect to windows machines with OpenSSH implementation, and the example works fine on them. By the way, what is your ssh configuration? (not any personal info or paths, just the server's general config) I am going to see what is actually going on, and fix and improve the example but it would be nice to replicate a similar problem on my machines. P.s: In my windows setups, the directory/file can be created or uploaded. However, the file list is not updated/refreshed after the operation. It is missing in the code. So that's one bug to fix Best regads, Oblivion
Hello Oblivion, Thanks for commit would make some time to fix bug. And it is not easy since not able duplicate similar env in your computer. The SFTP server I try to connect is setup by my company IT team(A big company), I do not have enough info of it. Should be shared to multiple people. But does able smoothly connect and list/download/upload data through other 3rd party software, such as RealFTP, WinSCP. I also able use PHP to manipulate that SFTP server data. The ultimatepp SFTP function actually should be work too, such as able SFtp::DirList and so on so forth. And it work well if connect to your demo SFTP server. But need you help deep dive below code, such as Load() and sfsi portion, it must have something not consider well. if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)) sortbyext ? SortByExt(list) : SortByName(list);
Hello Scott, Sure, I'll change the directory enumaration logic (removing the sfsi, and using direct calls) and update the example for testing later today. Best regards, Oblivion
Hello Scott, I've tried to improve the example. You can find it attached. It now reloads the directory after all actions and uses direct calls to load directory entries. See the LoadDir() method for changes and some customization possibilities. Please check. Best regards, Oblivion
Hi Oblivion, Yeah, it work as expected if under DEBUG build mode after update LoadDir() as below: Able list/download/upload file without issues. Note: As for build output mode, I select All static and Default debug info leve = None, and add below additional line: f.hidden = false;//add this sentence, or else summary function would not count it. void SFtpBrowser::LoadDir() { if(browser->InProgress()) return; list.Clear(); SFtp::DirList ls; if(browser->ListDir(GetWorkdir(), ls)) { for(const SFtp::DirEntry& e : ls) { String s = e.GetName(); if(s == "." || s == "..") continue; FileList::File f; // Can be easily costumized... f.name = s; f.time = e.GetLastModified(); f.font = StdFont(); f.ink = SColorText; f.icon = e.IsFile() ? CtrlImg::File() : CtrlImg::Dir(); f.isdir = e.IsDirectory(); f.length = e.GetSize(); f.extink = SLtBlue; f.hidden = false;//Scott: add this sentence, or else summary function would not count it. list.Add(f); } sortbyext ? SortByExt(list) : SortByName(list); } Summary(); Sync(); } But, there still have bug if under Release build mode: which is not able upload file, always error and end with 0 byte. LOG: SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (2 entries) SSH: SFtp, oid: 2: File handle freed. //Scott: I add 'RLOG(Format("dest: %s, UnixPath: %s ",dest,UnixPath(dest)));' in upload function for debug since assume may be path error. dest: //msp/config/temp\test.txt, UnixPath: //msp/config/temp/test.txt SSH: SFtp, oid: 3: Session successfully initialized. SSH: SFtp, oid: 3: File '//msp/config/temp/test.txt' is successfully opened. SSH: SFtp, oid: 3: File attributes successfully retrieved. SSH: SFtp, oid: 3: EOF received. SSH: SFtp, oid: 3: Stream write error. SSH: SFtp, oid: 3: File handle freed. SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (3 entries) SSH: SFtp, oid: 2: File handle freed. SSH: SFtp, oid: 3: Session deinitalized. SSH: SFtp, oid: 2: File '//msp/config/temp/test.txt' is successfully deleted. SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (2 entries) SSH: SFtp, oid: 2: File handle freed. So, please help on below: 1. Why debug mode able upload file without issue while release mode encounter EOF and create 0 byte file? 2. It will be show SFTP protocal error in pop up alert if intend delete a none empty folder. (in debug log file it show:SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error) So, suggest show frendly alert to say the folder not empty and cannot delete, instead of execute and then failed. 3. Do you know why before old code is wrong? Please better also find out root cause of sfsi or Upp::Load function too. Thanks. if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)) sortbyext ? SortByExt(list) : SortByName(list);
Hello Scott, In order to see what's wrong with writing a file and isolate it (Unfortunately I can't reproduce the error), could you replace the below method. Use a small file to test uploading please. void SFtpBrowser::Transfer(int opcode, const String src, const String& dest) { SFtp worker(session); Progress pi(this, src); worker.WhenProgress = [&pi](int64 done, int64 total) { pi.SetText( Format( t_("%1:s of %2:s is transferred"), FormatFileSize(done), FormatFileSize(total) ) ); return pi.SetCanceled(int(done), int(total)); }; pi.Create(); switch(opcode) { case GET: { pi.Title(t_("Downloading ") << GetFileName(src)); FileOut fout(dest); if(fout && !worker.LoadFile(fout, src)) ErrorOK(DeQtf(worker.GetErrorDesc())); break; } case PUT: { pi.Title(t_("Uploading ") << GetFileName(src)); // Not for use, only for testing String s = LoadFile(src); if(auto h = worker.OpenWrite(dest); h) { int n = worker.Put(h, ~s, s.GetLength()); RLOG(String() << "Uploaded: " << n << ", filesize: " << s.GetLength()); RDUMP(worker.IsError()); if(worker.IsError()) ErrorOK(DeQtf(worker.GetErrorDesc())); worker.Close(h); } else RLOG("Couldn't open file: " << src << ", error: " << worker.GetErrorDesc()); break; } } LoadDir(); } Best regards, Oblivion
Hi Oblivion, It worked without error in Release build mode with below new code: This new method try write src into a String , and then write String to dest, so no issue of small/middle files, but would losing visibility of progressing % during upload. Anyway to load by chunk and show progressing under new code? Thanks. case PUT: { pi.Title(t_("Uploading ") << GetFileName(src)); // Not for use, only for testing String s = LoadFile(src); if(auto h = worker.OpenWrite(dest); h) { int n = worker.Put(h, ~s, s.GetLength()); RLOG(String() << "Uploaded: " << n << ", filesize: " << s.GetLength()); RDUMP(worker.IsError()); if(worker.IsError()) ErrorOK(DeQtf(worker.GetErrorDesc())); worker.Close(h); } else RLOG("Couldn't open file: " << src << ", error: " << worker.GetErrorDesc()); break; } old wrong code is: case PUT: { pi.Title(t_("Uploading ") << GetFileName(src)); FileIn fin(src); if(fin && !worker.SaveFile(dest, fin)) ErrorOK(DeQtf(worker.GetErrorDesc())); break; } Note: The most tricky thing is old code failed upload file when build by CLANG RELEASE & DEBUG MODE or VS2022 RELEASE MODE. Only work in VS2022 DEBUG build mode. Do you able point out the key gap of the old code and try fix it? It is better have upload progressing bar too:) Thanks. Or else, I have to use VS2022 DEBUG BUILD,then exe size is big. Or use your new code under RELEASE BUILD, but losing upload progressing progress. Thanks. Regards, Scott Huang
Hello Scott, Thanks, that's what I suspected. I will upload a patched version -functionally identical to originally intended code- with some other improvements, likely later today. Progress display will be back, it is disabled temporarily. Best regards, Oblivion
Hello Oblivion, Good to know you will enhance code again to add progressing %. Thnaks. In the meanwhile, I also try upgrade the old code and testing by myself with below update: case PUT: { pi.Title(t_("Uploading ") << GetFileName(src)); FileIn fin(src); /* if(fin && !worker.SaveFile(dest, fin)) ErrorOK(DeQtf(worker.GetErrorDesc())); break; */ SFtpFileOut fout(worker, dest); int64 size = fin.GetSize(), count = 0; int64 chunkSize = 1024 * 64; Buffer<byte> chunk(chunkSize, 0); Gate<int64, int64> WhenProgress; WhenProgress(0, size); while(!fin.IsEof()) { RLOG(Format("size %d, count %d, chunkSize %d, min %d", size, count, chunkSize, (int)min<int64>(size - count, chunkSize) )); int n = fin.Get(chunk, (int) min<int64>(size - count, chunkSize)); RLOG(Format("n %d",n)); if(n > 0) { fout.Put(chunk, n); if(fout.IsError()) { RLOG("Stream write error. " + fin.GetErrorText() + " out stream: "+fout.GetErrorText()); ErrorOK(DeQtf(worker.GetErrorDesc())+"Stream write error. " + fin.GetErrorText() + " out stream: "+fout.GetErrorText()); break; } count += n; if(WhenProgress(count, size)) { break; } } if(fin.IsError()) { RLOG("Stream read error. " + fin.GetErrorText()); ErrorOK(DeQtf(worker.GetErrorDesc())+"Stream read error. " + fin.GetErrorText()); break; } } } The LOG: SSH: SFtp, oid: 2: File handle freed. dest: /\test.txt, UnixPath: //test.txt SSH: SFtp, oid: 3: Session successfully initialized. SSH: SFtp, oid: 3: File '//test.txt' is successfully opened. SSH: SFtp, oid: 3: File attributes successfully retrieved. size 47, count 0, chunkSize 65536, min 47 n 47 SSH: SFtp, oid: 3: EOF received. Stream write error. out stream: A non-blocking socket operation could not be completed immediately. SSH: SFtp, oid: 3: File handle freed. SSH: SFtp, oid: 2: Directory '/' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (7 entries) SSH: SFtp, oid: 2: File handle freed. SSH: SFtp, oid: 3: Session deinitalized. May be you can checking why EOF received? Is it caused by non-blocking socket & blocking socket different if compare with your SFTP and my company's SFTP server? Thanks. Regards, Scott Huang
Quote: May be you can checking why EOF received? Is it caused by non-blocking socket & blocking socket different if compare with your SFTP and my company's SFTP server? I have a suspicion, so I'll try to come up with a fix, but can you tell me the the size of the files that are failed to be uploaded? Do you encounter the same problem with file size >= 64K? Best regards, Oblivion
Hi Oblivion, Merry Christmas! Actually I testing different size files before last post, including 1M, all failed and create 0 size file in SFTP server. SFtpFileOut should have some bug in case some SFTP server not similar as your demo SFTP server. Anyway, I have upgrade code with your hint as below, it can successful upload files and also show progress %: void SFtpBrowser::Transfer(int opcode, const String src, const String& dest) { SFtp worker(session); Progress pi(this, src); worker.WhenProgress = [&pi](int64 done, int64 total) { pi.SetText(Format(t_("%1:s of %2:s is transferred"), FormatFileSize(done), FormatFileSize(total))); return pi.SetCanceled(int(done), int(total)); }; pi.Create(); switch(opcode) { case GET: { pi.Title(t_("Downloading ") << GetFileName(src)); FileOut fout(dest); if(fout && !worker.LoadFile(fout, src)) ErrorOK(DeQtf(worker.GetErrorDesc())); break; } case PUT: { pi.Title(t_("Uploading ") << GetFileName(src)); // Not for use, only for testing String s = LoadFile(src); if(auto h = worker.OpenWrite(dest); h) { /* int n = worker.Put(h, ~s, s.GetLength()); RLOG(String() << "Uploaded: " << n << ", filesize: " << s.GetLength()); RDUMP(worker.IsError()); if(worker.IsError()) ErrorOK(DeQtf(worker.GetErrorDesc())); worker.Close(h);*/ int64 size = s.GetLength(), count = 0; if(settings.log) RLOG(Format("File size %d", size)); int chunkSize = 1024 * 64, thisTimeSize = 0; worker.WhenProgress(0, size); while(!(count >= size)) { thisTimeSize = (int)min<int64>(size - count, chunkSize); if(settings.log) RLOG(Format("======= size %d, count %d, chunkSize %d, min %d", size, count, chunkSize, thisTimeSize)); int x = worker.Put(h, (~s) + count, thisTimeSize); count += x; if(settings.log) RLOG(String() << "This time should be uploaded: " << thisTimeSize << ", Real uploaded: " << x << ", Current Accumulated Count:" << count << ", total filesize: " << size); // RDUMP(worker.IsError()); if(worker.WhenProgress(count, size)) { break; } if(worker.IsError()) { ErrorOK("Upload failed : " + DeQtf(worker.GetErrorDesc())); break; } } worker.Close(h); break; } else if(settings.log) RLOG("Couldn't open file: " << src << ", error: " << worker.GetErrorDesc()); break; } } LoadDir(); } You can consider this issue fixed. But could you help improve it when you have time: 1. Any better way if compare to above? (I am new for UPP) 2. Enhance it to real multiple threads, above actually is single upload thread, quite slow if compare to 3rd party SFTP software. 3. Why stream way failed and we have switch to one time read file into String and then chunk write into SFTP? Regards, Scott Huang
Hello Scott, Thank you for your patience and feedback. I see no difference between your updated transfer code and SFtp::CopyData, which is used by SFtpFileStream derived class. However, I suspect the server's filesystem does not support truncation, which is set by SFtpFileOut, by default, and it is NOT set by plain SFtp::OpenWrite method. That might be the culprit, newer servers, especially flash-based ones don't support it, so the operation fails. (At least my theory). Can you modify the SFtpStream.cpp file in SSH package and then test the original code with streams? bool SFtpStream::Open(SFtp& sftp_, const char *filename, dword mode, int acm) { if(IsOpen()) Close(); sftp = &sftp_; int iomode = mode & ~SHAREMASK; handle = sftp->Open(filename, iomode == READ ? SFtp::READ : iomode == CREATE ? SFtp::READ|SFtp::WRITE|SFtp::CREATE|SFtp::TRUNCATE // <- remove SFtp::TRUNCATE : SFtp::READ|SFtp::WRITE, acm ); if(handle) { SFtpAttrs attrs; if(!sftp->GetAttrs(handle, attrs)) { sftp->Close(handle); handle = nullptr; return false; } OpenInit(mode, attrs.filesize); } return handle; } I will try to answer your other questions as well. Best regards, Oblivion
Hi Oblivion, I try remove SFtp::TRUNCATE as you said, it did not work, same error. I also testing remove both SFtp::CREATE|SFtp::TRUNCATE from SFtpStream.cpp, then the behavior like below: 1. Failed if SFTP file not existing, it show error: Failed opening remote file. I think it caused by I remove SFtp::CREATE flag too. 2. I try use other SFTP software to upload same file firstly. And then U++ exe can upload file, but it would write content to existing file, double contents. There should be have other subtle gaps if compare to OpenWrite function: Open(path, CREATE | WRITE, IRALL | IWUSR); I personal guess it caused by below code, so add several RLOG for debug: (test.txt file size is 47 byte) if(handle) { RLOG("Have handle, before checking GetAttrs"); SFtpAttrs attrs; if(!sftp->GetAttrs(handle, attrs)) { RLOG("Failed get attrs"); sftp->Close(handle); handle = nullptr; return false; } else{ RLOG("Succes get attrs"); RLOG("===Show filesize"); DUMP(attrs); RLOG(attrs.filesize); RLOG((int64)attrs.filesize); RLOG("----"); } OpenInit(mode, attrs.filesize); } void BlockStream::OpenInit(dword mode, int64 _filesize) { streamsize = _filesize; RLOG(String()<<"++In OpenInit function - DEBUG: file size"<<_filesize); RLOG(_filesize); RLOG(streamsize); RLOG("---------------"); LOG does show attributes retrievedm, so the issue should be in function OpenInit(mode, attrs.filesize); And why additional session created SSH: Session, oid: 1: Starting DNS sequence locally for deft.dell.com:22 SSH: Session, oid: 1: Successfully connected to deft.dell.com:22 SSH: Session, oid: 1: Using Upp's memory managers. SSH: Session, oid: 1: Session successfully initialized. SSH: Session, oid: 1: Compression is disabled. SSH: Session, oid: 1: Handshake successful. SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive] SSH: Session, oid: 1: Client succesfully authenticated. SSH: SFtp, oid: 2: Session successfully initialized. SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: / SSH: SFtp, oid: 2: Directory '/' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (6 entries) SSH: SFtp, oid: 2: File handle freed. SSH: SFtp, oid: 2: Directory '//msp' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (3 entries) SSH: SFtp, oid: 2: File handle freed. SSH: SFtp, oid: 2: Directory '//msp/config' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (171 entries) SSH: SFtp, oid: 2: File handle freed. SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (2 entries) SSH: SFtp, oid: 2: File handle freed. dest: //msp/config/temp\test.txt, UnixPath: //msp/config/temp/test.txt ++In OpenInit function - DEBUG: file size47 47 47 --------------- SSH: SFtp, oid: 3: Session successfully initialized. SSH: SFtp, oid: 3: File '//msp/config/temp/test.txt' is successfully opened. Have handle, before checking GetAttrs SSH: SFtp, oid: 3: File attributes successfully retrieved. Succes get attrs ===Show filesize 839918903936 839918903936 ---- ++In OpenInit function - DEBUG: file size839918903936 839918903936 839918903936 --------------- DEBUG: NOT READ SSH: SFtp, oid: 3: EOF received. SSH: SFtp, oid: 3: Stream write error. SSH: SFtp, oid: 3: File handle freed. SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (3 entries) SSH: SFtp, oid: 2: File handle freed. SSH: SFtp, oid: 3: Session deinitalized. SSH: SFtp, oid: 2: File '//msp/config/temp/test.txt' is successfully deleted. SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (2 entries) SSH: SFtp, oid: 2: File handle freed. Questions: Why addition SSH oid created? Why file size wrong? Anyway, I am ok with my last post to use worker.OpenWrite & worker.Put, it worked. Attach good code here for others reference. Regards, Scott Huang
Hello Scott, Thank you for your help. It would be better for us all if we fix the issue instead of using a workaround, if possible. Given your latest log, it appears that the structure returns an invalid file size for the newly created file (it should be 0). Hopefully, the below patch fixes the issue once and for all: bool SFtpStream::Open(SFtp& sftp_, const char *filename, dword mode, int acm) { if(IsOpen()) Close(); sftp = &sftp_; int iomode = mode & ~SHAREMASK; handle = sftp->Open(filename, iomode == READ ? SFtp::READ : iomode == CREATE ? SFtp::READ|SFtp::WRITE|SFtp::CREATE|SFtp::TRUNCATE : SFtp::READ|SFtp::WRITE, acm ); if(handle) { SFtpAttrs attrs; Zero(attrs); if(iomode != CREATE && !sftp->GetAttrs(handle, attrs)) { sftp->Close(handle); handle = nullptr; return false; } OpenInit(mode, attrs.filesize); // filesize always should be 0 for create|truncate } return handle; } I would be grateful if you could try this patch and report back. Also, I have updated the example (attached). It can now: 1) set chunksize 2) set waitstep 3) set compression 4) allows dropping files onto browser for uploading. As for your other questions: 1. Please keep in mind that SFtpBrowser is a very simple example, demonstrating how to integrate SSH package into a GUI. It is not intended to be a rival for WinSCP or any other commercial app. Of course, SSH package allows you to write such applications. In fact I do have a such professional app and its performance is on par with WinScp, if not faster. To achieve faster transfer speeds with SFtpFileBrowser, you can enable compression and increase the chunksize (preferably to 1 Mib). 2. Again, this is a simple example. However, file partitioning (parallel download of a single, huge file) is possible with SSH package, but it would be a complex app, beyond the scope of this demo. (my app does that too). Best regards, Oblivion
[RESOLVED] Re: SFTPBrowser example build failed
Hello Oblivion, It worked at last In short: 1. Fix LoadDir() issue, replaced by SFtp::DirList 2. Fix filesize (in SFtpStream.cpp open function) issue. if(handle) { SFtpAttrs attrs; Zero(attrs); if(iomode != CREATE && !sftp->GetAttrs(handle, attrs)) { sftp->Close(handle); handle = nullptr; return false; } OpenInit(mode, attrs.filesize); // filesize always should be 0 for create|truncate } And You also add more features: 1) set chunksize 2) set waitstep 3) set compression 4) allows dropping files onto browser for uploading. I do learn a lot from these discussions/update code/do testings Appreciate your professional and kindly support! Thanks. And please add parallel download/upload capability for single big file into this sample if you agree and when you free. Thanks. Btw, before I use Golang Fyne GUI framework and its networking package, one of simple solution for above purpose is create several thread read that big file different portion and do the parallel upload/download. Regards, Scott Huang
Re: [RESOLVED] Re: SFTPBrowser example build failed
Hello Scott, Happy to help. Here's an updated version. This version contains a switch to transfer files using Sftp or Scp. Scp transfers, if supported by the server, can be significantly faster. If this is OK with you, I will update the upp-components package. Then, I'll write a simple example for parallel download (file partitioning). But it will take some time, probably within this week. Best regards, Oblivion
Hello Oblivion, Per testing, your new code also working for Upload. That SFTP server seems support SCP transfer for upload. Thanks. Btw, do SCP mean Secure Copy Protocal, why it would be faster than normal SFTP way? thanks. But SCP way failed for download. (SFTP way good for download if I change setting back) So, split SCP for upload only? and download still use SFTP way, but enhanced to multiple thread when you have time? Thanks. LOG: SSH: Scp, oid: 10: Unable to open file //msp/config/temp/test.txt SSH: Scp, oid: 10: Failed. Code = -28, Failed to recv file SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened. SSH: SFtp, oid: 2: Directory listing is successful. (5 entries) SSH: SFtp, oid: 2: File handle freed. Appreciate your continious enhancements! Take your time of below. Merry Christmas! Quote: Then, I'll write a simple example for parallel download (file partitioning). But it will take some time, probably within this week. Regards, Scott Huang
Hello Scott, Quote: Btw, do SCP mean Secure Copy Protocal, why it would be faster than normal SFTP way? Well, long story short: SCP is different from SFTP in that SFTP has a protocol overhead (it internally chunks and "acks" transfered chunks), so it will be slower on big file transfers. The downside of scp is that it has no transfer/pause/continue mechanism, as it is only a data stream over SSH protocol. Difference is usually visible in huge file transfers. SCP download error: Possibly your server not supporting downloads, and only uploads by SCP? As for the file partitioning: I decided to integrate a dumbed-down version of it (can be selected in the settings) into the SFtpBrowser within a couple of days. Best regards, Oblivion
Hello Oblivion, Likely. Anyway, I adjust settings to able select upload through SCP, but download still SFTP. And several days later would be faster after you enable partitioning downloading. Thanks. Quote: SCP download error: Possibly your server not supporting downloads, and only uploads by SCP? Regards, Scott Huang
Hello Scott, I have attached the latest SFtpBrowser example (with parallel download, namely file partitioning). As I have noted in the code, ideally we would create cosessions only once (on login) and then use them as needed. But that would make the example very complex. In order to reduce the complexity in this example I am using a more direct approach. Keep in mind that it would make no sense to use SFTP parallel download feature with small files. It is useful for downloading large files (yet, in most cases a single threaded SCP tranfer will still be faster) Also, I omitted some checks that would be better implemented in real-life situations. It is up to you to write a better SFtpBrowser. In fact I encourage you to do so. This is what example codes are for. IIRC, this is the first time I am demonstrating the parallel download feature. Best regards, Oblivion
Hello Oblivion, Appreciate you share the tips of filepart/LoadFilePart/CoLoadFile, really helpful. It works only except one bug, we need add 'connected = true' in Connect function after successful connect to SFTP, or else, the mkdir & upload & context would not enabled. Thanks. if(CreateSession(session)) { connected = true;//Add by Scott browser.Attach(new SFtp(session)); basedir = browser->GetDefaultDir(); Workdir(basedir); LoadDir(); } else SessionError(); Regards, Scott Huang