Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Developing U++ » UppHub » FTP class and reference example for U++
Re: FTP class and reference example for U++ [message #47869 is a reply to message #47859] Fri, 14 April 2017 23:45 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

As Klugier suggested above, I've re-written the convenience functions, using a more object-oriented approach.
As a result, Ftp convenience functions, both single-threaded and multithreaded variants, are changed, and much less cluttered and more extendible now.

// Single-threaded.
Ftp::Result FtpGet(const Ftp::Request& request, Gate<int64, int64> progress = false, Event<> whenwait = CNULL);
Ftp::Result FtpPut(const Ftp::Request& request, Gate<int64, int64> progress = false, Event<> whenwait = CNULL);

// Multi-threaded.
int FtpAsyncGet(Ftp::Request& request, Event<Ftp::Result> progress = CNULL);
int FtpAsyncPut(Ftp::Request& request, Event<Ftp::Result> progress = CNULL);        


You can find the updated package in the first post of this topic.

FtpBrowser example and FTP package documentation are also updated to reflect the changes.


Regards.

Oblivion.


Re: FTP class and reference example for U++ [message #47876 is a reply to message #47869] Sat, 15 April 2017 19:23 Go to previous messageGo to next message
Klugier is currently offline  Klugier
Messages: 1075
Registered: September 2012
Location: Poland, Kraków
Senior Contributor
Hello,

It is good to hear you introduced my proposals Smile

_________________________________________

I see one minor things to improvement - you could change the parameter name from whenwait to whenWait it will be more readable.

DirEntry&       User(const String& u)                           { vm(USER) = u; return *this; }

Should it be SetUser(const String& u)? Moreover the u parameter is misleading for the API consumer it should be explicitly called user.

_________________________________________

You should also consider moving your enums to c++ style:
 enum Style      { UNIX, DOS }; 

enum class Type { UNIX, DOS }; // You could consider chaining the name to Type

// The user can refere:
DirEntry::Type::UNIX

instead of:
DirEntry::UNIX

_________________________________________

Do we rely need this friendship declaration here?
        ValueMap        vm;
        Bits            owner, group, other;
        friend bool     ParseFtpDirEntry(const String& in, Vector<Ftp::DirEntry>& out);


Why not just create
const ValueMap& GetVm(); // Read only access - i believe ParseFtpDirEntry doesn't set anything?

IMO the variable name "vm" is also misleading. I would call it dirInfo or something more verbose. The same improvment can be used in Result class - just call it resultInfo.

_________________________________________

(Cosmetic - tiny) The class beginning has got problem with readability. It could be:
class Ftp : private NoCopy {
public: // Just remove extra empty line.
	class Request;
	class Result; // Do we need forward declarations in class? I believe when we remove friendship we will not ;)

        class DirEntry final : Moveable<Ftp::DirEntry> {



Sincerely,
Klugier


U++ - one framework to rule them all.
Re: FTP class and reference example for U++ [message #47877 is a reply to message #47876] Sat, 15 April 2017 19:38 Go to previous messageGo to next message
Klugier is currently offline  Klugier
Messages: 1075
Registered: September 2012
Location: Poland, Kraków
Senior Contributor
Hello,

In my opinion handling multi-threading API using #ifdef in header is not good for the end user. Because, he/seh is obligated to use #ifdef in theire code. Personally, i would hide it inside the interface and use One container (Movable will not work for this class).

class Result // Interface - all methods are abstract
{
public:
   // The same as previous

   virtual bool IsAsync() = 0;

   // Mt methods are presented without #ifdef guard
};

class RegularResult : public Result // The name could be different
{
public:
    bool IsAsync() override { return false; }

    // Mt methods returning falses, -1 etc.
};

class AsyncResult : public Result
{
public:
    bool IsAsync() override { return true; }
    
    // Return correct values for MT
};

One<Result> result(new AsyncResult());



What do you think?

Sincerely,
Klugier


U++ - one framework to rule them all.
Re: FTP class and reference example for U++ [message #47882 is a reply to message #47877] Sun, 16 April 2017 18:28 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Quote:
Hello,

In my opinion handling multi-threading API using #ifdef in header is not good for the end user. Because, he/seh is obligated to use #ifdef in theire code. Personally, i would hide it inside the interface and use One container (Movable will not work for this class).

class Result // Interface - all methods are abstract
{
public:
// The same as previous

virtual bool IsAsync() = 0;

// Mt methods are presented without #ifdef guard
};

class RegularResult : public Result // The name could be different
{
public:
bool IsAsync() override { return false; }

// Mt methods returning falses, -1 etc.
};

class AsyncResult : public Result
{
public:
bool IsAsync() override { return true; }

// Return correct values for MT
};

One<Result> result(new AsyncResult());



What do you think?

Sincerely,
Klugier



Hello Klugier,

Thank you very much for your constructive criticism and suggestions.


I believe we might have a simpler solution. Smile

FtpAsyncGet() and FtpAsyncPut() functions call FtpAsyncIO().It is possible to return en error code if anything goes wrong.
So we can simply define a function, say, IsMultitihreaded(), to check the U++ multi-threading infrastructure automatically, in Ftp.cpp:


static inline bool IsMultithreaded() 
{
#ifdef flagMT
	return true;
#else
	return false;
#endif
}


Then we can call it in FtpAsyncIO:


Ftp::Result FtpAsyncIO(Ftp::Request request, Event<Ftp::Result> progress, bool put)
{
		// Check if U++ MT is enabled.
	Ftp::Result r;
	try {
		if(!IsMultithreaded())
			throw Exc("Multithreading is not enabled.");


// ...


	}
	catch(Exc& e) {
		// Couldn'r create or run the worker thread.
		r.info("state") = "failed";
		r.info("rc")    = -1;
		r.info("msg")   = e;
		LLOG("-- FTP worker failed. Reason: " << e);
	}
	return r;


In this way we can remove other ifdefs, and user won't be confused.
(Multithreading as a requirement is already mentioned in API docs for each method and function. )

Note that I also changed the return value from int to Ftp::Result. (Now they are similar to single-threaded variants)


Calling an async function such as Result::InProgress() now won't do harm, it wil silently return false.

AS for your other suggestions:

ParseFtpDirEntry() actually modifies the ValueMap. It parses the string into key-value pairs. However, now I put the actual parser code into DirEntry::Parser() method, and got rid of the friend declaration.

Same thing goes for the Request class. So, the ugly forward declarations are removed.

I updated the package to reflect the changes...

Regards,

Oblivion





[Updated on: Sun, 16 April 2017 20:01]

Report message to a moderator

Re: FTP class and reference example for U++ [message #47918 is a reply to message #43053] Sun, 23 April 2017 23:41 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

Another round of updates.
This time FtpBrowser example has received updates too. But first things first:

FTP package:

 

multithreaded functions are further improved. 

- Available worker slots are limited to 256 ftp worker thread per process.

- It is now possible to pre-set the thread priority of each worker.

- It is now possible to pass a user data (a U++ Value), using Request::UserData() and Result::GetUserData();

- Worker abort mechanism is both simplified and improved.  Vector is ditched in favor of a static Bits instance.
  This reduces both memory consumption and mutex lock/unlock times. Since, previously we needed to iterate over the vector to check if a worker was registered to be aborted.
  This was time consuming. Now we simply set a bit on or off. Since the worker counter resets at 65536 and max. worker slots are limeted to 256, it should be pretty safe.
  See the FtpAsyncIO(), Ftp::AbortWorker() and Ftp::IsWorkerAborted() for details. (I'm open to suggestions, if you have better ideas.) 

- Made room for NetworkProxy support. I will add NetworkProxy support (HTTP/SOCKS4/SOCKS5) via a preprocessor directive (USEPROXY flag), since NetworkProxy package is currently not in bazaar.



FtpBrowser example:

- It is now possible to multiselect the files to be downloaded or uploaded.

- FtpBrowser no longer asks for download location for each download. A default download location can be selected. And it can be changed via environment settings.

- FtpBrowser gained file editing support. While it is still a barebone example with ability to edit only one file at a time, it will change in the near future.
  (Mirek had suggested this when we were discussing SSH package last year on "Developer's Corner", which is on it's way, by the way.)
  Of course this is far from what he suggested, but it is operational in it's current state anyway. :)



That's it for now. Two features (thread priority settings for ftp worker threads, and a ftp text editor for FtpBrowser example) are this weeks highlights.


You can find the updated FTP package in the first post of this topic.


Cheers!

Oblivion.


[Updated on: Sun, 23 April 2017 23:54]

Report message to a moderator

Re: FTP class and reference example for U++ [message #47925 is a reply to message #43053] Mon, 24 April 2017 19:01 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello

I've made a video, demonstrating the basic capabilities (concurrent downloads of large files [three 4.5 GB files], remote file editing, browsing, etc.) of FTP package and browser.

Note that the example browser is still under development.)

Just select 1080p (HD)

https://vimeo.com/214530673

Cheers!

Oblivion.


[Updated on: Mon, 24 April 2017 19:04]

Report message to a moderator

Re: FTP class and reference example for U++ [message #47929 is a reply to message #47925] Mon, 24 April 2017 22:12 Go to previous messageGo to next message
Klugier is currently offline  Klugier
Messages: 1075
Registered: September 2012
Location: Poland, Kraków
Senior Contributor
Hello,

For me it is a great news that somebody published video related to U++. You could consider posting URL to our site in the movie description Wink

Backing to FTP - did you consider moving to enum class (c++11 feature) instead of old-c like enum. I think the whole API will gained on this change. For example:
class DirEntry {
// ...
enum Style      { UNIX, DOS }; // In my opinion "Type" world is a better alternative in this situation...
enum class Type { UNIX, DOS };


Usage:
if (dirType == DirEntry::Type::UNIX) {
  // .. do something
}


instead of
if (dirType == DirEntry::UNIX) {
   // ...
}


Much more readable. Isn't it?

Let me ask you one question - do you have GitHub repository for that project?

Sincerely,
Klugier


U++ - one framework to rule them all.

[Updated on: Mon, 24 April 2017 22:15]

Report message to a moderator

Re: FTP class and reference example for U++ [message #47932 is a reply to message #47929] Tue, 25 April 2017 08:37 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Quote:
For me it is a great news that somebody published video related to U++. You could consider posting URL to our site in the movie description Wink


Done. Credits go where it's due. Smile




Quote:
Backing to FTP - did you consider moving to enum class (c++11 feature) instead of old-c like enum. I think the whole API will gained on this change. For example:


Ok, I'll change this, it won't hurt much, all it requires is some minor adjustments.


Quote:
// In my opinion "Type" world is a better alternative in this situation...


Well this is more about semantics. According to Wiktionary and Oxford Dictionary, a style is a manner of doing, or presenting things, which is what directory listing in FTP is all about.
It does not necessarily say anything about the underlying file system or structure (FTP servers can and sometimes do have UNIX style dir listing, when they actually use Windows/DOS based file structure (and there are even newer and more detailed listing styles (See: https://tools.ietf.org/html/rfc3659) which I'm planning to support in next versions. That's why I refactored the Ftp::DirEntry class, using a ValueMap. It gives us a tremendous flexibility to support almost any dir listing syle used in ftp servers. Parsing of dir listings in ftp is a PITA, with no reliable or standard format around, by the way Smile )
It is about the way of presenting directory entries (they are same type of objects in an ftp context: both are directory listings). So, I'd prefer it stay as it is.






Quote:
Let me ask you one question - do you have GitHub repository for that project?


I do use and prefer git and github for my C/Python/Perl code, but I prefer using SVN for my U++ related projects, for it is supported by TheIDE. Smile
In 2015 I'd created a sourceforge SVN repo, named Ultimate++ Components (of course, with a disclaimer that the repository and the code it contains is not endorsed by
U++ in any way, and that it is simply an unofficial project to externally support the U++ framework.)
Initially it saw some activity, but I had to focus on my other projects and my job, so the repo lied dormant for a long time.
Not anymore Smile I revived it some time ago, and restructuring it now.
I'm uploading the development versions of the components I develop and use, with BSD license.

https://sourceforge.net/projects/ultimatecomponents/


Regards,

Oblivion


[Updated on: Tue, 25 April 2017 09:23]

Report message to a moderator

Re: FTP class and reference example for U++ [message #48274 is a reply to message #43053] Wed, 14 June 2017 00:06 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

FTP package is updated.
With this new version, Ftp class finally gained transfer restart (resume) capability for both (binary) uploads and downloads. Smile

The update is also reflected in the ftp browser example.
FTP browser example can resume/restart failed or aborted transfers (if the restart feature is supported by the connected server, of course.)
Also the ftp browser now allows drag and drop (for uploads).

Also, you'll notice two nested packages within the FtpBrowser example: BrowserCtrl and TrackerCtrl.
These widgets, which I'm preparing to publish as seperate packages, are resuable ctrls.

BrowserCtrl
is very similar to FileSel, but is embeddable and provides a simple api for a flexible, file system transparent file browser.
It is under heavy development.

TrackerCtrl is a transfer tracker. It has a generic api which can be used for tracking different type of file transfers (local file copy, SFTP, FTP, WEBDAV etc.) at once.
It is under heavy development too.

Which brings us to the next step: SSH (SFTP) package. Finally I have time to finish the technical preview version of SSH package.
I'll publish it within couple of weeks. And turn the FtpBrowser into a profile-based multibrowser capable of handling Local files, FTP, SFTP, WEBDAV (my next goal), etc.

Any suggestions, criticism, bug reports are always welcome.

Updated package can be found in the first post of this topic.

index.php?t=getfile&id=5311&private=0


Tested on Arch Linux (kernel: 4 11.3) and Windows 7/10

Cheers!

Oblivion.


[Updated on: Wed, 14 June 2017 00:13]

Report message to a moderator

Re: FTP class and reference example for U++ [message #48295 is a reply to message #48274] Fri, 16 June 2017 10:23 Go to previous messageGo to next message
Tom1
Messages: 1212
Registered: March 2007
Senior Contributor
Hi Oblivion,

This looks nice!

I'm looking forward to see the SFTP part... Smile

Best regards,

Tom
Re: FTP class and reference example for U++ [message #48524 is a reply to message #43053] Thu, 13 July 2017 17:47 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

FTP package is updated to version 1.2.1.
This version fixes a bug in the restart/resume mechanism.

FtpBrowser is not updated yet.

You can find the package in the first post of this topic.

Best regards,
Oblivion.


Re: FTP class and reference example for U++ [message #49680 is a reply to message #43053] Mon, 02 April 2018 00:17 Go to previous message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

It's been a long while since the last update.
A new and major release of FTP class is finally here: Version 2.0.0

New version is far superior to the older (v 1.2) version of FTP package. Basically it is simpler yet more powerful.
API-wise, although I've tried hard to keep the old interface intact, and partially succeded in this, some breakages were necessary.
Yet I will also maintain the old code for some time.

I had to redesign the Ftp class to let it take advantage of the non-blocking interface I used in SSH pacakge. Now they are almost identical. Smile

Here are the major changes:
MAJOR RELEASE: Ftp package, version 2.0.0
               Necessary API breakage.
               Ftp is redesigned, using an effective, and simple queue-model similar to the SSH package's.             
               Old multithreading code completely ditched in favor of the new AsyncWork based ftp workers.
               This resulted in noticable performance gain, and simplicity.
               Non-blocking mode is finally implemented. Ftp class can do non-blocking calls easily.
               FTPS mode improved. It is now possible to inspect the SSL information after handshake.
               Support for user-defined consumer functions (for incoming data) is added.
               A simple ftp URL scheme is implemented. This scheme replaces (partially) the old Ftp::Request class.
               UTF-8 path name encoding support is added.
               Optional Connect/Login() methods pair added. It is now possible to execute commands before login.
               Ftp::DirEntry now uses pcre to parse listing strings.
               Logging is improved.
               Reference examples are added to the package.


Reference examples provided with the package:

- FtpGet:           Demonstrates a basic FTP file download in blocking mode.
- FtpGetNB:         Demonstrates a basic FTP file download in non-blocking mode.
- FtpGetMT:         Demonstrates a basic Ftp file download, using worker threads.
- FtpMultiGetMT:    Demonstrates FTP dir listing and concurrent file transfers, using worker threads.
- FtpGUI:           Demonstrates a basic FTP browser with GUI (with upload, download, mkdir, rename, delete commands).
- FtpOverTLS:       Demonstrates the secure connection capability of FTP package in blocking mode.
- FtpQueryFeatures: Demonstrates the feature query mechanism, as defined in RFC 2389
- FtpRawCommand:    Demonstrates FTP raw command execution in blocking mode.


As usual, the updated package can be found in the first message of this topic.
And you can also grab it from:

Latest code is located at: https://github.com/ismail-yilmaz/upp-components/tree/master/ Core/FTP
Reference examples can be found at: https://github.com/ismail-yilmaz/upp-components/tree/master/ Examples
Older version (1.2) can be found at: https://github.com/ismail-yilmaz/upp-components/tree/master/ Attic/FTP

A screenshot of the simple FtpGUI example (GUI is inspired by the qt's example):

index.php?t=getfile&id=5535&private=0

Suggestions, bug reports, patches, etc. are appreciated.

Best regards,
Oblivion
  • Attachment: FtpGUI.png
    (Size: 138.14KB, Downloaded 486 times)


[Updated on: Mon, 02 April 2018 13:38]

Report message to a moderator

Previous Topic: Am I doing something wrong, or is there a bug? FIREBIRD
Next Topic: OpenCV and openCV_Demo copiling error
Goto Forum:
  


Current Time: Fri Mar 29 15:03:54 CET 2024

Total time taken to generate the page: 0.02081 seconds