Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Developing U++ » UppHub » FTP class and reference example for U++
FTP class and reference example for U++ [message #43053] Sun, 27 April 2014 18:11 Go to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

Ftp class is a U++ client side implementation of the File Transfer Protocol (FTP) as specified in RFC 959, with several advanced capabilities:

- Time-constrained blocking, and non blocking operation modes.
- Multithreaded file transfers, using worker threads.
- IPv6 connections and NATs, as specified in RFC 2428.
- Ftp over TLS/SSL (FTPS), as specified in RFC 2228.
- Feature negotiation mechanism as specifien in RFC 2389.
- Parsing of UNIX and DOS style directory listings.
- Extending the existing functionality of Ftp class.
- Transfer restart/resume mechanism, as specified in RFC 3959.
- UTF-8 encoded path names.


Reference examples provided with the package:

- FtpGet:           Demonstrates a basic FTP file download in blocking mode.
- FtpGetNB:         Demonstrates a basic FTP file download in non-blocking mode.
- FtpGetMT:         Demonstrates a basic Ftp file download, using worker threads.
- FtpMultiGetMT:    Demonstrates FTP dir listing and concurrent file transfers, using worker threads.
- FtpGUI:           Demonstrates a basic FTP browser with GUI (with upload, download, mkdir, rename, delete commands).
- FtpOverTLS:       Demonstrates the secure connection capability of FTP package in blocking mode.
- FtpQueryFeatures: Demonstrates the feature query mechanism, as defined in RFC 2389
- FtpRawCommand:    Demonstrates FTP raw command execution in blocking mode.



Latest code is located at: https://github.com/ismail-yilmaz/upp-components/tree/master/ Core/FTP
Reference examples can be found at: https://github.com/ismail-yilmaz/upp-components/tree/master/ Examples
Older version (1.2) can be found at: https://github.com/ismail-yilmaz/upp-components/tree/master/ Attic/FTP

A video demonstration of Ftp browser: (Note that the example browser is still under development.)

https://vimeo.com/214530673

Suggestions, bug reports, patches, criticism are always welcome.

Regards,
Oblivion



[Updated on: Mon, 02 April 2018 14:29]

Report message to a moderator

Re: FTP class and reference example for U++ [message #43104 is a reply to message #43053] Wed, 07 May 2014 01:28 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello guys,

Today I publish the first public beta of the FTP package (Link to the updated versions of the package can be found in the first post).

What is missing: Aside from the todo list in the first post, currently only GetReplyAsXml() method is missing,. but it will be added within this week. Also, I removed most of the internal error messages, I'll add them later.
I also wrote a FtpBrowser with a non-blocking GUI (not multithreaded), which will demonstrate how easy it is to set up a non-blcoking gui with Ftp and on U++.. It will be added as a reference example.
But not uploaded this time (code needs cleaning). Ftp class is also a demonstartion of TcpSocket based Server and Client sockets, since it actively uses both type of sockets.

And of course there will be bugs. I am sure because Ftp protocol is, by itself, a total mess. Smile

So, suggestions, bug reports, patches are always welcome.

Ftp class is tested under Linux 3.12+ and Windows (XP/7).

Cheers!


[Updated on: Wed, 07 May 2014 01:34]

Report message to a moderator

Re: FTP class and reference example for U++ [message #43115 is a reply to message #43053] Thu, 08 May 2014 11:52 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor

Update: NOOP command is added to the Ftp class. (See the first post for package), .

Regards.


Re: FTP class and reference example for U++ [message #43127 is a reply to message #43053] Tue, 13 May 2014 02:03 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello guys,

I updated the Ftp class and reference examples package (see the first post for package).
Reference examples package now contains a very simple (for demonstration only) Ftp Browser with non-blocking GUI.
This version of the example allows only one download at a time, but a multithreaded example which will allow multiple downloads, will follow. (It will be very similar to the U++ Reference/GuiMT example),
FtpBrowser utilizes two ftp objects (a browser and a worker). This makes possible to navigate while download. Also it demonstrates how to use FtpDirEntry class in conjunction with the FileList class.
Here is the screenshot of the ftp browser example.

index.php?t=getfile&id=4536&private=0

Bug reports, suggestions, requests, etc. are always welcome.


Regards.


[Updated on: Tue, 13 May 2014 02:06]

Report message to a moderator

Re: FTP class and reference example for U++ [message #43130 is a reply to message #43127] Wed, 14 May 2014 09:25 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
There already exists plugin/ftp. Could you please summarize advantages of your new implementation (before I will dwell into it myself...)?
Re: FTP class and reference example for U++ [message #43132 is a reply to message #43130] Wed, 14 May 2014 13:43 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello Mirek,

While it is far from perfect, and I'm not sure if the below mentioned features count as advantages, aside from the basic operations, but here they are:

1) Ftp class is completely based on native Upp classes. It is derived from TcpSocket. Therefore supports its features (except asynchronous connections, due to FTP protocols synchronic nature).
This makes bug tracking and understanding the source code easy (also ftp class source code is around 600 lines, and meant to be clean and concise). No external lib, no C code, no need for external memory management, etc..
2) It comes with a directory entry and file information parser (FtpDirEntry class) which makes parsing (both DOS and UNIX based directory listing) as simple as it can be (of course getting the raw Listing strings is also possible (see CLI based ftp example).
3) It is designed the U++ GUI in mind; For example, FtpDirEntry class perfectly fits with FileList class (see the supplied ftp browser example). and achieving a non-blocking gui is very easy.
4) Ftp class does work on U++ streams (any kind), not simply strings (so it can take advantage of FileIn or FileOut streams, without any other effort, which means reduced code).
5) Ftp class api can be extended via SendCommand method (see CLI based ftp example) by the user/developer (there are many useful FTP extension commands out there, added later).
6) Ftp class is not just based on RFC 959. It also takes into account some practical deviations (D. J. Bernstein's recommendations: http://cr.yp.to/ftp.html) from the original draft.
7) Proper support for aborting transfers.
8 ) Ftp class will support IPV6 (actually It already does in development branch, and I will merge it soon).
9) Ftp class will support proxies.
10) I am also planning to add SSL/TLS support.
11) It is documented. (API document is complete, and I am currently writing a document on its mode of operations).
12) It is actively developed.


I believe the code is in good shape (after all, it is in beta state), but there is always plenty of room for improvement, and bugs that I don't know of yet.

Regards.


[Updated on: Wed, 14 May 2014 14:15]

Report message to a moderator

Re: FTP class and reference example for U++ [message #44089 is a reply to message #43053] Wed, 24 December 2014 12:06 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello guys,

Ftp class is updated (rewritten, actually). It now takes advantage of EPRT and EPSV commands, which means it has from now on IPv6 support.

See first post for details and package.

Regards,
Oblivion.


Re: FTP class and reference example for U++ [message #44114 is a reply to message #43053] Thu, 25 December 2014 18:03 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,


Two convenience functions are added to the Ftp package: FtpGet and FtpPut:
int FtpGet(const String& path, Stream& out, const String& host, int port = 21, const String& user = Null, const String& pass = Null, 
	Gate2<int64, int64> progress = false, Callback whenwait = CNULL, int type = Ftp::BINARY, int mode = Ftp::PASSIVE);

int FtpPut(Stream& in, const String& path, const String& host, int port = 21, const String& user = Null, const String& pass = Null, 
	Gate2<int64, int64> progress = false, Callback whenwait = CNULL, int type = Ftp::BINARY, int mode = Ftp::PASSIVE);


These functions can simplify Ftp download and upload operations, while retaining the flexibility of the Ftp class. Ftp browser example is also updated to reflect this addition.

Regards,
Oblivion.


[Updated on: Thu, 25 December 2014 18:08]

Report message to a moderator

Re: FTP class and reference example for U++ [message #44146 is a reply to message #43053] Wed, 07 January 2015 20:52 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

Ftp class gained basic FTPS (Ftp over SSL, to be specific) support. From now on Ftp class can connect to secure Ftp servers, using "explicit mode" negotiation. See RFC 2228 for details.

There are some limitations though:

1) Currently only SSL is supported. TLS is not yet supported.
2) FTPS can be only activated per ftp session. Namely, it must be enabled or disabled prior to an Ftp::Connect() call.
3) Active connection mode and FTPS are mutually exclusive. When FTPS is enabled, the ftp data retrieval methods (such as Ftp::ListDir(), Ftp::Put() or Ftp::Get(), or convenience funtions FtpPut() and FtpGet()) will fail in active connection (PORT, EPRT) mode.

Bu reports, criticism, patches, etc. are welcome.

Regards,
Oblivion.


Re: FTP class and reference example for U++ [message #44188 is a reply to message #43053] Tue, 20 January 2015 01:17 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

It seems that U++ Core/SSL package already supports TLS connections (see: http://www.ultimatepp.org/forums/index.php?t=msg&th=9090 &goto=43822&#msg_43822 and https://www.openssl.org/docs/ssl/SSL_CTX_new.html).
So I added AUTH TLS support to the Ftp class.
I also tested this new feature with a secure ftp server which requires TLS and it apparently works well.

Ftp class will first explicity request a TLS mode and in case of rejection it will try SSL mode.
See first post for the details and the updated package.


Suggestions, bug reports, patches, criticism are always welcome.

Regards,
Oblivion.


[Updated on: Tue, 20 January 2015 01:25]

Report message to a moderator

Re: FTP class and reference example for U++ [message #44843 is a reply to message #44188] Tue, 07 July 2015 19:40 Go to previous messageGo to next message
unodgs is currently offline  unodgs
Messages: 1366
Registered: November 2005
Location: Poland
Ultimate Contributor

Hi Oblivion!

Great package. It works well on windows as well but do you plan to add SFTP (SSH File Transfer Protocol)?
PS: I would be grateful for adding upload in your example.

[Updated on: Tue, 07 July 2015 19:43]

Report message to a moderator

Re: FTP class and reference example for U++ [message #44846 is a reply to message #44843] Wed, 08 July 2015 08:50 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello Daniel!

Glad to hear that someone found it useful.

Sure. The example is rudimentary.
I will add upload/delete/rename/create etc. options to the example. And turn it into a usable FTP browser in the following days.


SFTP, on the other hand, needs some think-through. I had thought of adding a SSH client library to U++ last year, but I did not have time to implement it then. (it is a somewhat complex protocol, and I don't want to end up creating something half-baked.)
IMO the best and easiest option is to create a U++ wrapper for libssh2 which is AFAIK cross-platform.
So, I will have spare time in the following days and I can start implementing a SFTP client utilizing all the U++ goodies and having usual U++ type API, based on libssh2.
In the meantime maybe I will also resurrect my U++ SSH client library project (But I'm not making any promises on this. Smile ).

What do you think?


Regards,

Oblivion


[Updated on: Wed, 08 July 2015 09:05]

Report message to a moderator

Re: FTP class and reference example for U++ [message #44849 is a reply to message #44846] Wed, 08 July 2015 10:20 Go to previous messageGo to next message
unodgs is currently offline  unodgs
Messages: 1366
Registered: November 2005
Location: Poland
Ultimate Contributor

For me sounds like a great news. Thank you Oblivion in advance!
PS: libssh2 looks like a good choice (pure C, easy to wrap up)
Re: FTP class and reference example for U++ [message #47752 is a reply to message #43053] Sat, 18 March 2017 23:28 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello guys,

It's time for some spring cleaning. Smile

Ftp class is updated.

From now on it requires at least C++11.

And GetReplyAsXml() method is added. This method simplifies the parsing of FTP protocol messages and internal messages, using simple XML tags.


FtpExample is also updated so that it can work. I know it is a crude example but I'm in the process of writing a better version of it with multithreading in mind.
It'll be replaced asap.

As usual, you can find the updated package in the first post of this topic.

Regards,

Oblivion.


[Updated on: Sat, 18 March 2017 23:31]

Report message to a moderator

Re: FTP class and reference example for U++ [message #47833 is a reply to message #43053] Sun, 09 April 2017 02:32 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello,

I'll deliver on my promise and upload the new ftp browser example, along with the updated FTP package within next week.
As you can see from the below screenshot, the new browser example is quite capable.
(The funny thing is, it contains less than 400 LoC (subject to change, but not much) and most of the code is about GUI creation and management.

Yet it supports and demonstrates almost every aspect of the FTP package, including multithreading. Smile

Yep, FTP package finally gained multithreaded file transfer support. Cool

I have to say that it is very simple and easy to use. In fact the multithreaded DL/UL functions (FtpAsyncGet and FtpAsyncPut) are almost identical to their single-threaded counterparts, and allow the same flexibility and integration with the Upp's core and Ctrls!

Therefore, the upcoming release will mark the version 1.0 Smile


Cheers!

Oblivion

index.php?t=getfile&id=5222&private=0


[Updated on: Wed, 12 April 2017 23:12] by Moderator

Report message to a moderator

Re: FTP class and reference example for U++ [message #47858 is a reply to message #43130] Wed, 12 April 2017 22:34 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello guys,

Today marks a new milestone for FTP package. The new version (1.1) of FTP package is finally here.
After several iterations, I believe the package has become stable.
This version brings multithreaded file transfers to the ftp class, via two convenience functions: FtpAsyncGet() and FtpAsyncPut():


int FtpAsyncGet(const String& remote_file, const String& local_file, const String& host, int port = 21, const String& user = Null,
        const String& pass = Null, Event<int, int, String, int64, int64> progress = CNULL, bool active = false, bool ssl = false, 
        bool ascii = false);

int FtpAsyncPut(const String& local_file, const String& remote_file, const String& host, int port = 21, const String& user = Null,
        const String& pass = Null, Event<int, int, String, int64, int64> progress = CNULL, bool active = false, bool ssl = false, 
        bool ascii = false);


As you can see the syntax is very similar to the single-threaded FtpGet() and FtpPut() functions. Except these async functions do concurrent upload/download while retaining the flexibility of the Ftp class.
Moreover, these functions are implemented in a general way that, I believe, they can serve also as a reference implementation. While the Ftp class itself is still single threaded.

Also, in this version, the FEAT command is implemented. It is now possible to query the capabilities of ftp servers easily.


One more addition to the package is a multithreaded simple FTP browser example.
Example browser can use secure connection (FTPS), download, upload, manipulate files, and manipulate directories, show file information etc.

First on my todo list is to implement REST command to allow restart/resume of interrupted transfers.



And Mirek, I know that you are a very busy person, but you'd asked me about the advantages of my ftp implementation over the current one in Core/Ftp.
I now believe FTP package is eligible to Bazaar, considering that it has IPV6, FTPS, multithreading support, better integration with U++ core and UI classes, a clean code-base, active development, is well documented, and easily extendible (using SendCommand() method), and has an example browser built around it. It makes a good alternative. What do you think?


As usual, you can find the development log and updated package in the first post of this topic.

I tested FTP package with several public FTP servers, and private servers I set up.
I can recommend,

ftp.uni-erlangen.de, User: Anonymous, Password: Anonymous.
test.rebex.net,      User: demo, Password: password. Allows FTPS
demo.wftpserver.com,  User: demo-user, Password: demo-user. Allows FTPS and temporary uploads.




Cheers!

Oblivion


[Updated on: Wed, 12 April 2017 23:00]

Report message to a moderator

Re: FTP class and reference example for U++ [message #47859 is a reply to message #47858] Wed, 12 April 2017 23:05 Go to previous messageGo to next message
Klugier is currently offline  Klugier
Messages: 1075
Registered: September 2012
Location: Poland, Kraków
Senior Contributor
Hello,

I am not tested this package, but i would like to notice small issue with the API design. The problem is that you need to pass too many arguments to the single function, if you want to change single argument like ascii. Let's take a look at below function:
int FtpAsyncGet(const String& remote_file, const String& local_file, const String& host, int port = 21, const String& user = Null,
        const String& pass = Null, Event<int, int, String, int64, int64> progress = CNULL, bool active = false, bool ssl = false, 
        bool ascii = false);


This is the poor design, because the call will look like this:
FtpAsyncGet("remote.txt", "local.txt", "locaclhost, 21, Null, Null, CNull, false, false, true);

Looks terrible. Isn't it? In my opinion the good API has got maximum three parameters - otherwise it becomes hard to use by the client.

We can replace following code with the object approach (This is the example and I highly recommended to over thing this design):
int FtpAsyncGet(const FtpAssyncRequestConfig& config);

FtpAsyncGet(FtpAssyncRequestConfig("remote.txt", "local.txt", "locaclhost.txt").EnableAscii());

class FtpAssyncRequestConfig final
{
public:
    FtpAssyncRequestConfig(const String& remote_file, const String& local_file, const String& host)
        : // initialization ...
    {}

    // Get, Set, Enable interface...

private:
    // private members
};


Much more easier to set the n-th parameter, but require additional work for the library provider.

______________
You could replace NAMESPACE_UPP with namespace Upp { and END_NAMESPACE_UPP with }. The NAMESPACE_UPP approach was deprecated since last release.

______________
The next problem I see in the code is the ordering problem - you should put public class interface at the top then private things (Ftp and DirEntry classes). This is the general rule mentioned in following documentation topic.

______________
This code review aims to improve the code quality.

Sincerely,
Klugier


U++ - one framework to rule them all.

[Updated on: Wed, 12 April 2017 23:22]

Report message to a moderator

Re: FTP class and reference example for U++ [message #47860 is a reply to message #47859] Thu, 13 April 2017 08:17 Go to previous messageGo to next message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello Klugier,

Thank you very much for the feedback.


Quote:
You could replace NAMESPACE_UPP with namespace Upp { and END_NAMESPACE_UPP with }. The NAMESPACE_UPP approach was deprecated since last release.


It seems that I missed the announcement. Will change this asap.

Quote:
The next problem I see in the code is the ordering problem - you should put public class interface at the top then private things (Ftp and DirEntry classes). This is the general rule mentioned in following documentation topic.


I tend to follow the coding style of U++. It improves the readability a lot. But is this ordering mandatory? I mean even U++ has classes with members declared in "reverse" order.
In fact I used to declare class members in traditional order (public, protected, private) in the past, but changed my habit after I got used to U++ code base. Smile Nevertheless, no problem.
it will only take several seconds to reverse the order. YEt, imho, this is more a matter of aesthetic taste.

Quote:

We can replace following code with the object approach


Well, yes. In fact, I initially designed it with objective approach. Bu then decided otherwise.
Not that it wasn't useful. I had simply felt that it was bloating the code (this was maybe a side effect of being a former C programmer Smile ).

But what you propose is reasonable, actually.
I can depreceate the current convenience functions in favour of an objective approach.
Given the flexibility of FTP class and package, it will be very easy. FtpAsyncPut() and FtpAsyncGet() functions call, under the hood, FtpAsyncIO which has one more parameter to determine the direction of transfer.
An Ftp::Request object can take care of both synchronous and asynchronous conveninence functions' parameter list, without giving any headache, acting as an intermediate layer.

FtpGet(Ftp::Request(local_file, remote_file, localhost).SSL().Passive(), progress, whenwait)

FtpAsyncGet(Ftp::Request(local_file, remote_file, localhost).SSL().Active(), progress)



Also, I can change the Event<int, int, String, int64, int64>, which, frankly, I am not happy with, into an Event<Ftp::Result> (which can be returned by also (FtpGet and FtpPut):

Ftp:: Result {

public:
        int    GetErrorCode() const;
        String GetErrorDesc() const;
        int64  GetTotal() const;
        int64  GetDone() const;
        // Async only.
        int    GetId() const;
       
        bool   IsSuccess() const;
        bool   IsFailed() const;
        // Async only.
        bool   InProgress() const; 

        Result() {}

private:
// members...
};

//
// If we return Ftp::Result from FtpGet()
 
 if(FtpGet(...).IsSuccess())
   //....
 else
   //....

 // OR

  Event<Ftp::Result> r;
  FtpAsyncGet(..., r, ...);

  // In somewhere else, probably in the main thread:

void Foo::TransferProgress(Ftp::Result r) {

// First take care of UI thread synchronization, using whatever necessary, e.g PostCallback.
// then simply...

   if(r.InProgress()) 
     some_ctrl.Text(r.GetTotal(), r.GetDone());
   else
   if(r.IsSuccess())
     some_ctrl.Text("Done.");
   else
     some_ctrl.Text("Failed".);

}



What do you think?

Regards,

Oblivion







[Updated on: Thu, 13 April 2017 09:14]

Report message to a moderator

Re: FTP class and reference example for U++ [message #47861 is a reply to message #47860] Thu, 13 April 2017 12:31 Go to previous messageGo to next message
cbpporter is currently offline  cbpporter
Messages: 1401
Registered: September 2007
Ultimate Contributor
Does it work for HTTP(S) file gets?

I need to implement a large file downloader with pause and resume...
Re: FTP class and reference example for U++ [message #47862 is a reply to message #47861] Thu, 13 April 2017 13:55 Go to previous messageGo to previous message
Oblivion is currently offline  Oblivion
Messages: 1091
Registered: August 2007
Senior Contributor
Hello cbpporter,

cbpporter wrote on Thu, 13 April 2017 13:31
Does it work for HTTP(S) file gets?

I need to implement a large file downloader with pause and resume...


You mean Ftp over Http? Unfortunately, no. It's out of FTP package's scope.
But you can already download large files with it. Resume and pause is not implemented (yet).

Regards,

Oblivion.


Previous Topic: Am I doing something wrong, or is there a bug? FIREBIRD
Next Topic: OpenCV and openCV_Demo copiling error
Goto Forum:
  


Current Time: Thu Mar 28 11:58:42 CET 2024

Total time taken to generate the page: 0.01592 seconds