|
|
Home » Developing U++ » UppHub » Some projects are ready to become open-source
Some projects are ready to become open-source [message #43030] |
Thu, 24 April 2014 17:07 |
Mindtraveller
Messages: 917 Registered: August 2007 Location: Russia, Moscow rgn.
|
Experienced Contributor |
|
|
Hi! It's been a while since I've posted anything to forum, but still U++ is the best thing I've used. There's a number of U++ projects I'd like to make open-source and make them as bazaar packages. But I don't really know if people really need them. Besides, some of them require additional work to become really cross-platform.
So I'll post a list of possible packages, and if someone considers them useful, please vote here.
1. First of all, I'd like to present multithreaded worker-based server class. I don't know if it duplicates any of Skylark functionality (never really used it). The main idea behind it was to make lightweight and easy to use solution for high load servers. Upon incoming http-request, a free worker thread is reused or new worker thread is automatically created. The class was made with http keep-alive support in mind.
2. Persistent storages. Rather simple wrapping templates around U++ storage classes adding functionality for accessing them in multithreaded environment with background auto-save to file system. Classes are very helpful for effective management of not-very-big amounts of data where SQL-based solutions seem too "heavy". There are plans to decrease memory usage and serialization effectiveness for these storages if it comes to it.
3. BNF (Backus-Naur form) classic implementation based on stack machine. Stable but really needs upgrade to support some jit-compiled code for internal functions. I'd be very grateful if anyone is interested. I don't use BNF package very frequently alone, instead it is widely used as a dependency of next package.
4. Industrial-grade, completely asynchronous serial communication library. Based on Apache-licensed serial library. It doesn't work on send/receive bytes level. Instead, it operates with BNF-defined protocols, parses them from file upon startup. User code operates with high-level things like input and output Value vectors. If you want some simple experiments with serial port, this is too heavy solution. But if you write big project where serial communications involved, this is what you might dream of. Very robust and well tested through years.
5. ToJSTime/FromJSTime functions for browser time representation support. Very small, doesn't worth creating package for them.
6. ConvertTextToUtf8 function. Charset detection function which makes utf-8 text from almost any encoding. Uses UCI & iconv libraries. Succesfully detects almost any encoding and converts text with at least 10 characters long (smaller text might give too big detection error).
7. U++ packaged htmlcxx library. Very lightweight but accurate HTML/CSS parsing library with LGPL license.
Added utility function SafeguardHTML (with utf8 conversion from previous function). The main idea is to create 100% safe html output from user input in any encoding.
As always, any suggestions or critics are welcome.
|
|
|
Re: Some projects are ready to become open-source [message #43031 is a reply to message #43030] |
Thu, 24 April 2014 18:54 |
|
Hi Pavel,
Numbers 2 and 7 sound quite useful to me Not that I had any project in mind that would make use of them right now, but they sound like something one would need relatively often.
Other points of course sound interesting too, especially 3 & 4, but I personally try to stay as far from communication in programs as possible
Best regards,
Honza
|
|
|
|
|
|
|
|
|
Re: Some projects are ready to become open-source [message #43098 is a reply to message #43030] |
Tue, 06 May 2014 23:29 |
Mindtraveller
Messages: 917 Registered: August 2007 Location: Russia, Moscow rgn.
|
Experienced Contributor |
|
|
MtAlt - base package used by other packages, version 2.0. This is alternative approach to multithreading discussed here:
http://www.ultimatepp.org/forums/index.php?t=msg&goto=25 087#msg_num_7
This new version introduces bounds for callback queues, priorities and much better debugging support including runtime latency measurements with file/line indication (everything is completely wiped out release mode leaving no overhead).
Frankly speaking, it's a pity I was unable to write good enough documentation for this package, so people consider this approach too unnatural and don't use it. But proper usage of this package may eliminate many problems when you have 3-5-10 or even more threads.
Please update Bazaar/MtAlt package with this new version.
[Updated on: Wed, 07 May 2014 01:23] Report message to a moderator
|
|
|
|
Re: Some projects are ready to become open-source [message #43100 is a reply to message #43030] |
Tue, 06 May 2014 23:44 |
Mindtraveller
Messages: 917 Registered: August 2007 Location: Russia, Moscow rgn.
|
Experienced Contributor |
|
|
SerialPort - not just a simple serial port package. It is based on MtAlt and BNF.
Long story short, it loads the file with description of each serial port to be used. A multithreaded class is created for each port. It also loads a file with protocol descriptions (i.e. modbus,...).
Let's look how you request i/o with serial port:
void SendReceive_
(
String sendProto,
Vector<Value> sendArgs,
String receiveProto,
dword timeout,
void *custom,
Ptr<OBJECT> notify,
void (OBJECT::*cb)(bool success, Vector<Value> args, void *custom),
bool useAcc = false,
bool priority = false
)
Ending underscore means this is asynchronous function which may be called from any other thread ("mt safe").
sendProto and sendArgs are send data generation protocol and it's variables (string, integer, float, etc.)
receiveProto is data parsing protocol to parse incoming data. As soon as incoming bytes are parsed successfully, incoming datagram is considered succesfully received.
timeout is a receive timeout which is monitored by internal thread.
custom - is just a custom value to distinguish multiple SendReceive_() calls from each other.
notify/cb - is a CallbackQueue/CallbackThread object and it's member function which is called upon send/receive/timeout event.
useAcc - is a "caching" feature; Serial port classes keep some cache of incoming byte sequences already parsed. And if the same sequence is met, no actual parsing is made, instead returning cached result. This is extremely useful for slow processors like early celerons or arm. My tests revealed that more than 60% of incoming packages are the same and thus are cached very effectively leaving many processor resources for other tasks.
priority - is a simple true/false switch putting your call into the front of queue of into the end of it.
Yes, just imagine you call SendReceive_() before previous call is still working. According to MtAlt, all subsequent calls are added to the queue (not necessarily to the tail, because we have priorities).
This package needs further discussion and examples which will follow if someone interests. Should we make new topic for it?
|
|
|
Re: Some projects are ready to become open-source [message #43101 is a reply to message #43030] |
Wed, 07 May 2014 00:16 |
Mindtraveller
Messages: 917 Registered: August 2007 Location: Russia, Moscow rgn.
|
Experienced Contributor |
|
|
Utils - a minified part of utility package from one of my projects. It is not currently compiled and linked cross-platform. Actually it is not intended to be standalone package, but rather a bunch of possibly useful functions using:
- ICU ( http://site.icu-project.org/ ) a standard Linux/Unix library also present for Windows (may be even inside standard libs, I don't remember exactly).
- iconv a standard Linux/Unix library for charset conversion
- embedded htmlcxx library for html/css parsing
Interesting functions:
Encrypt/Decrypt - very simple AESStream wrapper for small strings conversion. I propose adding them to AESStream itself.
ToJSTime/FromJSTime - very simple conversion from/to javascript "integer" Time/Date representation.
ConvertTextToUtf8 - ICU's charset detector result routed to iconv library making output text with utf-8 charset. My tests introduced excellent results for big texts (about 1 KB) and rather good results for texts above 100 bytes. If text is about ~10 bytes, the charset detection generally results in wrong detected charsets.
SafeguardPlain - web-oriented function to convert text added by user to make it 100% safe before embedding it to web page.
SafeguardHTML - web-oriented function to convert html to make it 100% safe before embedding it to web page. The only way of doing it is parsing input html (with htmlcxx), then filtering it (with HTMLFilter class), and then reconstructing resulting html back again. You may of course take htmlcxx library from this package and use it the way you want (using SafeguardHTML as an example of htmlcxx usage).
ParseFormData - is a function to parse input form-data from socket (actually it is little outdated, but it is no problem rewriting it for TcpSocket). The main idea behind this function is to create streaming solution for downloading large files from Internet to the drive. So ordinary form fields are added to fields variable, and files are streamed directly to file system using file names from files variable. Very useful when working with big data in servers.
[Updated on: Wed, 07 May 2014 00:16] Report message to a moderator
|
|
|
Re: Some projects are ready to become open-source [message #43102 is a reply to message #43030] |
Wed, 07 May 2014 00:38 |
Mindtraveller
Messages: 917 Registered: August 2007 Location: Russia, Moscow rgn.
|
Experienced Contributor |
|
|
PersistentMap - a bunch of templates to make U++ vectors/maps persistent.
The idea behind it is very simple:
1) Take original storage class
2) Restrict or don't restrict access to it storage itself depending on flavour.
3) Add functions for "mt safety".
4) Add background thread to write data to disk if storage is changed with some timeout (if it is changed frequently).
5) Make writes atomic and stable across situation of write failure / io failure / power failure, so that user code transparently works with stored data even after failure.
currently added is
6) Versioning support for "open" flavour
and to be done is
7) Clustered incremental disk flushes for partial and effective disk writes.
First flavour is "restricted" PersistentStorage/PersistentMap<STORAGE>. This template doesn't give access to underlying container. Instead it introduces functions like Add, AddPick, Get, GetFirst, Put, CheckAdd or GetRemoveFirst. You may be sure about it's "mt safety" and persistency.
But this is not always enough. Sometimes you need more flexibility, and here you have "open" flavour, a OpenPersistentStorage<STORAGE> class. It gives you full access to underlying storage, but you are responsible for calling EnterRead()/LeaveRead() and EnterWrite()/LeaveWriteSetChanged(bool changed).
Currently these templates are in release stage and you may use them without any doubt of losing your data. Any suggestions or proposals are welcome.
|
|
|
Re: Some projects are ready to become open-source [message #43103 is a reply to message #43030] |
Wed, 07 May 2014 01:15 |
Mindtraveller
Messages: 917 Registered: August 2007 Location: Russia, Moscow rgn.
|
Experienced Contributor |
|
|
ServerWorker - a multithreaded high load server.
I'm of course aware of modern Skylark functionality created by Mirek. But there are cases when you need not a front-end server, but instead you need very effective intermediate processing server which effectively handles multiple connections simultaneously.
The base concept of this approach is a "worker". Worker is an instance executed in it's own thread which processes single user's request and finishes. In my implementation, if there's a free worker it is used for new request, or new worker/thread is added.
This approach leads to very efficient and extremely lightweight server which handles large amount of multiple connections.
Actually I think it is one of useful things among published, but it is up to you - if you consider it useful for your tasks (high load is not a kind of task you frequently meet in real life).
To demonstrate the class is really easy to use, I'll post example here.
How server with custom workers is created and used:
TcpSocket server;
server.Listen(port,connections);
ServerWorker::ServerThread<CommentWorker>(server);
ServerWorker::ClearWorkers_();
Making custom worker itself:
#ifndef _commentd_CommentWorker_h_
#define _commentd_CommentWorker_h_
#include <Core/Core.h>
#include "ServerWorker.h"
using namespace Upp;
class CommentWorker : public ServerWorker
{
public:
CommentWorker()
{
ServerWorker::AddHandler<CommentWorker>("/login", &CommentWorker::HandleAddLogin);
ServerWorker::AddHandler<CommentWorker>("/comment", &CommentWorker::HandleAddComment);
//...
}
void HandleAddLogin(const String &request, HttpHeader &header, TcpSocket &socket)
{
VectorMap<String,String> values = ParseUrlQueryString(header.GetURI());
String valueLogin = values.GetAdd("login");
String valuePass = values.GetAdd("pass");
//...
HttpResponseKeepAlive(
socket, false, 200, "OK", "text/html; charset=\"utf-8\"",
HTML, NULL,
VectorMap<String,String>()
);
}
//...
};
#endif
That's it. You don't care about creating threads or workers. You just write handlers keeping in mind the multithreaded nature of your code. PersistentStorage classes fit here perfectly.
The current state of this class is almost beta. It is working but some non critical issues are found (i.e. longer answer timeout on Windows) as well as functionality itself is yet to be finalized.
I'd be very grateful on your suggestions and thought on this topic.
[Updated on: Wed, 07 May 2014 01:20] Report message to a moderator
|
|
|
|
|
|
|
|
|
Goto Forum:
Current Time: Fri Nov 01 00:02:16 CET 2024
Total time taken to generate the page: 0.02155 seconds
|
|
|