Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » U++ Library support » U++ Libraries and TheIDE: i18n, Unicode and Internationalization » Making support for code pages unlimited
Making support for code pages unlimited [message #23056] Sat, 12 September 2009 09:56 Go to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

Recently I`ve came to need of U++ ability to encode bytes into different encodings. Actually, to ANY current encoding available (these bytes could be text in ANY language and ANY encoding).
As fas as I know U++ supports widening it`s internal supported encodings list thus supporting rather limited set of encodings by default.
So if I understand situation correctly adding more encodings into U++ would be handy. I hope to spend some time to make a kind of parser which will take iconv sources' tables and convert them into native U++ format. Does it make sense?
Re: Making support for code pages unlimited [message #23057 is a reply to message #23056] Sat, 12 September 2009 16:27 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Mindtraveller wrote on Sat, 12 September 2009 03:56

Recently I`ve came to need of U++ ability to encode bytes into different encodings. Actually, to ANY current encoding available (these bytes could be text in ANY language and ANY encoding).
As fas as I know U++ supports widening it`s internal supported encodings list thus supporting rather limited set of encodings by default.
So if I understand situation correctly adding more encodings into U++ would be handy. I hope to spend some time to make a kind of parser which will take iconv sources' tables and convert them into native U++ format. Does it make sense?


Yes. The native U++ format is a word table mapping 128 upper characters (< 128 is expected to be ASCII - that means some weird old encoding cannot be supported) to UNICODE.

Mirek
Re: Making support for code pages unlimited [message #23061 is a reply to message #23057] Sun, 13 September 2009 01:38 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

OK, first step is complete.
1. Filtered almost empty charsets and charsets with more than 2-byte unicode per character (seems like currently unsupported by U++ within common CHARSET_* and CHRTAB_* based internal functions).
2. Codepage names are taken from iconv, which is de-facto standard IMO (previous names could be left for backward compatibility).
3. Parsed and collected data into 2 files (2 pieces each), which are release candidates for insertion into Charset.cpp/.h.

Please look at these files. IMO generally they are good but some charsets could be filtered too.

UPD: Sorry, U++ forum fails to add attachments (sometimes more than one, sometimes more than zero). Will attach files ASAP.

[Updated on: Sun, 13 September 2009 08:21]

Report message to a moderator

Re: Making support for code pages unlimited [message #23062 is a reply to message #23061] Sun, 13 September 2009 01:42 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

UPD2: Uploaded files to some file sharing resource:
http://www.2shared.com/file/7757007/3819137b/CharSeth.html
http://www.2shared.com/file/7757008/a8a60eea/CharSetcpp.html

Sorry for using external resources, but forum still denies my attachments (maybe a kind of automatic anti-flood system spoils the game).

[Updated on: Sun, 13 September 2009 01:47]

Report message to a moderator

Re: Making support for code pages unlimited [message #23078 is a reply to message #23062] Tue, 15 September 2009 22:09 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

Mirek, could you please look into proposed sources and tell if I need to add anything more to make these codepages added to U++.
(Maybe as some bazaar extension?)
Also I`d like to add standard multibyte code pages for hieroglyphic languages like Japanese, Chinese, etc. Is it possible? I may widen my parser to convert these arrays from iconv sources.

[Updated on: Wed, 16 September 2009 18:00]

Report message to a moderator

Re: Making support for code pages unlimited [message #23085 is a reply to message #23078] Wed, 16 September 2009 18:14 Go to previous messageGo to next message
andrei_natanael is currently offline  andrei_natanael
Messages: 262
Registered: January 2009
Experienced Member
Files attached.
  • Attachment: CharSet.h.i
    (Size: 5.52KB, Downloaded 429 times)
  • Attachment: CharSet.cpp.i
    (Size: 89.99KB, Downloaded 359 times)
Re: Making support for code pages unlimited [message #23086 is a reply to message #23085] Wed, 16 September 2009 20:55 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

Thanks a lot! Strange forum still denies my attaching any files here.
Re: Making support for code pages unlimited [message #23110 is a reply to message #23086] Thu, 17 September 2009 17:21 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Mindtraveller wrote on Wed, 16 September 2009 14:55

Thanks a lot! Strange forum still denies my attaching any files here.


Thanks, this is a wonderful addition!

There were two minor problems, one solved (I have removed some duplicities), one persists - that third parameter of AddCharSetE should be charset that best represents charset on host platform.

In reality, it is perhaps not really useful... but obviously your settings are wrong Smile

Mirek
Re: Making support for code pages unlimited [message #23116 is a reply to message #23110] Thu, 17 September 2009 19:13 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

luzr wrote on Thu, 17 September 2009 19:21


In reality, it is perhaps not really useful... but obviously your settings are wrong Smile

Mirek

What settings do you mean?

As I understand, you accepted my addition of charsets, and I want to ask if there is a possibility to add multibyte character sets (like Chinese or Japanese) to make supported character sets list complete. I could parse and add them too, but I don't know how to add them to U++.
Re: Making support for code pages unlimited [message #23117 is a reply to message #23116] Thu, 17 September 2009 19:24 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Mindtraveller wrote on Thu, 17 September 2009 13:13

luzr wrote on Thu, 17 September 2009 19:21


In reality, it is perhaps not really useful... but obviously your settings are wrong Smile

Mirek

What settings do you mean?



#ifdef PLATFORM_WIN32
....
	AddCharSetE("iso8859-1",   CHRTAB_ISO8859_1,  CHARSET_WIN1252);


Here the last parameter, CHARSET_WIN1252, says that ISO8859_1 equivalent in Win32 is WIN1252 - it mostly contains same characters, although not at same codepoint.

Well, in fact, I think we can happily remove this info... I will check soon if that is possible.

Quote:


As I understand, you accepted my addition of charsets, and I want to ask if there is a possibility to add multibyte character sets (like Chinese or Japanese) to make supported character sets list complete. I could parse and add them too, but I don't know how to add them to U++.



Well, that will be tricky. I think we will have to change charset.cpp internals a bit to support them.

Also, I am not sure that I want a copy of big CJK conversion tables in each application. Maybe this could be in another package (of course, somehow registering into regular charset.h API).
Re: Making support for code pages unlimited [message #23121 is a reply to message #23117] Thu, 17 September 2009 20:18 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

You are right of course. I thought about that and came to conclusion that the best approach will be making some compile flags (defines, I mean) that will enable compiling additional charsets for those programs which need them.
Re: Making support for code pages unlimited [message #23159 is a reply to message #23121] Sun, 20 September 2009 09:20 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Mindtraveller wrote on Thu, 17 September 2009 14:18

You are right of course. I thought about that and came to conclusion that the best approach will be making some compile flags (defines, I mean) that will enable compiling additional charsets for those programs which need them.


What is wrong with another package?
Re: Making support for code pages unlimited [message #23162 is a reply to message #23159] Sun, 20 September 2009 13:17 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

If it is possible to make explicit package with these tables defined, then no problem. Cool
Re: Making support for code pages unlimited [message #23166 is a reply to message #23159] Sun, 20 September 2009 23:07 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Problem:

ARMSCII

and

CP1161

(U++ crashes on ASSERT in debug mode).

I have commented them out for now...

Mirek
Re: Making support for code pages unlimited [message #23171 is a reply to message #23166] Tue, 22 September 2009 09:58 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

What is the problem with these tables? How could they possibly lead to crash?

Will you make these tables an additional package as planned?

Also I'd like to ask you to change charset.cpp internals for multibyte charsets support. Then I will port to U++ additional ISO codepages for languages like Japanese or Chinese. This will make U++ support for character pages complete.

These will be handy for those who make really widely used apps with U++.

[Updated on: Tue, 22 September 2009 15:08]

Report message to a moderator

Re: Making support for code pages unlimited [message #23184 is a reply to message #23171] Thu, 24 September 2009 10:35 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
I am sorry I did not have to look into crashing tables more, now I did.

The problem with these crashes was mostly artificial, there were additional check to debug problems in tables:

- check that none of characters in the table is <128 (problem in ARMSCII_8)

- check that there are no duplicates (CP1161).

After removing the check, everything seems to be OK now.

As for multibyte character sets....

There is sort of problem, because some of charset.h expect single character.

So I guess all we can do is some sort of hook into 'whole string' functions that gets extended/reimplemented in "MBCS" package.

Maybe something like:

void RegisterMBCS(byte charset,
                  WString (*tounicode)(const char *s, int len),
                  String (*fromunicode)(const wchar *s, int len));


or maybe rather

void RegisterMBCS(byte charset,
                  WString (*tounicode)(const char *s, int len, int charset),
                  String (*fromunicode)(const wchar *s, int len, int charset));


or even

void RegisterMBCS(byte charset, void *param,
                  WString (*tounicode)(const char *s, int len, void *param),
                  String (*fromunicode)(const wchar *s, int len, void *param));


Mirek

[Updated on: Thu, 24 September 2009 10:36]

Report message to a moderator

Re: Making support for code pages unlimited [message #23186 is a reply to message #23184] Thu, 24 September 2009 10:47 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

As for MBCS support, the first option seems to me the most simple. As soon as these functions are implemented (simple axample of usage will be handy), I'll finish quasi-parser and convert remaining iconv tables into U++.
More important question IMO is if it will be possible to use these charsets in common U++ charset coversion functions.
Re: Making support for code pages unlimited [message #23187 is a reply to message #23186] Thu, 24 September 2009 11:05 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Mindtraveller wrote on Thu, 24 September 2009 04:47

As for MBCS support, the first option seems to me the most simple. As soon as these functions are implemented (simple axample of usage will be handy), I'll finish quasi-parser and convert remaining iconv tables into U++.
More important question IMO is if it will be possible to use these charsets in common U++ charset coversion functions.


Read above. Not in all.

E.g.:

int IsLetter(int c, byte chrset);

is obviously impossible with MBCS...
Re: Making support for code pages unlimited [message #23189 is a reply to message #23187] Thu, 24 September 2009 11:46 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

I mean functions like ToUnicode(), FromUnicode().
Re: Making support for code pages unlimited [message #23190 is a reply to message #23189] Thu, 24 September 2009 11:49 Go to previous messageGo to previous message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
Mindtraveller wrote on Thu, 24 September 2009 05:46

I mean functions like ToUnicode(), FromUnicode().


Uh, I guess that above proposed API is just for this...

Mirek
Previous Topic: same LNG_ in lang.cpp
Next Topic: Updated romanian translation
Goto Forum:
  


Current Time: Thu Mar 28 20:57:49 CET 2024

Total time taken to generate the page: 0.01367 seconds