U++ forum: Welcome to the forum

Status & Roadmap

Authors & License

Funding Ultimate++

Search on this site

Search in forums

Home » U++ Library support » U++ Libraries and TheIDE: i18n, Unicode and Internationalization » 16 bits wchar

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Return to the default flat view

Create a new topic

Submit Reply

Re: 16 bits wchar [message #11925 is a reply to message #11921]

Mon, 01 October 2007 14:28

mirek is currently offline

mirek
Messages: 14255
Registered: November 2005

Ultimate Member

cbpporter wrote on Mon, 01 October 2007 07:24

I think that the first thing that must be done is extended PutUtf8 and GetUtf8 so that it reads correctly th values outside of BMP. This is not too difficult and I will try to implement and test this.

I guess fixing Utf8 routines to provide UTF16 surrogate support (for now) is a good idea.

Quote:

The only issue is how to handle ill-formated values. I came to the conclusion that read and write operation must recognize compliant encodings, but it also must process ill-formated characters and insert them into the stream. If the stream is set to strict, it will throw an exception. If not, it will still encode. I propose the Least Complex Encoding TM possibility. Non-atomic Unicode aware string manipulation functions will should not fail when encountering such characters, so after a read, process and write, these ill-formated values (which could be essential to other applications) will be preserved. In this scenario, only functions that display the string must be aware that some data is ill-formated.

Well, the basic requirement there is that converting UTF8 with invalid sequences to WString and back must result in the equal String. This feat is successfully achieved by UTF8EE.

Also, I do not think that any string manipulation routine everywhere ever should be aware about UTF-8 or UTF-16 encoding. It is much practical to covert to WString, process and (eventually) convert it back. I think that in the long run, it might be even faster.

Quote:

Next, there should be a method to Validate the string, and a way to convert strings containing ill-formated string to error-escaped strings and back, so we can use atomic string processing if needed. This conversion should be done explicitly, so no general performance overhead is introduced.

bool CheckUtf8(const String& src);

You can add CheckUtf16 Smile

Smile

Anyway, seriously, I believe that the ultimate solution is to go with sizeof(wchar) = 4... The only trouble is converting this to UTF16 and back in Win32 everywhere... OTOH, good news is that after the system code is fixed, the transition does not pose too much backwards compatibility problems...

Mirek

Report message to a moderator

Send a private message to this user

[Message index]

		16 bits wchar By: riri on Mon, 05 February 2007 17:19
		Re: 16 bits wchar By: mirek on Mon, 05 February 2007 23:07
		Re: 16 bits wchar By: cbpporter on Tue, 25 September 2007 22:03
		Re: 16 bits wchar By: mirek on Tue, 25 September 2007 23:18
		Re: 16 bits wchar By: cbpporter on Wed, 26 September 2007 07:43
		Re: 16 bits wchar By: mirek on Wed, 26 September 2007 08:48
		Re: 16 bits wchar By: sergei on Wed, 26 September 2007 14:55
		Re: 16 bits wchar By: cbpporter on Wed, 26 September 2007 15:37
		Re: 16 bits wchar By: mirek on Wed, 26 September 2007 22:40
		Re: 16 bits wchar By: cbpporter on Mon, 01 October 2007 13:24
		Re: 16 bits wchar By: mirek on Mon, 01 October 2007 14:28
		Re: 16 bits wchar By: cbpporter on Wed, 03 October 2007 06:16
		Re: 16 bits wchar By: mirek on Wed, 03 October 2007 10:11
		Re: 16 bits wchar By: cbpporter on Wed, 03 October 2007 10:23
		Re: 16 bits wchar By: mirek on Wed, 03 October 2007 10:42
		Re: 16 bits wchar By: mirek on Wed, 03 October 2007 10:26
		Re: 16 bits wchar By: cbpporter on Wed, 03 October 2007 10:36
		Re: 16 bits wchar By: mirek on Wed, 03 October 2007 12:10
		Re: 16 bits wchar By: cbpporter on Wed, 03 October 2007 14:43
		Re: 16 bits wchar By: mirek on Wed, 03 October 2007 21:40
		Re: 16 bits wchar By: cbpporter on Thu, 04 October 2007 13:15
		Re: 16 bits wchar By: mirek on Thu, 04 October 2007 17:33
		Re: 16 bits wchar By: cbpporter on Thu, 04 October 2007 19:49
		Re: 16 bits wchar By: cbpporter on Fri, 12 October 2007 10:25
		Re: 16 bits wchar By: cbpporter on Fri, 12 October 2007 11:27
		Re: 16 bits wchar By: mirek on Fri, 12 October 2007 11:52
		Re: 16 bits wchar By: mirek on Fri, 12 October 2007 11:59
		Re: 16 bits wchar By: cbpporter on Fri, 12 October 2007 13:54
		Re: 16 bits wchar By: cbpporter on Fri, 12 October 2007 16:25
		Re: 16 bits wchar By: mirek on Fri, 12 October 2007 17:03
		Re: 16 bits wchar By: cbpporter on Mon, 15 October 2007 15:01
		Re: 16 bits wchar By: cbpporter on Mon, 15 October 2007 16:49
		Re: 16 bits wchar By: cbpporter on Tue, 16 October 2007 11:13
		Re: 16 bits wchar By: mirek on Sun, 21 October 2007 20:19
		Re: 16 bits wchar By: cbpporter on Sun, 21 October 2007 23:46
		Re: 16 bits wchar By: mirek on Sun, 21 October 2007 23:57
		Re: 16 bits wchar By: cbpporter on Mon, 22 October 2007 09:34
		Re: 16 bits wchar By: mirek on Mon, 22 October 2007 10:47
		Re: 16 bits wchar By: cbpporter on Mon, 22 October 2007 17:57
		Re: 16 bits wchar By: mirek on Mon, 22 October 2007 19:37
		Re: 16 bits wchar By: mirek on Sun, 21 October 2007 20:14
		Re: 16 bits wchar By: sergei on Wed, 26 September 2007 01:56
		Re: 16 bits wchar By: sergei on Wed, 26 September 2007 16:54
		Re: 16 bits wchar By: cbpporter on Wed, 26 September 2007 19:11
		Re: 16 bits wchar By: cbpporter on Wed, 24 October 2007 11:58
		Re: 16 bits wchar By: mirek on Wed, 24 October 2007 13:27
		Re: 16 bits wchar By: cbpporter on Wed, 24 October 2007 14:05
		Re: 16 bits wchar By: cbpporter on Thu, 25 October 2007 14:47
		Re: 16 bits wchar By: mirek on Sat, 27 October 2007 11:11
		Re: 16 bits wchar By: cbpporter on Tue, 06 November 2007 13:31
		Re: 16 bits wchar By: mirek on Fri, 09 November 2007 10:39
		Re: 16 bits wchar By: cbpporter on Sat, 10 November 2007 17:34
		Re: 16 bits wchar By: mirek on Sun, 11 November 2007 18:45
		Re: 16 bits wchar By: cbpporter on Fri, 04 July 2008 17:12
		Re: 16 bits wchar By: cbpporter on Wed, 23 July 2008 15:22
		Re: 16 bits wchar By: mirek on Wed, 23 July 2008 22:04
		Re: 16 bits wchar By: cbpporter on Sat, 02 August 2008 13:27
		Re: 16 bits wchar By: cbpporter on Sat, 02 August 2008 18:34
		Re: 16 bits wchar By: cbpporter on Sat, 02 August 2008 19:01
		Re: 16 bits wchar By: cbpporter on Sun, 03 August 2008 14:51
		Re: 16 bits wchar By: mirek on Mon, 04 August 2008 15:07
		Re: 16 bits wchar By: cbpporter on Mon, 04 August 2008 15:53
		Re: 16 bits wchar By: mirek on Mon, 04 August 2008 17:14
		Re: 16 bits wchar By: cbpporter on Mon, 04 August 2008 22:47
		Re: 16 bits wchar By: mirek on Tue, 05 August 2008 00:03
		Re: 16 bits wchar By: cbpporter on Tue, 05 August 2008 00:12
		Re: 16 bits wchar By: mirek on Tue, 05 August 2008 00:14
		Re: 16 bits wchar By: cbpporter on Tue, 05 August 2008 00:18
		Re: 16 bits wchar By: mirek on Tue, 05 August 2008 00:20
		Re: 16 bits wchar By: cbpporter on Tue, 05 August 2008 00:24
		Re: 16 bits wchar By: mirek on Tue, 05 August 2008 00:26
		Re: 16 bits wchar By: cbpporter on Tue, 05 August 2008 00:32
		Re: 16 bits wchar By: mirek on Tue, 05 August 2008 00:51
		Re: 16 bits wchar By: mirek on Tue, 05 August 2008 10:42
		Re: 16 bits wchar By: cbpporter on Tue, 05 August 2008 12:03
		Re: 16 bits wchar By: mirek on Tue, 05 August 2008 15:12
		Re: 16 bits wchar By: mirek on Tue, 05 August 2008 15:19
		Re: 16 bits wchar By: cbpporter on Tue, 05 August 2008 15:57
		Re: 16 bits wchar By: cbpporter on Wed, 06 August 2008 13:33
		Re: 16 bits wchar By: cbpporter on Thu, 07 August 2008 08:41
		Re: 16 bits wchar By: mirek on Thu, 07 August 2008 16:10
		Re: 16 bits wchar By: cbpporter on Thu, 07 August 2008 17:33
		Re: 16 bits wchar By: mirek on Thu, 07 August 2008 17:40
		Re: 16 bits wchar By: cbpporter on Thu, 07 August 2008 18:37
		Re: 16 bits wchar By: mirek on Thu, 07 August 2008 20:01
		Re: 16 bits wchar By: cbpporter on Fri, 08 August 2008 13:34
		Re: 16 bits wchar By: mirek on Fri, 08 August 2008 15:32
		Re: 16 bits wchar By: cbpporter on Fri, 08 August 2008 15:47
		Re: 16 bits wchar By: mirek on Fri, 08 August 2008 18:25
		Re: 16 bits wchar By: cbpporter on Sat, 09 August 2008 01:45
		Re: 16 bits wchar By: cbpporter on Fri, 05 September 2008 19:13
		Re: 16 bits wchar By: mirek on Sun, 07 September 2008 13:24
		Re: 16 bits wchar By: mirek on Mon, 04 August 2008 15:03
		Re: 16 bits wchar By: mirek on Sat, 27 October 2007 11:01

Previous Topic:	Arabic words from file
Next Topic:	Not possible to get .t files

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

PDF

]

Current Time: Mon Apr 28 16:41:31 CEST 2025

Total time taken to generate the page: 0.00697 seconds