|
|
|
Home » U++ Library support » U++ Library : Other (not classified elsewhere) » Building & using U++ without TheIDE
| Re: Building & using U++ without TheIDE [message #11462 is a reply to message #11460] |
Fri, 14 September 2007 02:21   |
sergei
Messages: 94
Registered: September 2007
|
Member |
|
|
| luzr wrote on Fri, 14 September 2007 00:58 |
| Quote: |
3) I don't understand how unicode is implemented. There is String, AString, WString, but there is no TString, or whatever the name, like there is TCHAR that expands into char or wchar_t, depending on whether UNICODE/_UNICODE is defined. How do I define whether I'm in unicode or not?
|
This is sort of irrelevant. There is no UNICODE mode. All the time you have 8-bit and 16-bit String/WString.
Recommended approach is to use UTF-8 encoding. In that case, both strings can contain unicode and there is simple conversion between them. (You can however use one of 15 WIN/ISO encodings as 8-bit default.
| Quote: |
I mean, MessageBox will expand into MessageBoxA or MessageBoxW?
|
Always into MassageBoxA. However, in U++ you rather use Prompt, which can work with UTF8.
| Quote: |
And why path handling routines use char - can I handle unicode filenames with U++?
|
Unfortunately, there is drawback caused by fact that we still have to support win98, so we cannot use W variants  . In practive, this is really minor trouble, but nothing to be happy about it.
Anyway, when you are using only functions from U++, there is automatic conversion between U++ default encoding and 8-bit encoding of Windows.
Mirek
|
That would be one pretty serious drawback IMHO. I'm using Russian and Hebrew, and I've used so many apps with bad unicode support that I really want to avoid making another one. Who uses Win98 nowadays  (and there's unicode layer for these)
I've tried entering non-English chars in TheIDE source editor, it didn't let me. So I made a UTF-8 file containing a name of a file (with both Hebrew and Russian characters). U++ was able to read the name but unable to load the file. I guess this is the reason:
handle = CreateFile(ToSystemCharset(name),
Since this is CreateFileA, it can't open files with unicode filenames.
I tried adding #define UNICODE and #define _UNICODE. After commenting some stuff (cAlternateFileName didn't exist) it built the program but didn't work. CreateFileW can't accept UTF-8, it needs unicode.
Could this be fixed by some creative editing of Util.h? And what about window title, and controls? I mean, any chance to bring this to the TCHAR style, so the program can be compiled in ANSI, and in Unicode (supporting several languages simultaneously)? This should be a matter of some defines and a function to convert UTF8 to Win32 unicode. Or are there other parts of U++ that rely on a specific String/WString encoding?
|
|
|
|
| Re: Building & using U++ without TheIDE [message #11488 is a reply to message #11360] |
Sat, 15 September 2007 18:57 |
sergei
Messages: 94
Registered: September 2007
|
Member |
|
|
I made some progress.
Tried to implement full Unicode (TCHAR/String/WString handling), but eventually TheIDE/debugger went crazy (stopped several lines after breakpoint, for the same #ifdef flagUNICODE in one place in file went to #ifdef, another place to #else). So I dropped the idea for a while and returned to static lib building.
I made a mini-parser for *.upp files. Used it to generate list of used files, deleted them, ended up with unused/forgotten files. I'm attaching a zip of persumably forgotten files, maybe they should be removed.
I also found several referenced (used) yet non-existant packages:
TCore (Geom/Coords)
TCtrlLib (Geom/Ctrl)
TCtrlLib/Calc (Ole/Ctrl/Calc)
TDraw (Geom/Draw)
VectorDes (ide/VectorDes - it actually references itself, and by short path)
Additionally, directory separators aren't consistent - in some files '/' is used, in others '\'.
I also noticed that in plugin/png, most files are "pseudo-forgotten" - they don't appear in *.upp, yet they are included in pnglib.c. Does this happen in any other package?
Are there any plans to clean up the source? These inconsistencies are somewhat preventing me from completing the project generator.
-
Attachment: Unused.zip
(Size: 487.30KB, Downloaded 323 times)
|
|
|
|
| Re: Building & using U++ without TheIDE [message #11489 is a reply to message #11462] |
Sat, 15 September 2007 20:11 |
 |
mirek
Messages: 13975
Registered: November 2005
|
Ultimate Member |
|
|
| sergei wrote on Thu, 13 September 2007 20:21 |
That would be one pretty serious drawback IMHO. I'm using Russian and Hebrew, and I've used so many apps with bad unicode support that I really want to avoid making another one. Who uses Win98 nowadays (and there's unicode layer for these)
|
Well, I agree. Believe me, I would be more than happy to be finally able to cancel Win98 support and make everything UNICODE.
But even year ago there were people asking for *TheIDE* to run 98.... I can undestand that people want apps to be Win98 compatible, but developing on it?...
| Quote: |
I've tried entering non-English chars in TheIDE source editor, it didn't let me.
|
Actually, this is different reason. See FAQ. (In short, theide keeps track of encoding of file used and if keystroke does not match encoding, it is not inserted-> switch file encoding to UTF-8.)
| Quote: |
Could this be fixed by some creative editing of Util.h?
|
Yes. In fact, the code is there for PocketPC version.
| Quote: |
And what about window title, and controls?
|
Window title already works both in Win98 and XP UNICODE (it is quite easy to provide both way).
All widgets are UNICODE capable, there really is no trouble.
| Quote: |
I mean, any chance to bring this to the TCHAR style, so the program can be compiled in ANSI, and in Unicode (supporting several languages simultaneously)?
|
No. Forget about TCHAR style. But it can be easily done just right:)
Which makes me thing, this can even be done right while still supporting Win98, it is just much more work (need to test platform and dynamically link A or W version). .dli would help.
| Quote: |
This should be a matter of some defines and a function to convert UTF8 to Win32 unicode.
|
Well, in fact, these functions are already there and they work. In fact, U++ has no problem to handle chinesse filenames (because chinesse windows are using multibyte 8bit encoding of characters, something similar to UTF-8, and it is trivial to convert this to UTF-8 and back...).
| Quote: |
Or are there other parts of U++ that rely on a specific String/WString encoding?
|
Well, once you have String<->WString conversion and default encoding (SetDefaultCharset), the rest of code is pretty encoding ambivalent. The only real problem is the one you have encountered - the outside world interface. This is basically filenames and fonts; fonts are working UNICODE in Win98 out of the box, so really the remaining problem is filenames.
But once again, current pragmatic solution is just a result of fact that 2 years ago we had customers that required Win98 support...
Mirek
PS.: BTW, Hebrew... Is not it RTL? (Another not yet resolved problem, this time because we just do not know how RTL is supposed to work...).
|
|
|
|
| Re: Building & using U++ without TheIDE [message #11492 is a reply to message #11489] |
Sat, 15 September 2007 21:22 |
sergei
Messages: 94
Registered: September 2007
|
Member |
|
|
Have you tried using MS Unicode Layer for Win9x? I never did, but from the description it doesn't look much complicated. That way all builds could be Unicode.
Actually TCHAR is supposed to work. I started to do it, didn't have the chance to complete (see previous post), but in theory my solution should just work.
I assumed that String always contain UTF-8, and WString always contain UTF-16. I also assumed that String.ToWString and WString.ToString convert between them properly. And I used Win32 functions - MultiByteToWideChar and WideCharToMultiByte (have no idea how stuff like that works on Linux).
TSTR is a class I made, that would be an interface between String/WString and Win32 functions taking strings. Example:
String s = "text"; WString ws = "cap";
MessageBox(NULL, TSTR(s), TSTR(ws), MB_OK);
If UNICODE is defined, MessageBox would be MessageBoxW, and TSTR would transform String and WString to WCHAR*. Otherwise MessageBox would be MessageBoxA, and TSTR would transform both to char* (ANSI charset, removing unknown characters). This should compile just fine on Win9x without UNICODE defined, and on WinXP with/without UNICODE defined.
In the file there's also UTFBOM, since I couldn't type in TheIDE (thanks for telling how) I wrote some file handling.
Usage (read any encoding with BOM into UTF-8 string without BOM):
String ftxt = LoadFile("File.txt");
String sf; WString wsf;
int res = UTFBOM::ReadBOM(ftxt, &sf, &wsf);
String strUTF8;
if (res < 2) strUTF8 = UTFBOM::WriteBOM(sf, 1, false);
else strUTF8 = UTFBOM::WriteBOM(wsf, 1, false);
There are probably bugs in the files (couldn't test thoroughly), yet I believe the concept is correct.
Hebrew is indeed RTL. RTL by itself isn't complicated. Flip your screen vertically - that's pretty much what you get. Fully RTL indeed flips icons, minimize|maximize|close becomes close|maximize|minimize on the left side of the screen. Cursor advances to the left when typing. Home key brings cursor to right, End to left (in LTR Home goes left and End goes right). Backspace deletes symbol to the right, Delete deletes symbol to the left. Thinking of it now, if you flip LTR screen, the only thing that will be different from RTL screen is the left/right keys. That is, left remains left and right remains right. Only that in RTL right actually goes towards beginning of text, not end.
Now, that was the correct behaviour, at least what I'm used to (MS Word is quite good in Hebrew). Problems arise when combining RTL with LTR, especially if you also insert "neutral" symbols like !,.?
Then most text/word processors go crazy and results are rather unpleasant. MS Word is good, yet it isn't perfect.
(LTR TEXT)|(RTL TEXT)
| is the cursor. Press right - you'll probably get the cursor about here:
(LTR TEXT)(RTL TEX|T)
Because of the direction change. That also depends on paragraph orientation - whether the paragraph is RTL or LTR makes a difference. The behavior of Left/Right/Home/End/etc. keys is usually like that only in RTL paragraphs, and would be weird in LTR paragraph just like English would type weird in RTL paragraph (try it - right Ctrl+Shift in a textbox). Not exactly intuitive, and I'm not sure if that's standards-conforming behavior, but that often happens.
Does U++ work right with RTL-only (single language)? E.g. correct Left/Right/Home/End/Backspace/Delete behavior? That shouldn't be so difficult to implement.
P.S. please tell me what can be done about source packages. I don't know whether these are indeed mistakes, or I'm again doing something wrong.
-
Attachment: TStr.h
(Size: 6.13KB, Downloaded 402 times)
|
|
|
|
| Re: Building & using U++ without TheIDE [message #11493 is a reply to message #11492] |
Sat, 15 September 2007 22:00 |
|
mirek
Messages: 13975
Registered: November 2005
|
Ultimate Member |
|
|
| sergei wrote on Sat, 15 September 2007 15:22 |
Have you tried using MS Unicode Layer for Win9x? I never did, but from the description it doesn't look much complicated. That way all builds could be Unicode.
|
Well, but that is not out-of-box solution...
| Quote: |
Actually TCHAR is supposed to work. I started to do it, didn't have the chance to complete (see previous post), but in theory my solution should just work.
I assumed that String always contain UTF-8, and WString always contain UTF-16. I also assumed that String.ToWString and WString.ToString convert between them properly. And I used Win32 functions - MultiByteToWideChar and WideCharToMultiByte (have no idea how stuff like that works on Linux).
TSTR is a class I made, that would be an interface between String/WString and Win32 functions taking strings. Example:
|
Actually, there is already existing TSTR there, it is called FromSystemCharset and ToSystemCharset (you need two ways of conversion). See in util.cpp how it is defined for PocketPC - in that case, ToSystemCharset is the same thing as your TSTR...
Anyway, I would still rather used dynamic loading.
| Quote: |
In the file there's also UTFBOM, since I couldn't type in TheIDE (thanks for telling how) I wrote some file handling.
Usage (read any encoding with BOM into UTF-8 string without BOM):
|
What is BOM?
| Quote: |
Now, that was the correct behaviour, at least what I'm used to (MS Word is quite good in Hebrew). Problems arise when combining RTL with LTR, especially if you also insert "neutral" symbols like !,.?
|
Yes, that is exactly the trouble I am afraid of...
Mirek
|
|
|
|
|
|
|
|
| Re: Building & using U++ without TheIDE [message #11496 is a reply to message #11495] |
Sat, 15 September 2007 22:48 |
|
mirek
Messages: 13975
Registered: November 2005
|
Ultimate Member |
|
|
Ah, thanks, I did not knew that there is one for UTF-8...
So, if I understand you well, EF BB BF at the start of UTF-8 sequence should be ignored, right?
Anyway, this rather looks like file issue... Not sure it should be part of basic UTF-8 code?
| Quote: |
Another problem is that ToSystemCharset always returns String, which is castable to char*, and TSTR is castable to _TCHAR - which would always be what Win32 functions expect.
|
Ah, I see. You just did not undestood me. Look at PocketPC versio - that one returns WString instead of String. The result of these goes always directly to system calls (or, for FromSystemCharset, directly uses value returned by system call), therefore this is possible.
IMO, all really need to do to try playing with UNICODE/TCHAR is to change #ifdefs to use PocketPC (WinCE) version for UNICODE too.
| Quote: |
Is UTF-8 correctly converted in String <-> WString conversions?
|
I hope so, for basic plane and except the BOM. There is one extension though: Wrong UTF-8 sequences are interpreted byte by byte as characters 0xEExx (where xx is the byte).
The purpose is simple: This way, you can convert ANY input data to WString and back without loosing any information. This proved the absolute neccessity if you have the editor capable of handling multiple encodings in single file.
(That is why we named this UTF-8EE, as "Error Escape" or "placing to 0xEExx")
| Quote: |
And are other plane symbols also supported (symbols that take 4 bytes in UTF-16)?
|
No, unfortunately, other planes are not implemented yet.
Mirek
|
|
|
|
| Re: Building & using U++ without TheIDE [message #11502 is a reply to message #11496] |
Sun, 16 September 2007 01:43 |
sergei
Messages: 94
Registered: September 2007
|
Member |
|
|
Yes, UTF-8 can have BOM. Actually it has, and all programs I've used write/recognize it, with the unfortunate exception of GCC (it takes UTF-8 without BOM - such files aren't always shown correctly in text editors).
There is no need to modify existing UTF-8 handling since I doubt BOM is used in strings (it's essential for files). Yet my UTFBOM might be useful since LoadFile/SaveFile aren't encoding-aware, and files that are saved in UTF-16 format (that's what I commonly use for non-English text) would be loaded as ANSI/UTF-8. Or maybe just add autodecode to LoadFile and optional encoding params to SaveFile.
Multiple encodings in one file? Any examples? I don't think any text editor would recognize such a file.
I tried your suggestion about WinCE. Is PocketPC Unicode-only (that would be awesome if MS actually made such good decision)? First of all, cAlternateFileName, from Path.h, is never defined. As such, GetMSDOSName can't be defined either, and can't be used in FileSystemInfo::Find. That's likely a bug - I just commented everything out, or is there a way to implement GetMSDOSName?
Well, it didn't really work. The same "craziness" returned. I replaced:
#ifdef PLATFORM_WINCE
with:
#if defined(PLATFORM_WINCE) || defined(UNICODE)
and define UNICODE in main.cpp:
#define UNICODE
#include <Core/Core.h>
using namespace Upp;
...
Still, in the debugger it insisted to go into the #else. Rebuild all didn't help. My guess is that the solution could've worked (WString should cast into WCHAR*).
BTW, I've found some mistakes in UTFBOM and fixed that. Now I seem to be able to load/save UTF-8/UTF-16 LE/BE with/without BOM. I'm attaching the updated code (class + demo that I tried to use to read unicode-filenamed file). UTF-32 is kinda rare, though I might add it too for sake of completeness. However, that would require a String/WString that can work with embedded nulls - can they?
As for RTL, if I have time, I'll read Unicode specs on how LTR and RTL is mixed in same paragraph. The issue is quite interesting, but I'm afraid a standards-conforming solution might end up being an unusual one since many programs tend to ignore the existance of RTL languages.
P.S. is there a portable way to get a key from console (only key, without Enter)? Like _getch()?
P.S.2 String (UTF-8) seems to be the native U++ representation of text. Is WString used / recommended to be used for anything
besides Unicode OS API calls?
-
Attachment: UniTest.cpp
(Size: 3.78KB, Downloaded 529 times)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Building & using U++ without TheIDE [message #11548 is a reply to message #11360] |
Mon, 17 September 2007 11:07 |
sergei
Messages: 94
Registered: September 2007
|
Member |
|
|
It's UTF-16. And the problem is solved by ignoring it - non-main plane characters aren't supported.
My attempts of creating a static lib have slowly become attempts to simulate BLITZ outside TheIDE. Overally, they ended up successfully. This isn't BLITZ yet, but build times are better than with precompiled header. EXE size is bigger, though (the way I implemented this, packages are either not included, or included as a whole - so even a basic GUI app would have all widgets and lots of other packages compiled and linked).
Animated Hello example (full rebuild):
Code::Blocks svn4421 / MinGW 3.4.5 (Debug) : 1:18 / 241 warnings / 11.7MB.
Code::Blocks svn4421 / MinGW 3.4.5 (Release): 2:59 / 252 warnings / 3.3MB.
TheIDE 708dev2b / MinGW 3.4.2 (Debug): 1:11 / 0 warnings / 13.2MB.
TheIDE 708dev2b / MinGW 3.4.2 (Optimal): 2:39 / 1 warning / 1.6MB.
Modifying the source to work as SCU on a per-package basis isn't so difficult. There were several "cosmetic" changes - adding underscores / commenting to prevent "redefined" errors (there are very few conflicting symbol names in U++, yet they exist). But there was one large change - instead of the included 1.1.4 / 1.2.2 zlib, I had to use zlib 1.2.3 and seriously modify it. For one, in SCU all files are compiled as C++, and zlib is K&R C - modifying all function declarations was necessary. Then, there were many conflicts between different c files, more in overall than int whole U++ without zlib. I couldn't resolve the conflicts in bundled zlib version, but I managed to in zlib 1.2.3 (underscores, include guards, etc.).
I could upload modified files if anyone is interested in trying it out / modifying main source. Would be nice if main U++ source could include at least the minor changes (without zlib), that way only replacing zlib would be necessary.
|
|
|
|
|
|
|
|
| Re: Building & using U++ without TheIDE [message #11553 is a reply to message #11552] |
Mon, 17 September 2007 15:46 |
|
mirek
Messages: 13975
Registered: November 2005
|
Ultimate Member |
|
|
| sergei wrote on Mon, 17 September 2007 07:50 |
I successfully compiled Animated Hello example, and it ran fine (jumping colorchanging letters). I'll try other examples later, though I'm afraid they might not work due to the way TheIDE treats .lay and .iml files (other IDEs probably won't know what to do with these).
|
Should not be the problem, these files are just "compiled" using preprocessor. TheIDE has editors for them, but during build process, they are ignored (they do not have .cpp nor .c extension after all 
| Quote: |
Maintenance would probably consist of watching that .upp files are correct (no broken references), and that there are no conflicting names in the whole U++. The latter is currently easy but might become more difficult in the future. That's because U++ has lots of functions and enums, that are members of the namespace yet aren't members of any class. That seriously increases the chance that 2 unrelated cpp files will have a function with the same name - erroneous when compiling as SCU.
|
a) This is unlikely. Also, just the same name is not enough, only same signature is the problem.
b) I do not quite understand this SCU issue. I thought that task is to make possible to use U++ with CodeBlocks and VisualC++. I believe that users rather expect .lib files?
| Quote: |
Please tell me if it's fine to update zlib to a newer yet modified version. Plus, it should get tested (any simple U++ programs that could be used to test if zlib works correctly?). I'll try to make smallest-possible changes list to the main sources.
|
Yes, it is OK. Alternatively, please put updated package here so that we can pick it up for the next dev release.
Mirek
|
|
|
|
| Re: Building & using U++ without TheIDE [message #11562 is a reply to message #11553] |
Mon, 17 September 2007 18:33 |
sergei
Messages: 94
Registered: September 2007
|
Member |
|
|
| luzr wrote on Mon, 17 September 2007 15:46 |
a) This is unlikely. Also, just the same name is not enough, only same signature is the problem.
b) I do not quite understand this SCU issue. I thought that task is to make possible to use U++ with CodeBlocks and VisualC++. I believe that users rather expect .lib files?
Mirek
|
b) I started with the goal of .lib, yet now I find it rather unattractive due to the 410MB debug lib. With a lib there are 2 possibilities - redistribute it, or redistribute a project to easily build it. 410MB makes the former impossible, and the latter would require maintaining projects for different compilers/IDEs. + even with precompiled headers build time is about 10 mins vs ~1 min in TheIDE.
So what I did now, instead, is use SCU approach to drastically reduce compilation time (it's not much worse than BLITZ's now), partially at the cost of EXE size non-modularity (simple GUI and complex GUI apps will have the same big EXE since the whole CtrlLib package is linked).
I have an interface header for each package, implementing SCU, so when I do: #include <Upp/CtrlLib.h> it's like adding CtrlLib package in TheIDE. These headers are auto-generated from U++ source (using .upp files). So basically, user can work in Code::Blocks, without a static lib (saving space and better debugging), yet with similar fast compiles. Environment becomes quite similar to TheIDE, though there are drawbacks - no embedded help / .lay and .iml editors, and larger EXE size.
I believe these drawbacks aren't too big, and I don't rule out the possibility of building (based on PCH, not SCU) and redistributing a release-only lib to reduce final EXE size.
a) Unlikely? Yes. Yet possible and it happens. IsLeapYear is once a function, another time a macro. BINS is defined in heap and in draw palette, with different values. INITBLOCK/EXITBLOCK also cause conflicts since some happen to appear on the same line number. z.cpp (zlib) is for some reason in Core, and it redefines ASCII_FLAG, HEAD_CRC etc. already defined in plugin/z. RichText/Para.h has Code enum, yet Code is #defined in plugin/z/lib/deflate.h. These are the minor changes in main U++ source I was talking about. That, zlib, and a few casts in png to make it C++ compatible.
P.S. tried to compile on MSVC8. This piece from deflate.h:
#define Freq fc.freq
#define Code fc.code
#define Dad dl.dad
#define Len dl.len
Doesn't let me compile since Code and Len are names of parameters of functions in winnt.h. => RichText/Para.h is fine, zlib should be further modified.
I think I'll just prefix everything troublesome in zlib with "zlib_". Should work. Any demo projects to test zlib? I want to make sure that the library still works after my modifications.
|
|
|
|
|
|
Goto Forum:
Current Time: Wed Apr 24 11:00:02 CEST 2024
Total time taken to generate the page: 0.04247 seconds
|
|
|
Members
Pages
Help
Register
Login
Home
