Home » Developing U++ » UppHub » Encrypted storage with streaming (OpenSSL, AES)
| Encrypted storage with streaming (OpenSSL, AES) [message #23087] |
Wed, 16 September 2009 22:17  |
Mindtraveller
Messages: 917 Registered: August 2007 Location: Russia, Moscow rgn.
|
Experienced Contributor |

|
|
Sometimes we may have task to store some large file (4+ GB) or small string inside encrypted storage. I tried to make a pair classes which make it easy. This is the first version, so any ideas are welcome.
This package assumes you have OpenSSL library successfully installed and its paths are added to TheIDE.
OK, let me introduce a pair of classes called AESEncoderStream and AESDecoderStream. They support streamed adding and encryption/decryption of data. Encryption is made with AES (Rijndael) with 128, 192 or 256 bit keys.
Encrypted data 32 bytes larger than source length aligned to 16-byte boundary. I.e. if your source data is 170 bytes long, the resulting length is:
170 rounded by 16-byte pieces = 176
plus
32 (header data)
= 176 + 32 = 208 bytes.
Not so ugly for a number of applications especially if source data is large.
Here is a simple self-explanating demo:
#include <Core/Core.h>
#include <openssl/aes.h>
#include <AESStream/AESStream.h>
using namespace Upp;
CONSOLE_APP_MAIN
{
AESInit();
// Generate cryptographically stable key
String key(AESRandomString(32));
// Encryption
String sIn,sOut;
sIn = "qwertyuiop[p\tasdfghjkl;zxcvbnm,./quwiueqiwueoiquweioquweioquweiqwueicuwinuqiweqiwue pqiueci eiqniuriryuweyruweyruewrycuwbrurbywuyrwquiercbbcrebrquwey";
AESEncoderStream aesEncoder(sIn.GetLength(), key);
aesEncoder << sIn.Left(10);
aesEncoder << sIn.Mid(10,10);
aesEncoder << sIn.Right(sIn.GetLength() - 20);
sOut << aesEncoder; //do streamed encoding
// Decryption
//key.Set(0, 'a'); //uncomment to see what happens with wrong key
AESDecoderStream aesDecoder(key);
aesDecoder << sOut.Left(15); //you may add by parts
aesDecoder << sOut.Right(sOut.GetLength() - 15);
try
{
String sDecoded;
sDecoded << aesDecoder; //throw exception if key is wrong
Cout() << (sDecoded == sIn) << "\n\n"; //check if all converted successfully
}
catch (const char *xp)
{
Cout() << "\n!!Error: " << ToSystemCharset(xp);
}
}
[Updated on: Fri, 18 September 2009 11:16] Report message to a moderator
|
|
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #23120 is a reply to message #23103] |
Thu, 17 September 2009 19:57   |
Weras
Messages: 4 Registered: August 2009
|
Junior Member |
|
|
Hi koldo!
I had resolved this problem and I did so.
1. Download openssl-0.9.8k.tar.gz from
https://www.openssl.org/source/
2. Unpack archive to C:\temp\openssl-0.9.8g
3. Download and install ActivePerl
4. Now type in the command line:
1) C:\temp\openssl-0.9.8g>perl Configure VC-WIN32 --prefix=c:/temp/openssl-bin/
2) C:\temp\openssl-0.9.8g>%comspec% /k ""c:\Program Files\Microsoft Visual Studio 8\VC\vcvarsall.bat"" x86
3) C:\temp\openssl-0.9.8g>ms\do_masm.bat
4) if you need static library write
C:\temp\openssl-0.9.8g>nmake -f ms\nt.mak
else, if you need dynamic library write
C:\temp\openssl-0.9.8g>nmake -f ms\ntdll.mak
5. The result is files*.lib & *.dll and include directory
6. Add .../openssl/inc32 as Include and .../openssl as Linker directories
Is enough to work with openssl
[Updated on: Thu, 17 September 2009 20:05] Report message to a moderator
|
|
|
|
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #23143 is a reply to message #23139] |
Fri, 18 September 2009 12:28   |
|
|
| Mindtraveller wrote on Fri, 18 September 2009 12:20 | Hi, koldo!
It was Weras who instructed me how to install OpenSSL so I asked him to answer yur question.
Thanks for trying Linux version, it's a great effort.
I have just found a pair of small bugs and reuploaded sources. If you wish to try AESStream, pleas rewrite sources with new versions. If you manage to make POSIX version of rdtsc - it would be great too (I'm no professional in GCC-accepted assembler).
I wonder if people need this too. If it is so, someone from authors may upload AESStream to official Bazaar.
|
Very nice package and wiki how to build openssl on win32.
Thank you!
Ion Lupascu (tojocky)
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #23144 is a reply to message #23087] |
Fri, 18 September 2009 13:23   |
 |
koldo
Messages: 3458 Registered: August 2008
|
Senior Veteran |
|
|
Hello Mindtraveller
About rdtsc, in fact it compiles in gcc as it has an ifdef so that:
- If MSC, it takes a value from rdtsc (time stamp counter 64-bit register)
- Else it takes Random()
It seems that to get a random number our Random() implementation is better than the clock (MT19937 algorithm), so perhaps rdtsc() would have to be changed with Random()
I have tried to compile in MinGW, but I get linking errors, in summary:
Openssl\out32\libeay32.lib(tmp32/ui_openssl.obj),(.text[_rea d_string_inner]+0xb): undefined reference to `__security_cookie'
Openssl\out32\libeay32.lib(tmp32/ui_openssl.obj),(.text[_rea d_string_inner]+0x149): undefined reference to `@__security_check_cookie@4'
Openssl\out32\libeay32.lib(tmp32/ecp_smpl.obj),(.text[_ec_GF p_simple_group_set_curve]+0x6): undefined reference to `_chkstk'
Does anybody know how to solve these problems with chkstk and security_cookie ?
Best regards
Koldo
Best regards
Iñaki
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #25394 is a reply to message #25392] |
Sun, 21 February 2010 12:01   |
Mindtraveller
Messages: 917 Registered: August 2007 Location: Russia, Moscow rgn.
|
Experienced Contributor |

|
|
Koldo, what is the package uppdev/textssl? Never heard of that before. You mean another OpenSSL-based package?
I also changed package a bit, added tutorial (English and Russian versions), added example. Please be so kind check if it compiles (currently don't have OpenSSL installed) and upload it to bazaar.
[Updated on: Sun, 21 February 2010 12:08] Report message to a moderator
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #25395 is a reply to message #25394] |
Sun, 21 February 2010 15:40   |
 |
koldo
Messages: 3458 Registered: August 2008
|
Senior Veteran |
|
|
| Mindtraveller wrote on Sun, 21 February 2010 12:01 | Koldo, what is the package uppdev/textssl? Never heard of that before. You mean another OpenSSL-based package?
I also changed package a bit, added tutorial (English and Russian versions), added example. Please be so kind check if it compiles (currently don't have OpenSSL installed) and upload it to bazaar.
|
Hello Mindtraveller
Web/SSL package is included in AESStream. A problem is it have memory leaks. I have found a solution to solve it, that is in Web/SSL/util.cpp:
INITBLOCK {
Socket::Init();
CRYPTO_set_mem_functions(SSLAlloc, SSLRealloc, SSLFree);
SSL_load_error_strings();
//SSL_library_init(); //SOLUTION TO MEMORY LEAK !!
}
Probably SSL_library_init(); is not necessary and as it not properly cleaned up, there are memory leaks.
To find this problem I have used uppdev/testssl package and Upp technology to detect memory leaks 
I have tested your changes but they do not work well . Check this:
sIn includes this:
| Quote: | qwertyuiop[p asdfghjkl;zxcvbnm,./quwiueqiwueoiquweioquweioquweiqwueicuwin uqiweqiwue pqiueci eiqniuriryuweyruweyruewrycuwbrurbywuyrwquiercbbcrebrquwey
|
sDecoded iincludes this:
| Quote: | qwertyuiop[p asdfghjkl;zxcvbnm,./quwiueqiwueoiquweioquweioquweiqwueicuwin uqiweqiwue pqiueci eiqniuriryuweyruweyruewrycuwbrurbywuyrwquiercbbcrebr
|
Best regards
Iñaki
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #25402 is a reply to message #25401] |
Mon, 22 February 2010 08:31   |
Mindtraveller
Messages: 917 Registered: August 2007 Location: Russia, Moscow rgn.
|
Experienced Contributor |

|
|
| koldo wrote on Mon, 22 February 2010 09:54 | 1) Does it mean that AES cannot be used for saving user files with user defined password ?
However there are programs that include this possibility with AES. For example 7zip offers AES-256 encryption http://www.7-zip.org/7z.html.
2) Is there a standard way to convert a 8 chars user defined password into an useful 256 AES bits key ?
|
1) Cryptography is no miracle, it's just math. If you use weak password, you get weak protection, and no algorithm saves you from it. This means if you want stable and strong protection, you must use stable and strong key. The one of few options here is to use key generated by OpenSSL itself.
You have to consider user password as worst type of key. Also, many passwords are too plain and dumb: 123, 111, 123456, etc. This is bad for cryptography.
Russian programmer Igor Pavlov who wrote 7zip, has chosen to use compromise solution. He takes user password, calculates SHA-256 function for it (AFAIK U++ has its realization too). Then he adds some calculations/changes to that 256-bit value and the final value is used as a key for AES encryption.
This represents fair protection, which is very much stronger than using user password as key, but at some rate weaker protection than with OpenSSL-generated key. In a number of uses it is rather good and satisfactory protection. Also it allows using protection without storing user password itself which is very good practice. But frankly speaking I haven't heard of SHA output as extremely cryptographically strong combination of bytes. This algorithm has another application field (generating unique digest "far" from original bytes).
2) AFAIK there is no "standard" way to convert user password to key. The best way is to use OpenSSL generated key. You may of course use any function like SHA-256 but you must be aware of the crytpographic strongness/weakness you give to user.
[Updated on: Mon, 22 February 2010 08:43] Report message to a moderator
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #25403 is a reply to message #25402] |
Mon, 22 February 2010 08:50   |
 |
koldo
Messages: 3458 Registered: August 2008
|
Senior Veteran |
|
|
| Mindtraveller wrote on Mon, 22 February 2010 08:31 |
| koldo wrote on Mon, 22 February 2010 09:54 | 1) Does it mean that AES cannot be used for saving user files with user defined password ?
However there are programs that include this possibility with AES. For example 7zip offers AES-256 encryption http://www.7-zip.org/7z.html.
2) Is there a standard way to convert a 8 chars user defined password into an useful 256 AES bits key ?
|
1) Cryptography is no miracle, it's just math. If you use weak password, you get weak protection, and no algorithm saves you from it. This means if you want stable and strong protection, you must use stable and strong key. The one of few options here is to use key generated by OpenSSL itself.
You have to consider user password as worst type of key. Also, many passwords are too plain and dumb: 123, 111, 123456, etc. This is bad for cryptography.
Russian programmer Igor Pavlov who wrote 7zip, has chosen to use compromise solution. He takes user password, calculates SHA-256 function for it (AFAIK U++ has its realization too). Then he adds some calculations/changes to that 256-bit value and the final value is used as a key for AES encryption.
This represents fair protection, which is very much stronger than using user password as key, but at some rate weaker protection than with OpenSSL-generated key. In a number of uses it is rather good and satisfactory protection. Also it allows using protection without storing user password itself which is very good practice. But frankly speaking I haven't heard of SHA output as extremely cryptographically strong combination of bytes. This algorithm has another application field (generating unique digest "far" from original bytes).
2) AFAIK there is no "standard" way to convert user password to key. The best way is to use OpenSSL generated key. You may of course use any function like SHA-256 but you must be aware of the crytpographic strongness/weakness you give to user.
|
Excellent explanation 
I will follow your advice. Anyway, could you add a function to convert an username password into a "fair" protection ?. Thanks 
I have checked your demo and now it works well. In a big program where I have applied it, it works well too .
You have done more changes than just a fix . You have removed dependencies to packages Web and Web/SSL.
This afternoon I will upload it to Bazaar. In some hours I will propose a possible application of your useful functions.
Great job !
Best regards
Iñaki
|
|
|
|
|
|
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #25420 is a reply to message #25415] |
Mon, 22 February 2010 15:19   |
|
|
Koldo,
Can you ad in BAZAAR?
Than you Koldo, and Mindtraveller!
Regards, ion Lupascu (tojocky).
| koldo wrote on Mon, 22 February 2010 13:46 | Hello Mindtraveller
Your package is very useful.
If somebody requires to encrypt seriously a String or raw data from small size to Gb, this is a simple way to do it .
This is not an encryption algorithms catalog. This is just one of the best options with an easy interface.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #25762 is a reply to message #25758] |
Wed, 10 March 2010 20:54   |
 |
kohait00
Messages: 939 Registered: July 2009 Location: Germany
|
Experienced Contributor |
|
|
hi koldo,
as far as i got the point of mindtraveler, AES and the other symetric algorithms are not to be thought of beeing based on a "password", a user defined and therefore week combination of signs (which would be scanned first in a brute force attack), but on a statistically well distributed *binary* key (128 bit should be made wise . it is hard for a human beeing to generate one. so the computer will take over and provide some random ones(AES key generator). this key should be thought of as a "password", what it of corse isn't. everything else would diminish the stability of the key.
maybe to get over it, think of it as kind a GUID which you generate once for your application (which in real world communication does not apply . dont think of AES as sort of alphanumerical password dependant encryption algorithm, it's indeed, just as mindtraveler mentioned: math. i had the luck to enjoy some lectures cryptology, and it confuses sometimes. but the first thing we learned there was to forget the idea of passwords / human readable strings as security base.
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #25763 is a reply to message #25762] |
Wed, 10 March 2010 21:33   |
 |
koldo
Messages: 3458 Registered: August 2008
|
Senior Veteran |
|
|
| kohait00 wrote on Wed, 10 March 2010 20:54 | hi koldo,
as far as i got the point of mindtraveler, AES and the other symetric algorithms are not to be thought of beeing based on a "password", a user defined and therefore week combination of signs (which would be scanned first in a brute force attack), but on a statistically well distributed *binary* key (128 bit should be made wise . it is hard for a human beeing to generate one. so the computer will take over and provide some random ones(AES key generator). this key should be thought of as a "password", what it of corse isn't. everything else would diminish the stability of the key.
maybe to get over it, think of it as kind a GUID which you generate once for your application (which in real world communication does not apply . dont think of AES as sort of alphanumerical password dependant encryption algorithm, it's indeed, just as mindtraveler mentioned: math. i had the luck to enjoy some lectures cryptology, and it confuses sometimes. but the first thing we learned there was to forget the idea of passwords / human readable strings as security base.
|
Yes yes, all of you are right 
However think about for example a file encrypting software to be used by different people. How would you do it ?
Option 1: The software gives the user a 32 bytes random key
Option 2: The user enters a key
Option 1 seems much stronger. However file and hard disk encrypting software seems to choose option 2.
Best regards
Iñaki
|
|
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #25770 is a reply to message #25767] |
Thu, 11 March 2010 09:12   |
 |
koldo
Messages: 3458 Registered: August 2008
|
Senior Veteran |
|
|
Hello Kohait00
Thank you for the reference. I will use it.
Coming to the issue, look at this:
- If it is open source, I cannot put the key in the code
- If the program creates a key for the user, and he/she is not let to change it, a 32 bytes password seems too hard to use
- If we use a user defined key, we could include in AESStream:
---1. A SHA 256 possibility to convert user password in a 32 bytes key
---2. The means to avoid a brute force attack.
For example, if AES 256 with a weak user key can resist within and acceptable probability, for example, 1000000 random keys, AESStream could let the main program to enter, for example, 1000 keys per day and after that, AESStream would refuse any additional key.
Best regards
Iñaki
|
|
|
|
| Re: Encrypted storage with streaming (OpenSSL, AES) [message #25772 is a reply to message #25770] |
Thu, 11 March 2010 10:29   |
 |
kohait00
Messages: 939 Registered: July 2009 Location: Germany
|
Experienced Contributor |
|
|
i dont know if i remember it correctly, but there are several technices combined to achieve encryption of data trggered by a user password.
1) the en/de cryption is done using a *fast* (symetrical) algorithm, like AES (they are blockorientated and relatively similar, only differ in their block functions (F functions, or Feistel Function)
2) the key used there, is the key we were speaking about, and is encrypted and stored with the data. as encryption can be used slow but really strong asymetrical (public / private key) algorithms like RSA.
3) the password thing comes into play with things like diffie hellman secure exchage of information with having it travel over the net.
but its quite a while now, and i may mix it up with things like vpn tunneling and handshaking and so on..
but in any way: encrypting decrypting to fit current standards is far from beeing trivial and involves a lot of steps, password is only a small part of it, maybe we should stick to common technologie here (means in openssl)
|
|
|
|
|
|
|
|
Goto Forum:
Current Time: Mon May 04 00:56:02 GMT+2 2026
Total time taken to generate the page: 0.01121 seconds
|