Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Developing U++ » UppHub » Encrypted storage with streaming (OpenSSL, AES)
Encrypted storage with streaming (OpenSSL, AES) [message #23087] Wed, 16 September 2009 22:17 Go to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

Sometimes we may have task to store some large file (4+ GB) or small string inside encrypted storage. I tried to make a pair classes which make it easy. This is the first version, so any ideas are welcome.

This package assumes you have OpenSSL library successfully installed and its paths are added to TheIDE.

OK, let me introduce a pair of classes called AESEncoderStream and AESDecoderStream. They support streamed adding and encryption/decryption of data. Encryption is made with AES (Rijndael) with 128, 192 or 256 bit keys.

Encrypted data 32 bytes larger than source length aligned to 16-byte boundary. I.e. if your source data is 170 bytes long, the resulting length is:
170 rounded by 16-byte pieces = 176
plus
32 (header data)
= 176 + 32 = 208 bytes.
Not so ugly for a number of applications especially if source data is large.

Here is a simple self-explanating demo:
#include <Core/Core.h>
#include <openssl/aes.h>
#include <AESStream/AESStream.h>

using namespace Upp;

CONSOLE_APP_MAIN
{
	AESInit();

	// Generate cryptographically stable key
	String key(AESRandomString(32));

	// Encryption
	String sIn,sOut;
	sIn = "qwertyuiop[p\tasdfghjkl;zxcvbnm,./quwiueqiwueoiquweioquweioquweiqwueicuwinuqiweqiwue	pqiueci	eiqniuriryuweyruweyruewrycuwbrurbywuyrwquiercbbcrebrquwey";
	AESEncoderStream aesEncoder(sIn.GetLength(), key);
	aesEncoder << sIn.Left(10);
	aesEncoder << sIn.Mid(10,10);
	aesEncoder << sIn.Right(sIn.GetLength() - 20);

	sOut << aesEncoder; //do streamed encoding
	
	// Decryption
	//key.Set(0, 'a'); //uncomment to see what happens with wrong key
	AESDecoderStream aesDecoder(key);
	
	aesDecoder << sOut.Left(15); //you may add by parts
	aesDecoder << sOut.Right(sOut.GetLength() - 15);
	
	try
	{
		String sDecoded;
		sDecoded << aesDecoder; //throw exception if key is wrong
		
		Cout() << (sDecoded == sIn) << "\n\n"; //check if all converted successfully
	}
	catch (const char *xp)
	{
		Cout() << "\n!!Error: " << ToSystemCharset(xp);
	}
}

[Updated on: Fri, 18 September 2009 11:16]

Report message to a moderator

Re: Encrypted storage with streaming (OpenSSL, AES) [message #23103 is a reply to message #23087] Thu, 17 September 2009 09:34 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Hello Mindtraveller

How are you installed OpenSSL in Windows ?

I have seen an installer in here http://www.slproweb.com/products/Win32OpenSSL.html but I would like to know your opinion.

Best regards
Koldo


Best regards
Iñaki
Re: Encrypted storage with streaming (OpenSSL, AES) [message #23120 is a reply to message #23103] Thu, 17 September 2009 19:57 Go to previous messageGo to next message
Weras is currently offline  Weras
Messages: 4
Registered: August 2009
Junior Member
Hi koldo!

I had resolved this problem and I did so.
1. Download openssl-0.9.8k.tar.gz from
https://www.openssl.org/source/
2. Unpack archive to C:\temp\openssl-0.9.8g
3. Download and install ActivePerl
4. Now type in the command line:
1) C:\temp\openssl-0.9.8g>perl Configure VC-WIN32 --prefix=c:/temp/openssl-bin/
2) C:\temp\openssl-0.9.8g>%comspec% /k ""c:\Program Files\Microsoft Visual Studio 8\VC\vcvarsall.bat"" x86
3) C:\temp\openssl-0.9.8g>ms\do_masm.bat
4) if you need static library write
C:\temp\openssl-0.9.8g>nmake -f ms\nt.mak
else, if you need dynamic library write
C:\temp\openssl-0.9.8g>nmake -f ms\ntdll.mak
5. The result is files*.lib & *.dll and include directory
6. Add .../openssl/inc32 as Include and .../openssl as Linker directories

Is enough to work with openssl

[Updated on: Thu, 17 September 2009 20:05]

Report message to a moderator

Re: Encrypted storage with streaming (OpenSSL, AES) [message #23130 is a reply to message #23120] Thu, 17 September 2009 22:55 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Hello Weras and Mindtraveller

Just excellent.

The Weras instructions worked perfectly the first time Smile.
Sorry to say that it is unusual.

And Mindtraveller wrapper and demo worked well the first time Very Happy .

Function dword rdtsc() uncludes a little of assembler so it only compiles with MSC. I will try to translate it to Gcc to see if it works with MinGW and Linux.

Best regards
Koldo


Best regards
Iñaki
Re: Encrypted storage with streaming (OpenSSL, AES) [message #23139 is a reply to message #23130] Fri, 18 September 2009 11:20 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

Hi, koldo!

It was Weras who instructed me how to install OpenSSL so I asked him to answer yur question.

Thanks for trying Linux version, it's a great effort.
I have just found a pair of small bugs and reuploaded sources. If you wish to try AESStream, pleas rewrite sources with new versions. If you manage to make POSIX version of rdtsc - it would be great too (I'm no professional in GCC-accepted assembler).

I wonder if people need this too. If it is so, someone from authors may upload AESStream to official Bazaar.
Re: Encrypted storage with streaming (OpenSSL, AES) [message #23143 is a reply to message #23139] Fri, 18 September 2009 12:28 Go to previous messageGo to next message
tojocky is currently offline  tojocky
Messages: 607
Registered: April 2008
Location: UK
Contributor

Mindtraveller wrote on Fri, 18 September 2009 12:20

Hi, koldo!

It was Weras who instructed me how to install OpenSSL so I asked him to answer yur question.

Thanks for trying Linux version, it's a great effort.
I have just found a pair of small bugs and reuploaded sources. If you wish to try AESStream, pleas rewrite sources with new versions. If you manage to make POSIX version of rdtsc - it would be great too (I'm no professional in GCC-accepted assembler).

I wonder if people need this too. If it is so, someone from authors may upload AESStream to official Bazaar.


Very nice package and wiki how to build openssl on win32.
Thank you!

Ion Lupascu (tojocky)
Re: Encrypted storage with streaming (OpenSSL, AES) [message #23144 is a reply to message #23087] Fri, 18 September 2009 13:23 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Hello Mindtraveller

About rdtsc, in fact it compiles in gcc as it has an ifdef so that:

- If MSC, it takes a value from rdtsc (time stamp counter 64-bit register)
- Else it takes Random()

It seems that to get a random number our Random() implementation is better than the clock (MT19937 algorithm), so perhaps rdtsc() would have to be changed with Random()

I have tried to compile in MinGW, but I get linking errors, in summary:

Openssl\out32\libeay32.lib(tmp32/ui_openssl.obj),(.text[_rea d_string_inner]+0xb): undefined reference to `__security_cookie'
Openssl\out32\libeay32.lib(tmp32/ui_openssl.obj),(.text[_rea d_string_inner]+0x149): undefined reference to `@__security_check_cookie@4'
Openssl\out32\libeay32.lib(tmp32/ecp_smpl.obj),(.text[_ec_GF p_simple_group_set_curve]+0x6): undefined reference to `_chkstk'

Does anybody know how to solve these problems with chkstk and security_cookie ?

Best regards
Koldo


Best regards
Iñaki
Re: Encrypted storage with streaming (OpenSSL, AES) [message #23145 is a reply to message #23144] Fri, 18 September 2009 14:01 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

Yes, I tried it too and had the same problems.
Currently had no time to discover this (actually MSC9 satisfies me on Win32), but some googling could help.

UPDATE: this link might be useful
http://wagner.pp.ru/~vitus/articles/openssl-mingw.html

[Updated on: Fri, 18 September 2009 14:09]

Report message to a moderator

Re: Encrypted storage with streaming (OpenSSL, AES) [message #23150 is a reply to message #23087] Fri, 18 September 2009 23:01 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Hello Mindtraveller

Thank you for your reference.

Unfortunately it has been impossible for me to compile it directly, with MSys or with Cygwin.
In any case sooner or later I get an error that finish the process. Sad

Best regards
Koldo


Best regards
Iñaki
Re: Encrypted storage with streaming (OpenSSL, AES) [message #23175 is a reply to message #23150] Wed, 23 September 2009 03:29 Go to previous messageGo to next message
kasome is currently offline  kasome
Messages: 78
Registered: July 2008
Location: Taiwan
Member
Great job. Surprised

Thanks, Mindtraveller.
Re: Encrypted storage with streaming (OpenSSL, AES) [message #25385 is a reply to message #23175] Sat, 20 February 2010 17:08 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Hello Mindtraveller

AESStream is not included in Bazaar yet.

I think it would be good to include it Smile


Best regards
Iñaki
Re: Encrypted storage with streaming (OpenSSL, AES) [message #25389 is a reply to message #25385] Sat, 20 February 2010 23:19 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

Please upload it. It looks like I have no access to bazaar.

Koldo, please tell, did you try to compile this under POSIX? Did it work well or you had any problems. It would be very good to keep this solution really cross-platform (at least MSC9/Win32 + GCC/POSIX).

[Updated on: Sat, 20 February 2010 23:20]

Report message to a moderator

Re: Encrypted storage with streaming (OpenSSL, AES) [message #25391 is a reply to message #25389] Sun, 21 February 2010 08:07 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Hello Mindtraveller

GCC/POSIX works perfect.

MSC9/Win32 works well for me only in non DEBUG mode.

When I run WIN32 version in DEBUG mode I get a "Heap leaks detected". I get the same using the openssl .dll version.

I think the problems come from Web/SSL package, not from AESStream.


Best regards
Iñaki

[Updated on: Sun, 21 February 2010 09:09]

Report message to a moderator

Re: Encrypted storage with streaming (OpenSSL, AES) [message #25392 is a reply to message #25391] Sun, 21 February 2010 10:38 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Hello Mindtraveller

I have got also a "Memory leaks detected" compiling uppdev/textssl with MSC in debug mode.

Its code is just:

#include <Core/Core.h>

CONSOLE_APP_MAIN
{ 
	
}


Best regards
Iñaki
Re: Encrypted storage with streaming (OpenSSL, AES) [message #25394 is a reply to message #25392] Sun, 21 February 2010 12:01 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

Koldo, what is the package uppdev/textssl? Never heard of that before. You mean another OpenSSL-based package?

I also changed package a bit, added tutorial (English and Russian versions), added example. Please be so kind check if it compiles (currently don't have OpenSSL installed) and upload it to bazaar.
  • Attachment: AESStream.zip
    (Size: 14.22KB, Downloaded 472 times)

[Updated on: Sun, 21 February 2010 12:08]

Report message to a moderator

Re: Encrypted storage with streaming (OpenSSL, AES) [message #25395 is a reply to message #25394] Sun, 21 February 2010 15:40 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Mindtraveller wrote on Sun, 21 February 2010 12:01

Koldo, what is the package uppdev/textssl? Never heard of that before. You mean another OpenSSL-based package?

I also changed package a bit, added tutorial (English and Russian versions), added example. Please be so kind check if it compiles (currently don't have OpenSSL installed) and upload it to bazaar.

Hello Mindtraveller

Web/SSL package is included in AESStream. A problem is it have memory leaks. I have found a solution to solve it, that is in Web/SSL/util.cpp:

INITBLOCK {
	Socket::Init();
	CRYPTO_set_mem_functions(SSLAlloc, SSLRealloc, SSLFree);
	SSL_load_error_strings();
	//SSL_library_init();		//SOLUTION TO MEMORY LEAK !!
}


Probably SSL_library_init(); is not necessary and as it not properly cleaned up, there are memory leaks.

To find this problem I have used uppdev/testssl package and Upp technology to detect memory leaks Smile

I have tested your changes but they do not work well Smile . Check this:

sIn includes this:
Quote:

qwertyuiop[p asdfghjkl;zxcvbnm,./quwiueqiwueoiquweioquweioquweiqwueicuwin uqiweqiwue pqiueci eiqniuriryuweyruweyruewrycuwbrurbywuyrwquiercbbcrebrquwey


sDecoded iincludes this:
Quote:

qwertyuiop[p asdfghjkl;zxcvbnm,./quwiueqiwueoiquweioquweioquweiqwueicuwin uqiweqiwue pqiueci eiqniuriryuweyruweyruewrycuwbrurbywuyrwquiercbbcrebr


Best regards
Iñaki
Re: Encrypted storage with streaming (OpenSSL, AES) [message #25396 is a reply to message #25395] Sun, 21 February 2010 16:01 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Hello Mindtraveller

MinGW works well using openssl .dll version.

Just add this to Web/SSL in "Package organizer":

index.php?t=getfile&id=2320&private=0

  • Attachment: sc.PNG
    (Size: 1.63KB, Downloaded 1900 times)


Best regards
Iñaki
Re: Encrypted storage with streaming (OpenSSL, AES) [message #25397 is a reply to message #25396] Sun, 21 February 2010 19:25 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Hello Mindtraveller

Other question. In the demo program you generate a key with AESRandomString(32).

I have integrated successfully AESStream in a program (very easy, just using your demo Smile ), but including in the program an array with the key previously generated with AESRandomString().

Is it possible to use instead an user generated key (a password) ?


Best regards
Iñaki
Re: Encrypted storage with streaming (OpenSSL, AES) [message #25399 is a reply to message #25397] Mon, 22 February 2010 00:48 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

Hello, Koldo.

I have found bug in AESStream which led to different original and decoded strings in the example. Also I got rid of Web/SSL dependency. Just tested it, and everything seems fine.

Please try updated version of AESStream and read it's tutorial if you want to use it in your app. Tutorial say that you should not use user password instead of generated key.

Why?
1) Key must be 128/192/256 bits long. User password may have ANY length.
2) Key is very important part of cryptographic strength of overall encryption. Using criptographically weak key (user password in 99,9% of cases is extremely weak) turns all AES encryption into weak and rather breakable system. As well as using cryptographically strong key makes overall AESStream system extremely unbreakable to anyone.
  • Attachment: AESStream.zip
    (Size: 14.29KB, Downloaded 505 times)
Re: Encrypted storage with streaming (OpenSSL, AES) [message #25401 is a reply to message #25399] Mon, 22 February 2010 07:54 Go to previous messageGo to previous message
koldo is currently offline  koldo
Messages: 3355
Registered: August 2008
Senior Veteran
Hello Mindtraveller

Quote:

1) Key must be 128/192/256 bits long. User password may have ANY length.
2) Key is very important part of cryptographic strength of overall encryption. Using criptographically weak key (user password in 99,9% of cases is extremely weak) turns all AES encryption into weak and rather breakable system. As well as using cryptographically strong key makes overall AESStream system extremely unbreakable to anyone.


Does it mean that AES cannot be used for saving user files with user defined password ?

However there are programs that include this possibility with AES. For example 7zip offers AES-256 encryption http://www.7-zip.org/7z.html.

Is there a standard way to convert a 8 chars user defined password into an useful 256 AES bits key ?


Best regards
Iñaki
Previous Topic: Protect packages - split code encryption,client and server
Next Topic: Added single and double linked lists
Goto Forum:
  


Current Time: Fri Mar 29 08:27:27 CET 2024

Total time taken to generate the page: 0.01453 seconds