Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Community » Newbie corner » Socket Communication 101
Socket Communication 101 [message #38236] Fri, 07 December 2012 18:02 Go to next message
nejnadusho is currently offline  nejnadusho
Messages: 60
Registered: October 2012
Member
Hi,

I need to figure out sockets.
And I have no much knowledge in that.

I run the reference examples and the communication works great and I understood what is going on both sides.

However, I still cannot imagine/assimilate how to transfer large chunks of data.

For example:
If a query to a database is returning an array of data how do I send it to the client side as one chunk?
Also how can I lable/mark that chunk to which class in my application is the receiver?


I already and I assume, but not sure, I can send an array with RawSend/RawRecv, then shoud I initialize/end RawOpen/RawClose the connection in a specific way?

Thank you very much.

Best,
Georgi

P.S. Please let me know if I should try to clarify myself.
Re: Socket Communication 101 [message #38237 is a reply to message #38236] Fri, 07 December 2012 18:20 Go to previous messageGo to next message
nlneilson is currently offline  nlneilson
Messages: 644
Registered: January 2010
Location: U.S. California. Mojave &...
Contributor
You question was clear.
What ever is sent over a socket, internet, is raw data AFAIK.
Just bits.
nejnadusho wrote on Fri, 07 December 2012 09:02

However, I still cannot imagine/assimilate how to transfer large chunks of data.

Remove any \0 from the string you will be sending except at the end.

http://www.ultimatepp.org/forum/index.php?t=msg&th=7136& amp; amp;start=0&

[Updated on: Fri, 07 December 2012 18:28]

Report message to a moderator

Re: Socket Communication 101 [message #38238 is a reply to message #38236] Fri, 07 December 2012 18:52 Go to previous messageGo to next message
nejnadusho is currently offline  nejnadusho
Messages: 60
Registered: October 2012
Member
Neil,

I am still very far from understanding how to send an array of information.

I mean I literally want to send an array through.

Something like

Array arr;

loop start{
  arr = fill with data
}

TcpSocket sock;

sock.RawSend(arr);


Is it possible or I am just a dreamer Rolling Eyes ?


Thank you very much.

Best,
Georgi
Re: Socket Communication 101 [message #38242 is a reply to message #38236] Fri, 07 December 2012 19:29 Go to previous messageGo to next message
lectus is currently offline  lectus
Messages: 329
Registered: September 2006
Location: Brazil
Senior Member
I'm no expert in U++ or Sockets or even C++.

But this is what I think (correct me if I'm wrong):

Fill the array:
Array<String> FruitList;

FruitList.Add("Apple");
FruitList.Add("Banana");
FruitList.Add("Orange");


Design your own protocol and send data:
TcpSocket sock;
// Do some initialization of sock here

sock.Put("-- BEGIN FRUIT LIST --\n");

for(int i = 0; i < FruitList.GetCount(); i++)
sock.Put(FruitList[i] + '\n');

sock.Put("-- END FRUIT LIST --\n");


On the other side you have to treat what's received like:
String line = sock.GetLine();

if(line == "-- BEGIN FRUIT LIST --")
{
while(line != "-- END FRUIT LIST --")
{
line = sock.GetLine();
FruitList.Add(line);
}
}


I think to send a full array U++ would require some kind of serialization of objects. I don't know if it already has it.

Wait for someone more knowledgeable. Smile
Re: Socket Communication 101 [message #38243 is a reply to message #38242] Fri, 07 December 2012 19:45 Go to previous messageGo to next message
nlneilson is currently offline  nlneilson
Messages: 644
Registered: January 2010
Location: U.S. California. Mojave &...
Contributor
sock.Put("-- BEGIN FRUIT LIST --\n");

for(int i = 0; i < FruitList.GetCount(); i++)
sock.Put(FruitList[i] + '\n');

sock.Put("-- END FRUIT LIST --\n");

Each sock.Put would be a different connect/disconnect, not good.

Add all the char to a single string or buf with the \0 at the end and send that.
One connect/disconnect

[Updated on: Fri, 07 December 2012 19:48]

Report message to a moderator

Re: Socket Communication 101 [message #38244 is a reply to message #38243] Fri, 07 December 2012 20:14 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
nlneilson wrote on Fri, 07 December 2012 13:45

sock.Put("-- BEGIN FRUIT LIST --\n");

for(int i = 0; i < FruitList.GetCount(); i++)
sock.Put(FruitList[i] + '\n');

sock.Put("-- END FRUIT LIST --\n");

Each sock.Put would be a different connect/disconnect, not good.




What makes you think that?

Mirek
Re: Socket Communication 101 [message #38246 is a reply to message #38236] Fri, 07 December 2012 20:41 Go to previous messageGo to next message
lectus is currently offline  lectus
Messages: 329
Registered: September 2006
Location: Brazil
Senior Member
Since it's using TCP protocol, I believe there's a connection alive.

If you used UDP then yes, it doesn't use connections.

I think what you mean is that you can send it all in ONE chunk. Yes, it's also possible.

Maybe something like this:

String acc;
for(int i=0; i < FruitList.GetCount(); i++)
{
acc += FruitList[i];
acc += "-";
}
acc += '\n';

sock.Put(acc);


Then on the other side you can do:
String line = sock.GetLine();

String fruit;
for(int i=0;i < line.GetLength(); i++)
{
if (line[i] != '-')
fruit += line[í];
else
{
FruitList.Add(fruit);
fruit="";
}
}


Data gets sent like this:
Quote:

Apple-Banana-Orange


[Updated on: Fri, 07 December 2012 20:45]

Report message to a moderator

Re: Socket Communication 101 [message #38247 is a reply to message #38244] Fri, 07 December 2012 20:53 Go to previous messageGo to next message
nlneilson is currently offline  nlneilson
Messages: 644
Registered: January 2010
Location: U.S. California. Mojave &...
Contributor
In the for() loop
sock.Put(FruitList[i] + '\n');

it looks like a sock.Put is called each time through the loop or is each char sent through the same connection and then '\n' is sent after all the char been sent?

or is it ch \n ch \n ch \n

Also does String GetLine() stop at each \n or is it looking for \0

Maybe String GetAll(int len) will read a bunch of data including several \n until the max len is reached or \0

My client is in U++ and it will send a string with several \n and it is parsed after the server in the Java app.

edit: The last post by lectus where the data is added and that is sent is the way I do it.

[Updated on: Sat, 08 December 2012 00:14]

Report message to a moderator

Re: Socket Communication 101 [message #38248 is a reply to message #38247] Fri, 07 December 2012 20:57 Go to previous messageGo to next message
lectus is currently offline  lectus
Messages: 329
Registered: September 2006
Location: Brazil
Senior Member
nlneilson wrote on Fri, 07 December 2012 14:53

In the for() loop
sock.Put(FruitList[i] + '\n');

it looks like a sock.Put is called each time through the loop or is each char sent through the same connection and then '\n' is sent after all the char been sent?

or is it ch \n ch \n ch \n

Also does String GetLine() stop at each \n or is it looking for \0

Maybe String GetAll(int len) will read a bunch of data including several \n until the max len is reached or \0

My client is in U++ and it will send a string with several \n and it is parsed by the server in the Java app.


Each sock.Put sends a fruit name indexed as FruitList[i] with an ending \n.

It uses the same connection started by sock.Connect().

On the other side the sock.GetLine() reads everything until it finds \n and stores in a string without the \n.

Unless you're dealing with binary data I see no reason to not use strings ended with \n.
Re: Socket Communication 101 [message #38250 is a reply to message #38248] Fri, 07 December 2012 22:08 Go to previous messageGo to next message
nlneilson is currently offline  nlneilson
Messages: 644
Registered: January 2010
Location: U.S. California. Mojave &...
Contributor
lectus wrote on Fri, 07 December 2012 11:57

1. It uses the same connection started by sock.Connect().

2. On the other side the sock.GetLine() reads everything until it finds \n and stores in a string without the \n.

3. Unless you're dealing with binary data I see no reason to not use strings ended with \n.


1. That makes sense.

2. And that must be under while data waiting or similar.

3. I use \n but it is with the \0 that ends the send and receive.

The data is passed to the client to send as a string with \0
The server receives as a string with the \0

Nothing is added or combined in the client and nothing is parsed in the server. That is done with error handling outside.

Less time in the socket and better error handling.
Re: Socket Communication 101 [message #38253 is a reply to message #38236] Sat, 08 December 2012 00:42 Go to previous messageGo to next message
lectus is currently offline  lectus
Messages: 329
Registered: September 2006
Location: Brazil
Senior Member
Lower level sockets work with \0 and buffer sizes.

GetLine is a higher level function which parses the string until it finds \n. It's good for protocols based on lines of string such as HTTP protocol where you need to send a \n at the end.

[Updated on: Sat, 08 December 2012 00:43]

Report message to a moderator

Re: Socket Communication 101 [message #38255 is a reply to message #38253] Sat, 08 December 2012 02:36 Go to previous messageGo to next message
nlneilson is currently offline  nlneilson
Messages: 644
Registered: January 2010
Location: U.S. California. Mojave &...
Contributor
It has been several years since tinkering with different code for sockets so my memory is a bit fuzzy.

What you mentioned about "Lower level sockets work with \0 and buffer sizes" may be a clue as why \0 seems important to me.

I still do much of my parsing a character at a time rather than split at ','. The time difference now with faster CPUs is negligible.

I think it was back using Python for a client the Sleep time before data was out of order or garbled was 2 milli second on a dual core and 5 on a slower single core when using \0 as end of transmission. Any other way the time would be more. The server is on a thread and can accept several clients and have tried up to 5 without a problem except when trying to open and close the connections too fast. The amount of data sent doesn't make much difference as it is so fast from a few bytes to a few thousand as long as any parsing is not done in the server code.

There may be better ways like shared memory but that is above me.

It's good to go over stuff again and have my memory corrected when necessary.
Re: Socket Communication 101 [message #38262 is a reply to message #38236] Sat, 08 December 2012 15:37 Go to previous messageGo to next message
lectus is currently offline  lectus
Messages: 329
Registered: September 2006
Location: Brazil
Senior Member
Yes, there's some optimization at CPU level when you compare a char with \0.

In ASM code a comparison happens like this:
CMP AX, ','
JZ FOUND_COMMA
.
.
.
FOUND_COMMA:
; Do something...


When you compare to zero it happens like this:
CMP AX, 0
JZ FOUND_ZERO
.
.
.
FOUND_ZERO:
; Do something...


The same with some instruction optimization:

XOR AX, AX
JZ FOUND_ZERO
.
.
.
FOUND_ZERO:
; Do something...


XOR instruction is a few bytes smaller than CMP, but XOR raises a CPU flag and so it works to compare to zero.
This saves a few CPU cycles. But with a good compiler and modern CPUs it becomes irrelevant.
Re: Socket Communication 101 [message #38264 is a reply to message #38262] Sat, 08 December 2012 16:48 Go to previous messageGo to next message
nlneilson is currently offline  nlneilson
Messages: 644
Registered: January 2010
Location: U.S. California. Mojave &...
Contributor
Thanks for the information.

I remember a Radio Shack from many years back. Very slow and the amount of memory was so small it hooked up to a cassette player to save bigger files.

I just got used to using \0 for end of transmission.
To be able to see as a string what is passed to be sent and what is output from the server was an advantage as nothing was done in the client or server except send and receive. With the C++ client in a header file and the Java server in it's own file it was easier to understand in different applications.

I looked at assembly language years ago but it was more than I wanted to get into.

I also got hung up on memory and saving space. External hdd are cheap now and recently picked up a 32GB micro SD card for $10 so spending too much time concerning memory is irrelevant also.

Times change and I am a bit slow keeping up and also hard headed to change my ways if an old way works.

Using U++ has been great to see what can be done and easier.
Re: Socket Communication 101 [message #38287 is a reply to message #38236] Sun, 09 December 2012 05:11 Go to previous messageGo to next message
nejnadusho is currently offline  nejnadusho
Messages: 60
Registered: October 2012
Member
nlneilson, lectus,


Thank you very much to both of you, for the great help.

I just want to say that this thread is not 101 anymore.
It should be more like 101-102.

It became very in depth discussion which I am so happy I have been learning a lot from you. Smile




Best,
Georgi
Re: Socket Communication 101 [message #38403 is a reply to message #38244] Fri, 14 December 2012 10:07 Go to previous messageGo to next message
nlneilson is currently offline  nlneilson
Messages: 644
Registered: January 2010
Location: U.S. California. Mojave &...
Contributor
mirek wrote on Fri, 07 December 2012 11:14

nlneilson wrote on Fri, 07 December 2012 13:45


Each sock.Put would be a different connect/disconnect, not good.




What makes you think that?

Mirek


Let me clarify what I meant.
Once a client and a server get connected that is fine and can stay connected even when nothing is being sent.

"Each sock.Put would be a different packet."
instead of connect/disconnect

AFAIR, going back to tinkering with sockets in Python, the amount of data sent in a large number of packets before errors is much less than the same amount of data sent in a single packet in the same amount of time.

Maybe something similar is uploading data to a server.
FileZilla does not have the ability to zip/unzip.
I tried uploading a directory/sub directories with several hundred files, took forever. The zip size is very close to the unzipped.
Zip it up and it went fairly fast and on the server I could unzip it with cpanel.

The biggest chunk I usually send is about 1GB and over 16,000 files. This of course is not a "socket" but it may be comparable to some extent.

I don't have any test data from before but I think the large packet was much faster than a large number of packets.

Maybe lectus has some information on this.
Re: Socket Communication 101 [message #38432 is a reply to message #38403] Sat, 15 December 2012 16:37 Go to previous messageGo to next message
nlneilson is currently offline  nlneilson
Messages: 644
Registered: January 2010
Location: U.S. California. Mojave &...
Contributor
This is a GPS GGA NEMA sentence:
$GPGGA,175226.000,3502.264328,N,11758.158283,W,2,8,,810.661, M,-31.8,M,,*73

This is stripped of everything except the latitude, longitude and altitude.
Then an ID such as a planes tail number and position number relative to a pilots aircraft.
35.030438,-118.167888,828.3,N10801,1
35.034438,-118.173888,1438.3,N10802,2
35.036438,-118.158888,1133.3,N10803,3
35.038438,-118.179888,1438.3,N10804,4
35.040438,-118.152888,1133.3,N10805,5
.........

To send this data with 50 lines each on individual packets with sock.Put(...) every second does not work without errors.

Adding all 50 lines and stripping the \0 except at the end it can be sent in one packet every second without errors or problems.
Re: Socket Communication 101 [message #39663 is a reply to message #38432] Tue, 16 April 2013 03:54 Go to previous messageGo to next message
nlneilson is currently offline  nlneilson
Messages: 644
Registered: January 2010
Location: U.S. California. Mojave &...
Contributor
Spent some time optimizing code because of a sprained foot.
Some data was being transferred from a Upp app to a Java app through a socket as a character array.

Some survey paths were still being sent individually and Sleep(5); was necessary on a fast machine between each packet for each path or data would get corrupted or garbled. Sleep(10); was necessary on a slower machine.

Putting the data for 30 paths (60 lines) into a character array (about 2000 size) it was almost instantly transferred, no Sleep time necessary.
Re: Socket Communication 101 [message #39664 is a reply to message #39663] Tue, 16 April 2013 08:22 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
That is kinda weird, my understanding is that kernel always buffers data before forming bigger packets, e.g.:

http://books.google.cz/books?id=ptSC4LpwGA0C&pg=PA58& ;lpg=PA58&dq=kernel+socket+buffers&source=bl&ots =Kr6DLpdkNs&sig=oqyxevI71j7yqDxnrRhq4heArik&hl=en&am p;sa=X&ei=qe1sUaaxJ4mNtQbn04H4Bg&sqi=2&ved=0CG8Q 6AEwCA#v=onepage&q=kernel%20socket%20buffers&f=false
Re: Socket Communication 101 [message #39671 is a reply to message #39664] Tue, 16 April 2013 19:51 Go to previous messageGo to previous message
nlneilson is currently offline  nlneilson
Messages: 644
Registered: January 2010
Location: U.S. California. Mojave &...
Contributor
Thanks for the link, I saved that and also the document so it can be read off line.

Yes the data does get buffered but the difference was in my code.
Before it was like this:

getline
reformat
send

That was one packet.

getline
reformat
send

That was another packet.

etc. for each line.

It is the time required to send each individual packet that is important before they get corrupted or out of order.


Now the code is like this:

getline
reformat, keep the \n but remove the \0
add to buffer
get next line
reformat, keep the \n but remove the \0
add to buffer

etc until all lines are added to buffer
add \0
send

That is one packet

The amount of data or size of the buffer is irrelevant at least for the amount I have been working with.
By the time the Upp file choose box disappears all paths are displayed in the Java app.

Upp is great to be able to make changes.

[Updated on: Tue, 16 April 2013 19:54]

Report message to a moderator

Previous Topic: ScatterCtrl / ScatterDraw - ShowLegend() problem and general Question
Next Topic: Spin + Text
Goto Forum:
  


Current Time: Thu Mar 28 19:20:55 CET 2024

Total time taken to generate the page: 0.01699 seconds