Real-time socket

[daves]

I have implemented my own TCP library so my Windows application can get and set blocks of user shared memory (basically a Dual Port Ram emulator).

 

This works well. However I am seeing a process called 'gatekeeper/0' running at high CPU (20%). I have read up on xenomai real-time socket programming and discovered I am having a lot of unwanted Mode SWitches. The recommended approach is to use the RTDM functions (access the drivers using rt_dev_* functions).

 

The rtdm.h header appears on the PC folders:

 

"C:\Program Files (x86)\Delta Tau Data Systems Inc\2.0\Power PMAC Suite\powerpc-460-linux-gnu\opt\eldk-4.2\debian_rootfs\usr\local\xenomai-2.4.10\include\rtdm"

 

and

 

"C:\Program Files (x86)\Delta Tau Data Systems Inc\2.0\Power PMAC Suite\powerpc-460-linux-gnu\opt\eldk-4.2\debian_rootfs\usr\src\xenomai-2.4.10\include\rtdm"

 

On the PPMAC it is in

 

"./usr/local/xenomai-2.4.7/include/rtdm/rtdm.h"

"./usr/local/x3/include/rtdm/rtdm.h"

"./usr/local/xenomai-2.4.10/include/rtdm/rtdm.h"

 

But I cannot get it to build.

 

I get errors of the form

 

"C:\ABD\PowerPMAC\SPMM\Centre\Centre\Centre.ppproj(249,5): Error : tcptest.o: In function `SetUpSocket':

C:\ABD\PowerPMAC\SPMM\Centre\Centre\C Language\Background Programs\tcptest\/cygdrive/c/ABD/PowerPMAC/SPMM/Centre/Centre/C Language/Background Programs/tcptest/tcptest.c(27,0): Error : undefined reference to `rt_dev_socket'"

 

and

 

"C:\ABD\PowerPMAC\SPMM\Centre\Centre\Centre.ppproj(249,5): Error : tcptest.o: In function `rt_dev_bind':

C:\ABD\PowerPMAC\SPMM\Centre\Centre\C Language\Background Programs\tcptest\/opt/eldk-4.2/debian_rootfs/usr/local/xenomai/include/rtdm/rtdm.h(389,0): Error : undefined reference to `rt_dev_ioctl'"

 

What do I need to do to code properly for xenomai?

 

Also I wonder if it possible to move to a xenomai version where rt_dev_select was added to RTDM (it happened in 2008 I think, although I see your xenomai is from 2009 so I don't quite get that).

 

Cheers

Dave

[daves]

Sorry, after some recoding I think the high CPU usage was due to inefficiency in my code.

 

This post can probably be ignored unless out of interest you think I should have been able to compile the RTDM stuff (might it be necessary when code complexity or my network traffic increases?)

 

[hbausley]

Currently the IDE only generates a makefile for use with the POSIX skin. To work with the RTDM skin the -lrtdm option would have to be added to the makefile the IDE generates. If you had to have that skin right now you would have to manually add the -lrtdm to the makefile, then issue make from a command prompt, followed by manually copying the file to the PowerPMAC.

[shansen]

Dave,

 

I would be extremely interested in seeing your socket code, or just an example if you don't want to share the full code.

 

Regards,

 

Steven

[daves]

Hi Steven

 

I would like to help out but I am unsure what my company would think about giving all the code out (my boss is away). Plus I am still finding and fixing bugs in it so wouldn't necessarily want to inflict it on anyone!

 

I will try and put together an outline of what it does without giving away our 'secrets' until I find out about publishing some code.

 

When I get chance I will reply again...

 

Dave

[hbausley]

Dave,

 

I would be extremely interested in seeing your socket code, or just an example if you don't want to share the full code.

 

Regards,

 

Steven

 

Attached is a simple IDE project udptest.zip that transfers data from user shared memory via UDP sockets. It contains a client and server that can be run from the PowerPMAC. It may be a good starting point for some test.

 

 

 

[shansen]

Dave,

 

I have tried out the demo that Henry posted. I am having several problems that you may have experience with:

 

1) The PPMAC CPU usage is going up to 80% or higher when I call the socket often (every 2 ms), what was your fix for this issue?

 

2) I am getting about 12-14 MBytes/sec data transfer rate over gigabit Ethernet. This is only about ~14% of the realizable speed of gigabit, so I am curious if you are getting similar speeds. To do the benchmark, I transferred 8192, 16384, and 32767 bytes from the user shared memory buffer using the GET_MEM emulator.

 

3) Did you try out the RTDM sockets? I am trying to find out if there are any performance benefits to the real-time sockets.

 

Regards,

 

Steven Hansen

[daves]

1) My TCP server logic structure is very different from the UDP example (see below). I modified the UDP example to call every 2ms and only saw 10% CPU, this was using the loopback client. I'm not sure how you made your test.

 

2) I made the UDP example transfer 32767 bytes (I had to enlarge the buffers to allow this, I assume you did too. My TCP program splits large commands into 1400 byte packets). This gave an apparent rate of 10MBytes/s, but it is hard to know where to do the timings on this to get a representative value. I have not got those numbers on my server performance but I am running 6hr intensive test (~10x my requirements) without problem.

 

3) I gave up on the RTDM sockets as after I fixed a bug I was seeing acceptable performance.

 

My TCP server uses some of the ideas in the UDP example. In particular I use EthernetCmdProc and the PETHERNETCMD structure Henry posted as the core processing of packet data (GetMem, SetMem, and I also implemented a GetResponse).

 

Here is my TCP architecture in pseudocode (there is a lot more error checking and breaking out of loops correctly). It took a long time and a lot of research to combine all the various examples on the web which seemed to deal with one aspect whereas for robustness you need all aspects!

 

--- INITIALISATION
signal(SIGPIPE, SIG_IGN); --- Ignore errors writing to broken pipe and take as disconnection

--- MAIN LOOP --- to recover from broken server socket
while (1)
{
 --- Set up socket
   socket(AF_INET, SOCK_STREAM, 0)
   bind()
   listen()

 --- Server is only active socket to listen to at first
   FD_ZERO(&sactive);
   FD_SET(sockfd, &sactive);

 --- INNER LOOP --- to wait for data/keep track of clients
 while (1)
 {
   --- Check for data (no timeout gives low CPU...)
   sready=sactive --- select on copy of active set as it gets modified
   select(FD_SETSIZE, &sready, NULL, NULL, NULL)

   for (i = 0; i < FD_SETSIZE; i++)
     if (FD_ISSET(i, &sready)) --- Data from someone
       if (i == serversock) --- It's a client talking to the server
         accept
         FD_SET(csock, &sactive); --- Store as an active client
       else --- It's a message from an existing client
         read()
         EthernetCmdProc
         write()
         --- FD_CLR() on any failures to treat as disconnect
 }
}

 

The key points are

 

A) understanding the select() mechanism, I would try and google it as much as possible. Of course there are other mechanisms (multi-threaded, forking) but I found the blocking select() approach to be very good.

 

B) Using a fd_set to keep track of clients.

 

C) Handling all error conditions and dropping back to a recovery state, I want my server to run on PPMAC and never need to be manually restarted no matter what abuse comes down the network. I use a combination of all errors I've seen on my google-sourced examples (the examples usually handle one type of error)

 

I hope this makes some sort of sense and helps you on your way...

 

Dave

[daves]

Hi Henry

 

I think I remember this UDP example being developed with/for us and I see our company name still in one of the comments!

 

I also spotted a bug on line 134 of the host:

 

   memcpy((unsigned *) + pEthCmd->wValue,pEthCmd->bData,pEthCmd->wLength);

 

I think should be

 

   memcpy((unsigned *)pushm + pEthCmd->wValue,pEthCmd->bData,pEthCmd->wLength);

 

Otherwise you get a segmentation fault on the first SETMEM

 

Dave