Sendall Coordinate System Race Condition?

[abeard]

I'm receiving coordinate system runtime errors (Coord[n].RuntimeError) sporadically after many minutes to hours of running. We're running four motors in four separate coordinate systems. The four coordinate systems run an identical program which simultaneously moves the motors at the same time using external time base.

 

'list apc' is showing that the program counter was pointing at 'sendall' immediately following a 'send' command at the time of the abort in all four coordinate systems.

 

I've tried increasing Sys.PreCalc as recommended by other threads here to no avail. I also wrote a plc to monitor the high water mark of the Coord[n].BufferWarn variable - this never exceeded 0. There are also no additional errors such as FeLimit. 'getsends' is running on all communications ports being used.

 

I'm at a loss to explain the RuntimeError and am thinking now that perhaps this is due to a race amongst the four simultaneous calls to 'sendall'. Any help or insight would be greatly appreciated.

[Omron Forums Support]

Using "Sendall" in a realtime routine might cause this if your Sys.RtIntPeriod=0, you have a high servo rate, and a lot of text being sent out.

 

Can you please try implementing process locks around the "sendall" command in your motion program?

 

The syntax would look like this:

 

while(Sys.Lock[0] == 1){}  // Wait for this program instance's turn to use "sendall"
sendall; 
Sys.Lock[0] = 0; // Release the process control

 

If you have a lot of these in your program, you can make a subprogram call something like:

 

open subprog SafeSendall(locknumber)
while(Sys.Lock[locknumber] == 1){}  // Wait for this program instance's turn to use "sendall"
sendall; 
Sys.Lock[locknumber] = 0; // Release the process control
close

 

and then call it with

 

call SafeSendall(0); // 0 is the desired lock number, can be 0 to 31

 

See page 594 for more details on process locks.

 

An alternate solution would be to place small timer delays between SEND commands. A timer subprogram we use often in the office is below:

 

OPEN SUBPROG Timer(delay_time) // Open subroutine buffer
LOCAL EndTime; // Local variable
EndTime = Sys.Time + delay_time; // Store current time + user specified delay
WHILE (EndTime > Sys.Time){}; // Wait until real time reaches stored value
return
close

 

You can call it like

 

call Timer(0.001); // Wait 0.001 seconds

 

This is better than using a Dwell since it will not affect motion sequencing if you keep the amount of wait time very small. You may need to increase Coord[x].GoBack (takes values of 0 to 255) to use this in a motion program without breaking blending. Same applies to waiting for the locks in a while loop.

[abeard]

Thanks for the recommendations Charles. I went ahead and tried your first suggestion but to no avail. Oddly, all 4 coordinate systems stopped at the same time with "CoordExecStatus[n] - Stopped at breakpoint...", this is despite having no breakpoints defined.

 

I have verified that the sendall is the culprit - removing it removes the symptoms at least.

 

If I get a chance, I'll try the timer suggestion as well. Thanks.

[Omron Forums Support]

OK; sounds good. Could you also post a short excerpt of the kind of code that is causing you a problem? I can test it here on my system.

[abeard]

Could you also post a short excerpt of the kind of code that is causing you a problem? I can test it here on my system.

 

It's pretty simple, basically just an infinite loop with a move, delay, send and sendall. The same code runs in four different coordinate systems with a single motor each. All coordinate systems are synchronized with a single time-base trigger and remain synchronized via an external time-base such that all four motors move at exactly the same time (down to the same servo period AFAICT).

 

// ... Setup time-base trigger & arming...
while ( done != 1 ) 
{
   // ... calculate move pos...

   x(movePos) ta(moveTime/2) td(moveTime/2) ts(0) tm(moveTime)
   delay(delayTime)
   send 0 "This is a status update"
   sendall 
}

 

This will run for random amounts of time before triggering the runtime-error (I've seen anywhere from 5 to 45 minutes). getsends is always running, etc...[/code]

[Omron Forums Support]

Hi again,

 

OK, I have been playing with your code throughout the day and experienced program failures as well. My initial suggestions do not seem to help.

 

I think due to the nature of the send and sendall commands, it might not be a good idea to use them in a motion program directly. Instead, I propose that you set a csglobal flag in the motion program that indicates to a waiting background Script PLC to send your status updates via send and sendall. Is that a possibility for your architecture? That way, the realtime routine isn't used to perform a task (sending the string) that is time-consuming and does not fit well in a realtime routine, and instead the asynchronous background PLC can take care of it.

 

For example, the motion program:

 

open prog SendFlagProg
linear inc;
movePos=1; // user units
moveTime = 200; // msec
enable plc SendPLC;
while (done == 0) 
{
   x(movePos) ta(moveTime/2) td(moveTime/2) ts(0) tm(moveTime)
   ReadyToSend == 1; // Indicate to waiting PLC that the motion program wants to send an update
   delay(delayTime)
// PLC then sets this flag to 0 after sending the update
}
close

 

And the PLC:

 

open plc SendPLC
local CSNumber = 1; // Used to index coordinate system numbers
while(CSNumber < 5)
{
Ldata.Coord = CSNumber; // Change this PLC's coordinate system number
if(ReadyToSend > 0)
{
	send 0"Coordinate System %u Sends a Status Update",CSNumber;
	sendall;
	ReadyToSend = 0;
}	
CSNumber++;
if(CSNumber == 5)
	CSNumber = 1;
}



close

 

And the global definitions:

 

csglobal moveTime = 1;
csglobal movePos = 1;
csglobal delayTime = 10;
csglobal done = 0;
csglobal ReadyToSend = 0;

[abeard]

OK, I have been playing with your code throughout the day and experienced program failures as well. My initial suggestions do not seem to help.

 

Great! I'm glad to hear you can reproduce this.

 

I think due to the nature of the send and sendall commands, it might not be a good idea to use them in a motion program directly. Instead, I propose that you set a csglobal flag in the motion program that indicates to a waiting background Script PLC to send your status updates via send and sendall. Is that a possibility for your architecture?

 

Yes this is possible, sounds reasonable and will be easy to implement. My only question is whether or not I need to worry about the plc sporadically dying for the same reason?

 

Thanks for the code samples too.

[Omron Forums Support]

I'm virtually certain the PLC will not fault out. I will let a test run all day on my box here in the office and let you know at the end of the day.

[abeard]

I'm virtually certain the PLC will not fault out. I will let a test run all day on my box here in the office and let you know at the end of the day.

Cool. This is a small data point, but I've ran all day off your recommended plc with no failures.

[Omron Forums Support]

As have I. 22290 seconds and counting. Looks like this is an acceptable solution!

 

Let me know if you have any further issues.