Intermittent $1000a EtherCAT errors

[rsipkema]

L.S.,

I'm having an issue where after some period of time (often multiple hours) I get an EtherCAT error, i.e., ECAT[0].Error=$1000a. I can't seem to find any documentation on what this error means. I have seen Acontis errors show op in ECAT[i].Error, and a list of those is in the IDE manual, but this value is not amongst those. Can anyone provide me with a clue as to what this error means and/or provide any pointers on how to fix this issue?

My setup: CK3E with firmware 2.7.1.0, IDE 4.6.1.12, 2x KEBA ServoOne three-axis drives in CSP mode @ 1 kHz, several Beckhoff I/O terminals and an EL6601 Beckhoff Ethernet terminal.

[rsipkema]

The problem turned out to be a bad network cable, but I would still be interested in a more extensive list of possible errors which can show up in ECAT[i].Error and their meaning.

[Omron Forums Support]

The full code is $9811000A, i.e. the "981" is dropped in that response.  So, in the table, $1000a is given as 

       0x9811000A
       EC_E_NOMEMORY

       Not enough allocatable memory available (memory full / corrupted).

I attached a more detailed list.

EcMaster_31103 ErrorCodes 1.pdf

[Sanjay_RSS]

I'm seeing this $1000A error popping up in my CK3M system as well.

It seems to be somehow correleated with cutter-comp being turned on.
Right after a "ccmode1" or "ccmode2" in a program, the Ecat[0].DcClockDiff jumps to a very large time (bigger than my Sys.ServoPeriod) and then is pulled back to the normal band via the Ecat[0].DcRefPlus & Ecat[0].DcRefMinus

After a few of these spikes, I end up seeing this $1000A in the Ecat[0].Error register even though everything else seems to be functioning normally.. and all the other Ecat error / status registers are mostly normal. I do see a bit wierd status (Ecat[0].Status[0]) value of $2000200 .. which is bit 25 & bit 9 on. As per the PowerPMAC manual, bit 25 is ReserveErr10 and bit 9 = FrameResponseErr .. Also, I see the Ecat[0].MaxSemTime rise to 480 - 520 usec (typically its around 40-50 usec max).

Any idea why turning on cutter-comp would trigger a spike in the Ecat clock and how to prevent it?

[Omron Forums Support]

We might learn something by monitoring the serial port.  It seems like something is messing with PMAC's memory.

[rsipkema]

13 hours ago, Gregs said:

The full code is $9811000A, i.e. the "981" is dropped in that response.  So, in the table, $1000a is given as 

       0x9811000A
       EC_E_NOMEMORY

       Not enough allocatable memory available (memory full / corrupted).

Thanks for the info. So if I understand correctly, it is possible for ECAT[i].Error to get filled with an error code from the Acontis EtherCAT master at any time (when EtherCAT is running)? I'm pretty sure I have also seen an error starting with 981 in there sometimes; does it always drop the 981 for specific errors in the list?

[Sanjay_RSS]

Further troubleshooting confirmed that the issue was the cutter-comp-radius commands creating a momentary blip while doing the initial pre-calculations when a ccmode1 / ccmode2 statement is encountered.

Our normal settings have been Coord[x].CCSize = 20 and Coord[x].CCDistance = 10 .. which basically means that up to 10 upcoming moves would be scanned to check for interference and blend the appropriate tool path adjustments for the upcoming moves. However, it seems that the algorithm for doing these calculations are not multi-threaded much as it seems to take over the CPU and until the CCR calculations are done, ethercat packets are not being processed. This in turn triggers the big spike in Ecat[x].DcClockDiff and then eventually the $1000A appears.

I was able to lower my Coord[x].CCDistance to 4 and saw the spike diminish substantially back down to normal "background" deviations and now my Ethercat seems more stable.

BTW, This is all on the latest firmware version 2.7.1.0 although I have also seen it on 2.7.0.0 .. not sure about the earlier firmwares.

Could the firmware team check into the cutter-comp functions and see if they can be properly multi-threaded so that the Ethercat logic also gets a chance to run as needed during the calculations ? Also, I'm thinking there is some way that the CCR logic is also corrupting or filling up the same buffer that Acontis stack is trying to use which then triggers the EC_E_NOMEMORY error...

[steve.milici]

I will have the FW engineers take a look at this.

[Omron Forums Support]

As for ECAT[i].Error, it will report errors while EtherCAT is activated, and I believe that the 981 is always dropped for purpose of looking up the error in the documentation.

Edited November 10, 2023 by Gregs

[Sanjay_RSS]

No .. I often see other $98xxxxxx errors in the Ecat[i].Error register.
As far as I can remember, the $1000A is the only error where the $981 is trimmed out

[Omron Forums Support]

I was just talking about how the errors are listed in the error description document, but now that you mention it, I should have said that the 981 is always used (or added), not dropped as shown in the document.

[Alex Anikstein]

@Sanjay_RSS, can you share what firmware version you're using?

[Sanjay_RSS]

I'm on the latest as far as I know .. 2.7.1.0 and also using latest PowerPmacIDE .. version 4.6.2.15

[MattReid]

We are seeing the error as well in 2.7.1.0 and the latest PowerPMACIDE. We do not use cutter-comp-radius. We only have real-time PLCs enabled and gantry following/skew correction enabled.

Please let us know of any updates to solve this? Currently, we just slowed down the Servo/EtherCAT Clock rates to see if this reduces the occurrence.

 

[steve.milici]

Is there any particular PMAC activity that seems to be associated when you get this error?

[tecnico]

I have a similar issue with a slightly different error   $1000F  that should be read $9811000F

This happens and sends my 1S drives in safeop (err 83.03)

Can't understand what does thies mean tough...

System is a CK3E running @2kHz, fw was 2.7.0.0 now upgraded to 2.7.1.0

 

 

Edited February 26 by tecnico

[tecnico]

to avoid further confusion here are the actual error codes I got from Brad

/* 0x0001000a: Got no response on a sent Ethernet frame */ 

/* 0x0001000b: Got no or unexpected response on a sent ecat init command from slave */

/* 0x0001000c: Got no response on a sent ecat master init command */

/* 0x0001000e: Timeout when waiting for mailbox init command response */

/* 0x0001000f: Not all slave devices are in operational state when receiving cyclic frames */

In my case the error 1000F could have been a bit misleading because the root cause was not cabling but rather a IO slave not liking inductive kickback from a mistakenly unfiltered 24V power relay