Geo Macro power supply and error logging

[piefum]

Hi All

I am having some strange alarm on a macro chain composed of 3 GMH152.

Sometimes one or more Geo Macro trip the UnderVoltage Fault or Ring Fault after I send the ENABLE command to the motor.

 

This is really strange because the boot sequence is performed correctly without rising any error.

 

Is it possible that the power supply of the room where the system is installed is that bad that it cannot supply current to the driver, blocking one or more? The current requested on the main line is quite small (less than 3 A) when the motors are energized. Can the measure of the DC bus so disturbed that it reads UnderVoltage even if the DC bus is correct?

 

And, on top of this: does the Geo Macro have a sort of logging of the last NN errors occurred?

 

thanks

gigi

[Richard Naddaf]

Do you clear MACRO faults on power up?

What AC bus voltage are you bringing into the drives? Are they sharing he same line?

Can you recover from the fault?

 

There is no logging history. A noisy read of the line is possible. How is your grounding/filtering done, if any?

[piefum]

Do you clear MACRO faults on power up?

What AC bus voltage are you bringing into the drives? Are they sharing he same line?

Can you recover from the fault?

 

I'm using 380V AC 3phase. The main line coming from "the wall" is passed through a filter (schaffner FN258 EMC/RFI 3phase filter) and the out of the filter is divided in the 3 GeoMacros. To recover from the fault I reboot the system (the customer does not have the option to send the ClearFault command to the driver, I can try only when I will be in front of the machine tomorrow).

 

There is no logging history. A noisy read of the line is possible. How is your grounding/filtering done, if any?

 

We have 3 racks, each rack contain a GeoMacro; each rack is sealed and it presents a connector to receive 380V ac 3phase and 220V. Inside the rack, all the grounds are brought to a terminal block and then return to the main ground coming from the connector. In the cabinet, we have a single ground point (for all the devices placed inside the cabinet).

 

Inside each Geo Macro rack I have a 220V switching to get the 24V dc (passing through a common chocke) for the logic of the GeoMacro.

 

thanks

gigi

[steve.milici]

It is possible that when you enable the drives there is a power load that causes a "brown-out" of other devices sharing this power source (source of bus power for the drive)? Put a scope on this power source and monitor it during the enable.

[piefum]

It is possible that when you enable the drives there is a power load that causes a "brown-out" of other devices sharing this power source (source of bus power for the drive)? Put a scope on this power source and monitor it during the enable.

 

In this cabinet, the 380V is used only for the GeoMacros, anyway I was thinking about the same issues because in the same building there are a lot of other 380V high-current devices.

I already already asked the people there to perform this measure, I am waiting for the results.

 

BTW, I'm concerned about the bus protection thresholds of the GeoMacro. The manual states that the overvoltage fault trips at 828V DC, while the undervoltage trips at 20V DC.

Are you sure about the UnderVoltage value? It seems pretty low! I tested a spare GeoMacro that I have here in the lab: the undervoltage fault trips when I cut off one of the 3 phases of the 380V ac. How it would be possible? Even with 2 phases the DC voltage should be much higher than the 20V DC. I tested this because I'm told that one phase-loss is a pretty common problem in this kind of power plant. What to you think about it?

 

thanks

gigi

[Richard Naddaf]

No, the under-voltage is at 97VAC now. This has been implemented for years, the documentation has not been updated.

 

A phase loss will definitely trip the line monitor/under voltage fault.

[bradp]

The undervoltage protection is a little more complete. Around 20VDC is the ultimate lower threashold. You can never run the drive with a BUS lower than that. Since the drives allow you to supply a DC voltage it is possible that people do use these at lower voltage. If the BUS ever goes above 97VDC then if it falls back below 97VDC you get undervoltage. So you see there are two levels, one for those running above 97VDC and one for those using low voltage.

[piefum]

No, the under-voltage is at 97VAC now. This has been implemented for years, the documentation has not been updated.

 

A phase loss will definitely trip the line monitor/under voltage fault.

 

something is strange: a phase loss on a 380V AC would lead to a system that is always above the 97V AC. Am I doing the wrong math?

 

Anyway, I have some more doubt on the system because I got all the other errors (Ring Fault, OverVoltage, DC/DC error). Is it possible that EMC noise can trigger all these faults?

 

On the Ring Fault: how does the macro work? Does it have its own error correction algorithm or it trips a fault after a single faulty reading? If it is the second option, this is leading me to the EMC problems on the cabinet.

 

ciao

gigi

[piefum]

The undervoltage protection is a little more complete. Around 20VDC is the ultimate lower threashold. You can never run the drive with a BUS lower than that. Since the drives allow you to supply a DC voltage it is possible that people do use these at lower voltage.

 

so, more tests are ongoing here. In the next days I'm planning to run the GeoMacro only with DC.

On top of the manual is written that to run the GMH152 in DC "optional DC

power input (24 – 350 or 24 – 700 VDC)".

It is really needed an option (that I don't see in any part of the document) or it is only necessary to connect DC to L2 and L3 (as stated in the section "Wiring AC Input, J1" at page 35?)

 

thanks

gigi

[bradp]

In this case the word "optional" is like "or". You can supply one or the other.

[Richard Naddaf]

You can bring in DC instead of AC. Do not plug it in hot, wire it in (to L2 and L3) then turn on your DC power supply.

 

Yes, electrical noise could cause some of these faults.

 

The drive sets itself on start-up for single phase of three phase operation. Once in three phase, and a phase is lost the line monitor trips.

 

The MACRO ring faults are governed by the clock settings, MI8, MI9, and MI10. Attached spreadsheet (courtesy of Sina) for your reference.

 

I would try to isolate as much as possible, maybe run one drive at a time, use an external 24V etc...

MACRO Ring Error Settings.xls

[bradp]

Gigi, please correct me if I am wrong, the system we are speaking about is RJ45 Macro, correct? This is important for Richard to know in regards to the Macro Ring Error discussion.

[piefum]

Gigi, please correct me if I am wrong, the system we are speaking about is RJ45 Macro, correct? This is important for Richard to know in regards to the Macro Ring Error discussion.

 

yes, correct, the ring is over RJ45 and the drivers + CPU are in an electronic cabinet that contains many other instruments.

 

I'm going to try the DC power supply in the next days on the spare driver.

I have here in the lab only a limited-power DC PSUs, so yesterday I tried only to power up the spare GeoMacro and it seemed fine, without any error. Even if the power that I can deliver now is very limited, I tried only few actuator motions and it seemed to work fine. I'll let you know how this will evolve.

 

 

Thanks to all

gigi

[piefum]

On a side note, I'd like to share this strange behavior of the same actuator controlled by the same GeoMacro.

 

Sometimes (once per hour), it seems that the power output drops and immediately restart. In attach a picture of a typical stress test that is running since few days here in the lab: 1 step of 1 um, wait for 30s, step back to 0, wait for 30s, repeat.

In the middle of the plot you can see that the current suddenly "falls" (around 728s) and then the loop recovers this unexpected motion.

I don't think this is related to the servoloop: this actuator is now running with an Open Servo algorithm that has an output filter at 20 Hz (when the actuator is in positon), and this sudden "jump" is clearly out of my bandwidth.

 

Any idea?

 

thanks

gigi

[bradp]

I think you need to explain how this data was sampled and compiled to help us understand. What are we seeing?

 

The reason is that when you look at the position loop bandwidth and the current loop bandwidth both seem much slower than the jump in position and drop in current. To me this means the control loops could not command either of these. But not knowing where this data comes from makes guessing at other causes hard.

 

Also, does your data have enough resolution to tell which came first, the position jump or current drop?

[piefum]

I think you need to explain how this data was sampled and compiled to help us understand. What are we seeing?

 

yes, sorry for that, I forgot to paste the part where I explain the curves.

 

The first plot represent the commanded position (line in black) and the position read from the encoder (red). The red-dotted line is the difference between these two values. The commanded position is recorded by gathering M161 and the actual position is M162 (this is not accounting for the backlash compensation that you can see at t = 710s and 740s.) [plot is in microns]

 

The second plot is the actual current flowing in the motor, gathering M175. [plot is in Amperes]

 

The reason is that when you look at the position loop bandwidth and the current loop bandwidth both seem much slower than the jump in position and drop in current. To me this means the control loops could not command either of these. But not knowing where this data comes from makes guessing at other causes hard.

 

I measured both the position loop and the PI loop of this system: the position loop has a bandwidth of ~70 Hz, that is reduced at ~20Hz when the actuator is InPosition (bandwidth reduced in the Open Servo algorithm).

The PI loop has a much higher bandwidth, around 300 Hz.

 

Also, does your data have enough resolution to tell which came first, the position jump or current drop?

No, I cannot tell which came first; unfortunately, with this setup (ethernet connection to the UMAC) I cannot go faster than these 32 Hz.

 

I could try to use the rotary gathering buffer, but I need some flag that I can poll to stop the rotary buffer.

 

 

thanks

gigi

[Richard Naddaf]

How often does this blip occur? Is it deterministic? And does it happen if you are just holding position?

 

Is this a good know actuator setup configuration?

 

I don't understand does the current plot not show the blips at the rise and fall of the step.

 

I hope you get a lead somewhere, this is going to be tough to troubleshoot through writing. I would be glad to help over the phone or online webex at some point.

[piefum]

How often does this blip occur? Is it deterministic? And does it happen if you are just holding position?

 

it happens more or less once in 1 hour. It happens more often when I am holding the position, in very rare occasion (1 every few hours) when the actuator is moving.

 

Is this a good know actuator setup configuration?

 

yes, the configuration has been running for months, but since we had few problems (noises etc), it has been decided to run long fatigue test, such as this one.

 

I don't understand does the current plot not show the blips at the rise and fall of the step.

 

I don't get this point: the current behavior seems ok to me, changing sign when holding position of different signs. When it "jumps" (or turns-off), the position is changing as well (probably due to the motor cogging torque).

 

 

I hope you get a lead somewhere, this is going to be tough to troubleshoot through writing. I would be glad to help over the phone or online webex at some point.

 

thanks, we can discuss of this at some point later, when I will have more data to talk about.

 

thanks

gigi

[bradp]

How have you done the backlash compensation, please explain the variables used.

 

Can you send your motor i-variables, just for this motor otherwise it will be too big (ixx00 to ixx99) plus the i0 to i99.

[piefum]

How have you done the backlash compensation, please explain the variables used.

 

I used the i185, i186 and i187 variables. The compensation is equivalent to 0.2 microns (i185=400 == 0.2 micron)

 

Can you send your motor i-variables, just for this motor otherwise it will be too big (ixx00 to ixx99) plus the i0 to i99.

 

here all the variables, just uploaded while it is under test right now.

 

 

; Upload of PMAC I Variables 0 to 199.
; Device # 0 [uMAC TURBO] V1.945   07/02/2008: Ethernet Port
; Executed at Thu Jan 30 09:50:50 2014
;
I0=0;Serial Card Number
I1=0;Serial Port Mode
I2=1;Control Panel Port Activation
;I3=2;I/O Handshake Control
;I4=0;Communications Integrity Mode
I5=2;PLC Program Control
;I6=1;Error Reporting Mode
I7=0;Phase Cycle Extension Period
I8=2;Real Time Interrupt Period
;I9=2;Full/Abbrev. Listing Control
I10=3727928;Servo Interrupt Time
I11=0;Programmed Move Calculation Time
I12=0;Lookahead Time Spline Enable
I13=0;Foreground In-Position Check Enable
I14=0;Temporary Buffer Save Enable
I15=0;Deg/Rad Control for User Trig. Functions
I16=5;Rotary Buffer Request On Point
I17=5;Rotary Buffer Request Off Point
I18=10;Fixed Buffer Full Warning Point
I19=6807;Clock Source I-Var. Number (Turbo PMAC2 Only)
;I20=$78400;MACRO IC 0 Base Address (Turbo PMAC2 Only)
;I21=$0;MACRO IC 1 Base Address (Turbo PMAC2 Only)
;I22=$0;MACRO IC 2 Base Address (Turbo PMAC2 Only)
;I23=$0;MACRO IC 3 Base Address (Turbo PMAC2 Only)
;I24=$60000;Main DPRAM Base Address
I25=0;(Reserved)
I26=0;UMAC Electrical MACRO Enable
I27=0;Alternate TWS input format
I28=0;(Reserved)
I29=$0;(Reserved)
I30=1;Compensation Table Wrap Enable
I31=0;(Reserved)
I32=0;(Reserved)
I33=0;(Reserved)
I34=0;(Reserved)
I35=0;(Reserved)
I36=0;(Reserved)
I37=$0;Additional Wait States
I38=0;In-Line CALL Enable
I39=0;UBUS Accessory ID Variable Display Control
I40=0;Watchdog Timer Reset Value
;I41=0;I-Variable Lockout Control
I42=0;Spline/PVT Time Control Mode
I43=0;Auxiliary Serial Port Parser Disable
I44=0;(Reserved)
I45=0;Foreground Bin. Rot. Buf. Transfer Enable
I46=0;P And Q Variable storage location
I47=0;DPRAM Motor Data Reporting Period
I48=0;DPRAM Motor Data Reporting Enable
I49=0;DPRAM Background Data Reporting Enable
I50=20;DPRAM Background Data Reporting Period
I51=0;Compensation Table Enable
I52=23;CPU Frequency Control 
I53=12;Auxiliary Serial Port Baud Rate Control
I54=12;Main Serial Port Baud Rate Control
I55=0;DPRAM Background Variable Buffers Enable
I56=0;DPRAM ASCII Communications Interrup Enable
I57=0;DPRAM Motor Data Background Reporting Enable
;I58=1;DPRAM ASCII Communications Enable
I59=0;Motor/C.S. Group Select
I60=15;Filtered Velocity Sample Time
I61=8;Filtered Velocity Shift
I62=0;Internal Message Carriage Return Control
I63=1;Control X Echo Enable
I64=1;Unsolicited Responses Tag Enable
I65=0;User Configuration Variable
I66=0;(Reserved)
I67=$0;Modbus TCP buffer start address (UMACs only)
I68=15;Coordinate System Activation Control
I69=$0;Modbus TCP software control panel start address(UMACs only)
I70=$3333;MACRO IC 0 Node Auxiliary Register Enable
I71=$3333;MACRO IC 0 Node Protocol Type Control
I72=$0;MACRO IC 1 Node Auxiliary Register Enable
I73=$0;MACRO IC 1 Node Protocol Type Control
I74=$0;MACRO IC 2 Node Auxiliary Register Enable
I75=$0;MACRO IC 2 Node Protocol Type Control
I76=$0;MACRO IC 3 Node Auxiliary Register Enable
I77=$0;MACRO IC 3 Node Protocol Type Control
I78=32;MACRO Type 1 Master/Slave Communications Timeout
I79=32;MACRO Type 1 Master/Master Communications Timeout
I80=0;MACRO Ring Check Period
I81=2;MACRO Maximum Ring Error Count
I82=0;MACRO Minimum Sync Packet Count
I83=0;MACRO Parallel Ring Enable Mask
I84=0;MACRO IC Number for Master Communications
I85=0;MACRO Ring Order Number
I86=0;(Reserved)
I87=0;(Reserved)
I88=0;(Reserved)
I89=0;Cutter Comp Outside Corner Break
I90=$39;VME Address Modifier
I91=$4;VME Address Modifier Don’t Care Bits
I92=$FF;VME Base Address Bits A31-A24
I93=$7F;VME Mailbox Address Bits A23-A16,ISA DPR Address A23-A16
I94=$A0;VME Mailbox Addr Bits A15-A08,ISA DPR Addr A15-A14 And Control
I95=$7;VME Interrupt Level
I96=$A1;VME Interrupt Vector
I97=$0;VME DPRAM Base Address Bits A23-A20
I98=$60;VME DPRAM Enable
I99=$30;VME Address Width Control
I100=1;Motor 1 Activate
I101=3;Motor 1 Commutation Enable
I102=$78420;Motor 1 Command Output Address
I103=$350F;Motor 1 Position Address
I104=$350F;Motor 1 'Velocity' Address
I105=$35C0;Motor 1 Master Position Address
I106=0;Motor 1 Master Follow Enable
I107=96;Motor 1 Master Scale Factor
I108=96;Motor 1 Position Scale Factor
I109=96;Motor 1 Velocity Scale Factor
I110=$100;Motor 1 Power-on Servo Position Address
I111=1048000;Motor 1 Fatal Following Error Limit
I112=52400;Motor 1 Warning Following Error Limit
I113=2620000;Motor 1 + Software Position Limit
I114=-2620000;Motor 1 - Software Position Limit
I115=1;Motor 1 Abort/Lim Decel Rate
I116=98;Motor 1 Maximum Velocity
I117=0.262144;Motor 1 Maximum Acceleration
I118=0;(Reserved)
I119=1;Motor 1 Maximum Jog Acceleration
I120=0;Motor 1 Jog/Home Acceleration Time
I121=50;Motor 1 Jog/Home S-Curve Time
I122=30;Motor 1 Jog Speed
I123=-35;Motor 1 Homing Speed And Direction
I124=$48001;Motor 1 Flag Mode Control
I125=$3440;Motor 1 Flag Address
I126=0;Motor 1 Home Offset
I127=0;Motor 1 Position Rollover Range
I128=300;Motor 1 In-Position Band
I129=160;Motor 1 Output/1st Phase Offset
I130=25000;Motor 1 PID Proportional Gain
I131=840;Motor 1 PID Derivative Gain
I132=840;Motor 1 PID Velocity Feed Forward Gain
I133=18000;Motor 1 PID Integral Gain
I134=0;Motor 1 PID Integration Mode
I135=0;Motor 1 PID Acceleration Feed Forward Gain
I136=0;Motor 1 PID Notch Filter Coefficient N1
I137=0;Motor 1 PID Notch Filter Coefficient N2
I138=0;Motor 1 PID Notch Filter Coefficient D1
I139=0;Motor 1 PID Notch Filter Coefficient D2
I140=0;Motor 1 Trajectory Filter Constant
I141=0;Motor 1 Desired Position Limit Band
I142=$0;Motor 1 Amplifier Flag Address
I143=$0;Motor 1 Overtravel-Limit Flag Address
I144=$0;(Reserved)
I145=0;(Reserved)
I146=0;(Reserved)
I147=0;(Reserved)
I148=0;(Reserved)
I149=0;(Reserved)
I150=0;(Reserved)
I151=0;(Reserved)
I152=0;(Reserved)
I153=0;(Reserved)
I154=0;(Reserved)
I155=$0;Motor 1 Commutation Table Address Offset
I156=0;Motor 1 Commutation Delay Compensation
I157=2907;Motor 1 Continuous Current Limit
I158=53;Motor 1 Integrated Current Limit
I159=1;Motor 1 User Written Servo Enable
I160=0;Motor 1 Servo Cycle Period Extension
I161=0.04999995232;Motor 1 Current Loop Integral Gain
I162=0.5499999523;Motor 1 Current Loop Prop. Gain (Forward Path)
I163=4000000;Motor 1 Integration Limit
I164=0;Motor 1 'Deadband Gain'
I165=2000;Motor 1 Deadband Size
I166=20000;Motor 1 PWM Scale Factor (PMAC2 Only)
I167=4194304;Motor 1 Position Error Limit
I168=0;Motor 1 Friction Feedforward
I169=5815;Motor 1 Output Command Limit/Scale
I170=6;Motor 1 Number of Commutation Cycles (N)
I171=8388607;Motor 1 Counts Per N Commutation Cycles
I172=1365;Motor 1 Commutation Phase Angle
I173=0;Motor 1 Phase Finding Output Value
I174=0;Motor 1 Phase Finding Time
I175=6329465;Motor 1 Phase Position Offset
I176=0.5;Motor 1 Current Loop Proportional Gain (Back Path)
I177=0;Motor 1 Induction Motor Magnetization Current
I178=0;Motor 1 Induction Motor Slip Gain
I179=128;Motor 1 2nd Phase Output (DAC) Bias
I180=0;Motor 1 Power-Up Mode
I181=$100;Motor 1 Power-On Phase Position Address
I182=$78422;Motor 1 Current loop Feedback Address
I183=$78420;Motor 1 Commutation Position Address
I184=$FFF000;Motor 1 Current-Loop Feedback Mask Word
I185=400;Motor 1 Backlash Take-up Rate
I186=400;Motor 1 Backlash Size
I187=1500;Motor 1 Backlash Hysteresis
I188=0;Motor 1 In-Position Number of Cycles
I189=0;(Reserved)
I190=0;Motor 1 Rapid Speed Select
I191=$740000;Motor 1 Power-on Phase Position Format
I192=10;Motor 1 Jog-To-Position Calculation Time
I193=0;(Reserved)
I194=0;(Reserved)
I195=$740000;Motor 1 Power-On Servo Position Format
I196=0;Motor 1 command Output Mode Control
I197=0;Motor 1 Position Capture/Trigger Mode Control
I198=0;Motor 1 Third-Resolver Gear Ratio
I199=0;Motor 1 Second-Resolver Gear Ratio
; ** End of Upload **

[piefum]

 

I'd like to add some other infos about this strange "jump": I am now testing the actuator in a fixed position in close loop, without any motion, and I get the same strange jump.

[piefum]

Gigi, please correct me if I am wrong, the system we are speaking about is RJ45 Macro, correct? This is important for Richard to know in regards to the Macro Ring Error discussion.

 

FYI: we think we solved the startup false error problem with an extra power line filtering.

The main power supply now pass through a 3phase transformer that reduces the V ac input of the GeoMacro by a factor of ~4 (400 to ~90V ac), that is the voltage necessary to the application. At the beginning of the project we decided to keep all the voltage available for contingency, but now we decided to go for the extreme reliability of the system.

 

The motors are now equipped with pure sinusoidal filters (before we had only PWM slew-rate filter), and the system seems more quite (in terms of both acustic and EMC).

 

bests

gigi

[piefum]

No, I cannot tell which came first; unfortunately, with this setup (ethernet connection to the UMAC) I cannot go faster than these 32 Hz.

 

I finally managed to gather at high frequency the strange blip; in attach you can see a plot showing the following error, the actual current and the commanded current.

It seems that the blip is starting from the actual current (readout of the ADC of the geomacro).

 

Does this means anything to you?

ciao

gg

[bradp]

Stating what I see to make sure we agree. The current starts to rise then the FE starts to decrease. The current then rises quite fast, peaks and then suddenly starts to decrease as FE continues to decrease. The current then gets back to the "normal" level and oscillates as FE oscillates. The FE settles at a lower value than it was before and the current is also a little lower. The change in current from start to end of this event seems to be of a reasonable magnitude comparing to how much it was changing during the oscillation.

 

Question then is how can the current rise like this when FE is not growing? And in previous plots you have the current dropping to almost 0 then coming back.

 

Current loop is a closed loop system. It relies on the ADC feedback. One possibility is something changes on one or both of the ADC feedbacks.

Current loop is coupled to phase position. The reading of the phase encoder is usually not the same as the position encoder as one goes through the ECT and might even be interpolated. So something could have changed on the phase encoder value.

[piefum]

Current loop is a closed loop system. It relies on the ADC feedback. One possibility is something changes on one or both of the ADC feedbacks.

Current loop is coupled to phase position. The reading of the phase encoder is usually not the same as the position encoder as one goes through the ECT and might even be interpolated. So something could have changed on the phase encoder value.

 

ok, good to know. In fact, the ECT entry of this channel has an Exponential Filter (Max change + LPF filter).

How can I check that the phase encoder value has some problem?