[Upload configuration error]

I just attempted to update my Power PMAC to the latest version XXX. After downloading and building my project, I attempted to upload my previous configuration (file attached), but I get an error. I had top save as a text file because the forum wouldnt let me attach. I've pasted the output below.

 

/var/ftp/usrflash/Project/Configuration/042916_simulator_config.cfg:7:18: error #75: Struct Write Error: Sys.CPUTimerIntr=$1

Downloading Configuration Files Failed!

 

Any idea what could be happening?

042916_simulator.txt

[New release 2.1.0.107 crashes while motor tuning]

I am having the same issue where the IDE (2.1.0.107) crashes when I attempt to do a open step response.

 

Primary option: I know that i can do an update to my firmware to fix this, however right now I'm stuck at 1.5.8 since when I upgrade any higher, my code doesn't run correctly at it did before. So I always had to revert back

 

Backup option: I believe that I was able to do the tuning fine with my old IDE (version 1.6.0.109) however, when I go to download filedepot to download it, the file is only about 9 mb in size. Where is it supposed to be ~700 mb in size. Is there a way I can get access to this old version of the IDE?

 

thanks,

windell

[IDE minimum requirements]

Hi,

 

What is the minimum system requirements for the IDE?

 

Is there a list of firmware versions that are compatible with each IDE version?

 

thanks,

windell

[Motion Program Stopping Itself]

Thanks for asking bgoskouie. Weʻll I did have the IDE running and I was using the watch window, but all I was looking at was two structure elements:

Coord[0].ProgActive and Coord[1].ProgActive

 

so there wasnʻt an abort anywhere in the watch window or a mistyped command.

 

Do you have IDE open while this happening?

IDE watch window (or another session connecting to PMAC) may be sending the abort command. For example if you have put "a" in watch window or a variable that starts with "a" and isn't recognized by the global defs.

[System Identification Using Sine Sweep]

Aloha,

 

Iʻm attempting to use the sine sweep to determine my plant transfer function. Iʻve done a search on the forum and see references to the PMAC Servo Analyzer, but that was 5 years ago. So Iʻm wondering if the IDE can produce the Bode magnitude and phase plots from a sine sweep test?

 

I understand that I can do individual sine tests, let the plant come to steady state and then manually and estimate the magnitude and phase, but Iʻm trying to see if there is a way to do this automatically.

 

Thanks,

windell

[Motion Program Stopping Itself]

Aloha,

 

Attached is a copy of the motion program that Iʻm running. I have a C program that starts it and pretty much never does an abort to stop it. However, Iʻve realize that after a few cycles of running it will stop itself. Iʻve verified that no following error or travel limit has occured. Iʻm wondering what could be going on here? How can I prevent it from stopping?

 

The P_PVT_HA_TRACKING flag is toggled to a 1 or 0 by the C program and is dependent on how far away the axis is from the target.

 

My thinking is that the linear move ends and doesnʻt see another move in front of it ready to execute and so the motion program ends.

 

My firmware is 1.5.8 (I know its old, but upgrading has caused some problems that I donʻt have time to deal with at the moment. My goal is to upgrade soon)

 

thanks,

windell

prog3.pmc

[Repeated Data Gather]

I certainly understand what you're saying about the JogTa limit because that is the way I interpret the setting when I read the SRM.

 

Background:

However, I have empirical evidence where 1 on 500 or so jogs will have a DesPos trajectory that violates its own acceleration limit by 2 to 3 times. Of course, I understand that it is not intended to do this, but I've tried all of your suggestions about things to check and monitor and the problem still remains.

 

What seems to be a trivial thing is seems to not be so simple. Here are three threads in the past that I've raised about the issue. Initially, I had a race condition where large steps were making it into my pvt commands, but now I am certain that there is no race condition and the motion program is aborted before the jogs are called.

 

http://forums.deltatau.com/showthread.php?tid=1972

http://forums.deltatau.com/showthread.php?tid=1987

http://forums.deltatau.com/showthread.php?tid=1884

 

What I've done:

What I've gathered through conversations with Steve is that when a Jog is called it uses measured velocity, acceleration, and jerk values as the initial point for the Jog and propagates that forward. So encoder noise could create an issue so that the jog will plot a trajectory from a false velocity that was calculated due to noise.

 

However when I speak with Curt, he says that encoder noise isn't an issue for the Jog trajectory since it is based on the last DesPos value right before the jog is called.

 

So I'm a bit confused about the conflicting information. It seems that I'm missing part of the puzzle.

 

What I'm doing now:

From my previous conversations with you, you were saying that jogs aren't meant to blend with pvt commands. I understand this so I'm currently following your old suggestion to use linear mode and not stopping the motion program. I'm rewriting some of our code now to use linear mode as opposed to jog calls. So that is the status.

 

Example:

To provide you with a clear example of what I mean, I've attached a screen shot of a fault that we received on the night of 4/7.

 

You will want to look at the red trace for the hour angle axis (the second plot). The red trace is the value of Motor[0].DesPos with time. The Y-axis is in degrees. X-axis is in seconds (Linux time). You'll also see that in the 5th plot, the current for the hour angle axis also saturates to 20A for this brief time. This current maxing out is the first sign that the trajectory is not right. When you go through the numbers, you get

 

(2*(4.149638-4.12615 deg)*3600 arc-sec/deg)/(.123201 sec) = 1372 arc-sec/s^2. At 1035 motor units/arc-sec, I convert to milliseconds/motor unit we get a value of 0.7 ms^2/count. The value we have set for JogTa for this move is -2.41 ms^s/count.

 

I'd be happy to speak with you on the phone to work directly with you on this issue.

 

thanks,

windell

 

 

 

If you use my suggested technique of logging to a USB device, you should not have storage limitation problems. You can plug in a USB hard drive, for example. The application note is called "Saving Files on Power PMAC or External Media" and is on the FileDepot on our forums (this site). It gives C examples. You just need to spawn a thread to run the code in the example (adjusted to log the data you want rather than the arbitrary P-Variables listed in the example) at the instant you want to start gathering.

 

I am confused about your first statement. It is impossible to command a jog beyond JogTa. You simply cannot do it. Power PMAC looks at your JogTa setting when generating the trajectory and it is an absolute limit in the command.

[Repeated Data Gather]

Hi Charles, Thank you for your help. One of the road blocks is debugging the issue with the jog call that generates a trajectory that violates its own JogTa setting. The trajectory remains parabolic (Ts=0 for us) but just simply is too quick and generates a following error.

 

1) We currently have a separate background program that runs all night and logs Motor[x].DesPos (as well as a lot of other data) at 50 ms intervals, but the issue is that it has been difficult to effectively work with deltatau to resolve the issue because we don't use the high resolution Gather function. Instead we use the data logging by our separate process and currently have an offline program that we use to parse it manually when there is a problem to troubleshoot. If I can get a list of items of what DT would need to more effectively debug my situation so that I can provide the necessary data at the sample rate that you would desired then that would really help me.

 

For faster sampling than 20 Hz maybe up to 100 Hz, I likely will need to create yet another separate process that samples for only a small window and has some smarts since continually recording data at a rate faster than 20 Hz might be storage repohibitive if doing so for the entire night.

 

2) Can you point me to where the application note that you are referring to is located?

 

thanks,

windell

 

 

 

Well technically you can use our built-in gathering features, but you have to trap your condition smartly and probably fork() in a C program and toggle the Gather.Enable smartly too. Or you could manually write structure/variable values into a C file that you create and ignore our gather feature entirely, but you would probably have a little bit more jitter on the sampling period than you would if you used built-in data gathering. This would be a simpler approach but less deterministic.

 

Give us more information on what kind of determinism you need and we can help you better.

[Repeated Data Gather]

Aloha,

 

I have a trajectory generation issue with the jog that over accelerates the telescope and only occurs once every thousand or so jog calls. It is not predictable when it will occur. However, I want to do data gathers before each jog is called for X amount of servo cycles and stop. Unfortunately, since I don't know when the fault will occur, I will need to come up with a way to either store consecutive data gathers for the entire night (~500 jogs calls) or gather for the maximum amount of servo cycles and like a fault doesn't occur during that time, I overwrite the gather buffer. If it does occur during the set amount of time, then I store that data gather somewhere.

 

I've read up on the Data Gather section in the Power PMAC Users Manual and looked at the training slides and it doesn't seem to go this advance.

 

I'm wondering if anyone has an example of how to do this?

 

thanks,

windell

[Motor[x].ActVel dropping to zero occasionally]

Looks like we're onto something. I will do that plot to see what is up.

 

We have a max delta filter running in the encoder table for each of our encoders. Since the ActVel is for the motor position, I'm assuming that seeing these drops means that they are making it past this filter?

 

thanks,

windell

 

Yes - this is from the inner loop position sensor. It is basically the difference of the inner-loop actual position values for this and the previous servo cycle. Compare it to Motor[x].DesVel to see if this is the actual command value. It could be encoder noise - plot this also to see if there is a relationship.

[Motor[x].ActVel dropping to zero occasionally]

Aloha,

 

While moving our telescope I was watching the Motor[x].ActVel register in the IDE scope and saw that it was tracking the telescope velocity then it starts to briefly drop to 0 and then goes back up to the constant velocity that the telescope is at.

 

What could be the causes of this? Is this register calculated from the Pos2 encoder entry? Could it be an encoder noise issue or corruption in bits from the absolute encoder?

 

On the large scale it looks like the currents are fine so I'm not sure if the servo loop is being affect by this.

 

thanks,

windell

[jog initial conditions]

1) We do have the different encoder scaling factors such that the scaling is the same as seen by the motor. For the absolute encoder we keep the value at 1, and for the incremental we pretty much set the scale factor to counts_absolute/counts_incremental. So with no slip, they should read the same value.

 

2) Haven't tried putting the different encoders for the inner and outer loops. That does look like an attractive option. I'm troubleshooting an existing operational system so changing the way we use our encoders in the inner and outer loops might be somewhat of an uphill battle for me. I would need to be sure that this issue is caused by the encoder switching in order to get permission to modify how were using the encoders in the feedback loop.

 

3) This sounds like a position estimation scheme by using the virtual motor. It does sound interesting and I can see its benefits, but we haven't done anything to this exotic level.

 

4) In between two jogs what would the velocity, acceleration, and jerk be? Would the last trajectory velocity, acceleration, and jerk values be passed to be the initial conditions for the next consecutive jog?

 

 

When you change encoders, are they scaled in the conversion table so that their effective scaling is the same when use by the motor? Otherwise the acceleration values change with the square of the scaling change, and jerk values with the cube.

 

Have you tried using the incremental encoder for the inner velocity loop and the absolute encoder for the outer acceleration loop? That is common in these telescope applications.

 

Have you tried using the load absolute encoder as feedback for a virtual motor whose servo output is cascaded to modify (offset) the position of a real motor that uses the motor incremental encoder as feedback? A couple of telescope guys have told me this gives the best results.

 

If the motor is at a closed-loop stop when the jog command is issued, starting velocity, acceleration, and jerk are all 0.0.

[jog initial conditions]

Thanks Curt for explaining so clearly.

 

1) Here is my scenario.

If a motor is currently running on a jog that is close to completing as it is getting close to the new position so its decellerating, then

a) the encoders are switched to a different one,

b) the servo gains are changed while acting on that encoder

c) the HomePos is changed for that motor

d) a jog command to a position is issued to a new position

 

Would any of the stuff before the jog cause the derivatives to be miscalculated for the second jog?

 

It would seem that from your previous response that the answer is no, but I just wanted to make sure that I was understanding right.

 

2) The reason I'm asking is that for our telescope application, sometimes (once in >1000) jogs, a jog is issued behind another jog and the acceleration of its trajectory is way faster than what is specified for the Jog.Ta and Jog.Ts values. It causes too much current to be drawn and we abort the motion. I want to note that the previous jog hasn't yet finished, so were pretty much updating the end point for the move by reissuing the jog.

 

What is happening between the two jog calls is that we switch encoders, change the servo gains, and modify the HomePos for that motor.

a) We switch encoders because we want the finer resolution when we get close to our target, but for big movements, we stay with a courser resolution.

b) We change gains when we're close because we want a tight loop for velocity, but when were moving to a new target we ease off on the gains since the following error isn't critical.

c) We modify the HomePos because the previous incremental encoder builds up error as it is mounted to the telescope via friction drive, so we do an absolute encoder (can't slip) read and adjust HomePos for the motor so that the motor position is now consistent with the absolute encoder read. Pretty much we zero out any incremental encoder error this way.

 

3) Say the motor is at a stop, but in closed loop and then a jog is called to move that motor to a new location. How does it calculate the Desired sub-structure coefficients? Do these come from the velocity of the encoder values or the velocity of the motor position for this first point? I'm really trying to figure out how the initial position, and velocity and acceleration for the trajectory is calculated when the motor was not commanded to go anywhere.

 

I apologize if I'm asking my questions in multiple ways. I'm really trying to get as low level as possible to remove as many assumptions as possible.

 

 

 

Your first understanding is basically correct; your second is not.

 

If the motor is executing a trajectory move (jog or home), the initial states for the new command come from the Motor[x].Desired sub-structure coefficients (and time within the move section). This all commanded trajectory information, and is unaffected by actual position issues like encoder noise.

 

You can change Motor[x].HomePos before commanding a jog move without messing up the derivatives. This is what PMAC does itself when the trigger is found in a homing search move. It writes to HomePos, then commands a move relative to the trigger position.

 

Pending trajectory move sections are stored in the Motor[x].New sub-structures, ready to be copied into the Desired sub-structure as the move execution progresses from one section to another.

[jog initial conditions]

Hi,

 

My understanding is that the current motor velocity, acceleration, and jerk is calculated for the initial condition for a jog call. I've also gathered that these are calculated from the encoders position updates somehow so if there is encoder noise, this can throw off these calculations and send the jog off on a faster than intended trajectory.

 

1) Can someone tell me from what registers are the velocity, accelerations and jerk values are being calculated from?

2) A more specific question, if I change the value of the Motor[x].HomePos just before a jog is called, could this mess up the velocity, acceleration, and jerk calculations?

3) The resultant of these calculations, are they stored in the Motor[x].Desired. Trajectory Substructure Elements?

 

Much thanks!

windell

[Switching Encoder Algorithm]

Thanks Charles. So for motor 0 if I want to do an absolute read of the encoder then I should pretty much set Motor[0].AbsPosSf to the same value as EncTable[0].ScaleFactor?

 

 

Note that Motor[x].AbsPosSf's default value is 0. You will have to set it nonzero (and to the appropriate value for your encoder) to use this feature.

[Switching Encoder Algorithm]

Aloha Steve,

 

When I try the homez command, it actually zeros out the counts for that encoder. Am I doing something wrong?

 

thanks,

windell

[Switching Encoder Algorithm]

Aloha,

 

How do you do a specific absolute read? I'm sorry if this is mentioned in the manual and I haven't seen it. If so can you point me to the page?

 

Thanks

Windell

[mounting spinning drive on Power PMAC]

Forgive me for not being specific enough. We have a Power PMAC UMAC CPU with a Power UMAC rack.

 

Do you know of any customers who have mounted a spinning drive to the Power UMAC? Other than writing to the flash card or using a USB harddrive, do you have any suggestions of how to store log files locally?

 

thanks, f

windell

 

We currently do not offer the SATA interface on any Power PMAC platform. On the Power UMAC platform we do offer the PCIe interface. The Linux installation however does not have any installed drivers for disk drives.

[mounting spinning drive on Power PMAC]

Hi Steve,

 

I know that this is an old thread, but I was just looking through the users manual (pg 33 of 1/6/2015 edition) and I saw that it looks like a spinning drive can be connected to the Power PMAC via SATA or PCIe. Do you have more information on how to go about doing this? Do I need to buy an additional card from DT?

 

thanks,

windell

 

This is not something we offer.

[Waiting between stopping motion program and a jog]

Hi,

 

I understand that this is not the deltatau recommended thing to do, but in my application, when I'm tracking a target I'm running a motion program with PVT mode and when i want to go to another target thats far away, I first stop the motion program and then I call a jog to accelerate the telescope to the next location.

 

1) From previous conversations, I believe that immediately after calling a jog, it will take on the current measured acceleration as measured by the encoders as opposed to taking on the command internal trajectory acceleration and propagate that forward according to Jog.Ta and Jog.Ts. Is that correct?

 

2) Based on your answer to #1, I would like to do a check before calling the jog to make sure that the currently measured acceleration is within certain limits. How would I go about doing this? I guess what I'm asking is that I want to poll the register that the jog uses for its initial acceleration to make sure that it is low enough before calling a jog.

 

Again, I understand that the recommendation is to use the linear mode and never stop the motion program, but I want to try this first before making a larger change to the motion program.

 

thanks,

windell

[Mutexs between scripts and C programs]

Mahalo Charles I'll see if this works. thank you for the reminder about the P-variables.

 

You can of course use P-Variables, but be aware you cannot do integer logic checks unless you cast to integer in C:

 

For example:

 

while((unsigned int)pshm->P[MyMutex] == 1){}

 

Please note that Sys.Cdata in Script maps the same memory as

 

volatile char* Cdata = (char*)pushm;

 

Cdata

 

maps in C and can be observed globally.

[Mutexs between scripts and C programs]

Thanks Charles.

 

Is it possible to do the same thing using global p-variables? I know its inefficient use of memory, but I like being able to view these variables in the scope.

 

thanks,

windell

 

Windell,

 

On the Script side, no such optimization is performed because it is all interpreted in the first place and evaluated at run time.

 

In C, I am not completely sure if the compiler will try to optimize, but you can force it not to by making your semaphore variable a volatile:

 

volatile char* Cdata = (char*)pushm;

[Mutexs between scripts and C programs]

Thank you Charles.

 

As I was referring to in the OP, I'm assuming that by you suggesting the "poor man's semaphore" method that you're certain that the compile optimizer will not detect that the semaphore isn't updated from within the while loop and use Loop-invariant Code Motion resulting in a potential infinite loop. Is that correct? I just wanted to make sure because this is one caveat of someone doing their own mutex.

 

This was my primary motivation to post here and ask before doing my own semaphore.

 

Look at the notes under Dekker's Algorithm here and the :

https://en.m.wikipedia.org/wiki/Dekker%27s_algorithm

 

thanks,

windell

[Mutexs between scripts and C programs]

Hi Charles,

 

That will be a bit difficult since on the C side, I'm writing to a set of P-variables and on the script side, I am reading from the same set. So I'm not sharing any critical areas between C and script, but I am read and writing to P-variables between the two.

 

1) When writing to P-variables is this an atomic operation such that on the script side I won't be able to read a half modified 64-bit double?

2) How about when reading a P-variable on the script side is this an atomic operation too?

3) I'm fine with C always taking priority. However, if I set Sys.Lock[1] == 1 when I'm doing a read of the P-variables on the script side, I want to make sure that the C program will wait until the lock is released rather than overriding the lock and doing a write anyway. Can you send me an example of what you're trying? I want to compare apples to apples here.

4) How would you propose to implement a mutex for our application without using the Sys.Lock element? Please refer to below.

 

Here is an example of what I'm doing so that were on the same page. Please note that this is greatly simplified from what were doing on our C side, I haven't checked it lately, but I'm guessing we have at least 10k lines of code. The position commands on the C side come from a host computer that is communicating with the Power PMAC.

 

C side:
Process 0:

main(){
...
while(1){
pshm->P[400] = a
pshm->P[401] = b
pshm->P[402] = c
pshm->P[403]= d

usleep(1000);
}
...
}

Process 1:
Motion program side:

while(1){
var1 = P400;
var2 = P401;
var3 = P402;
var4 = P403;

pvt(10);
X(var1):(DEC_MOTION_VEL)
X(var2):(DEC_MOTION_VEL)
X(var3):(DEC_MOTION_VEL)
X(var4):(DEC_MOTION_VEL)
X(var5):(DEC_MOTION_VEL)
}

 

 

Sys.Lock is not really designed to be used between a C program and Script. I suggest you put all of your critical areas that need to be shared either in Script or in C, not both.

 

Basically C always takes the Lock priority, so it will ignore what your Script program is doing. I just tested it.

[Errors when using _PPScriptMode_]

Thanks Steve. I really appreciate you looking so deeply into this. I think this is a worthy reason to upgrade our firmware. In case the new firmware doesn't work, is there a repository of old firmware versions so that I can revert back to 1.5.8 if I need to?

 

thanks,

windell

 

 

Actually its not until a later 2.x.x.x version.