Gather losing samples

[george.kontogiorgos]

Hello,

 

I'm doing some experiments with data gather system and I relized that the gather program missed some samples.

 

The pointer Sys.PhaseSamples presented a more samples number than line numbers on the gather text file. The difference is about 20 samples but it is not repetitive for every gather.

 

The gather setup was did on IDE Terminal as follows:

 

Gather.PhaseEnable=0
Gather.PhaseAddr[0]=Sys.PhaseCount.a
Gather.PhaseItems=1
Gather.PhasePeriod=1
Gather.PhaseEnable=1
Gather.PhaseEnable=0
Gather.PhaseMaxSamples=500000

 

After I issued the gather program with wait option on Linux:

 

./gather_csv -p -w > /var/ftp/gather/test1.txt

 

Triggered the rotative gather on IDE Terminal:

 

Gather.PhaseEnable=3

 

After some time, usually when Gather.PhaseSamples > Gather.PhaseMaxSamples (To check the rotative buffer), I stopped the data acquisition:

 

Gather.PhaseEnable=0
Gather.PhaseEnable=1
Gather.PhaseEnable=0

 

The last gather gave me Gather.PhaseSamples=32818200. The first line of data acquisition file test1.txt (After Waiting to phase gather) is 71693224 phase counts and the last line of the same file is 104511413 phase counts. The difference is 32818189, so threre are 32818190 samples because we have to take the first line in account. So this data acquisition lost 10 samples this time.

 

I don`t know if there is any delay on the gather system at the begining or at the end of the acquisition due to initialization or finalization of the program respectively and this lead to missing some samples.

 

The difference between one sample and another is 1 in every line (I tested it with a Python script), according to Gather.PhasePeriod=1. This led me the conclusion that these lost samples are not at the middle of the gather.

 

I would like to know why I`m losing samples and understand it to set up things correctly for future applications.

 

Thanks in advance.

[steve.milici]

Try using the “-u” option instead of the Linux pipe. That may be the source of the “delay”.

[george.kontogiorgos]

Try using the “-u” option instead of the Linux pipe.

 

Your tip led me to the following error:

 

10.20.56.18:/opt/ppmac/gather_csv# ./gather_csv -p -w -u /var/ftp/gather/test1.txt
Too many command line arguments.  If using a long file name
enclose it in " " ie. "long filename.txt"
Proper usage :
gather_csv [-p] [-u] [filename]
           -p optionally gather on phase
           -u upload last gathered data
           -w wait until for gather to started
           -r upload last rotary gathered data

 

Remember that I'm trying to perform a rotary gather and the option -u is used to upload gathered data on buffer. How could be the usage with -r (which is the option for rotary gatered data)?

 

I would like to keep the option -w, so I could start the gather and take the first samples on a triggered gather for example.

[george.kontogiorgos]

Some information about my controller

 

Type: POWER PMAC BRICK

CPU: PowerPC, AMP86xxx

Firmware: 2.5.4.0

[steve.milici]

Try enclosing the long filename in quotes.

 

Also, since you are gathering to a file you may not need “Gather.PhaseEnable=3”. Use “2” instead.

[wehg]

Hello,

I'm trying to accomplish the same functionality as George regarding streaming gathered data to a file.

I can successfully gather data to the gather buffer using Gather.Enable=2, and then write the gathered data to a file at gather completion using the gather_csv -u utility. This works fine, but is limited by the size of the gather buffer.

What I am trying to do is stream the gathered data to a file in rotary fashion, so that I am not limited by the gather buffer size. So far, I have not been successful with this rotary method.

Can you share an example of how to do this?

I'm using an ARM cpu with firmware 2.6.0.1.

[george.kontogiorgos]

3 minutes ago, wehg said:

What I am trying to do is stream the gathered data to a file in rotary fashion, so that I am not limited by the gather buffer size. So far, I have not been successful with this rotary method.

Hi wegh,

The way I found to circumvent the problem of rotary gather was issuing the following command to the PowerPMAC linux terminal (using pipe):

gather -p -w > "yourfile"

Once gather program armed, using Gather.PhaseEnable=2 to trigger the gather and setting large number on register Gather.PhaseMaxSamples to not reach the end of the file quickly.

At some point I stopped the gather with Gather.PhaseEnable=0 and issued Ctrl+C on PowerPMAC linux terminal. In another test I wait the gather to reach the Gather.PhaseMaxSamples and issued Ctrl+C on PowerPMAC linux terminal. On both situations the gather didn't lose any samples.

Nevertheless, I could not successfully do the rotary gather with the option Gather.PhaseEnable=3.

I also create a custom phase loop to be able to sample the Gather.PhaseSamples register (Setting Gather.PhaseAddr[0]=Gather.PhaseSamples.a is not possible) by assigning its value into a global variable so I can acquire it with PowerPMAC gather tool.

void user_phase(struct MotorData *Mptr)
{
	pshm->P[100]=pshm->Gather.PhaseSamples;
}

For now it solves my problem, but I think that future implementations would be in risk by this limitation of Gather.PhaseEnable=3.

[wehg]

Thank you for your reply. I probably should have clarified that I'm not having any issues with loosing samples, only that I am limited by a finite amount of data based on the gather buffer size. Currently I am reading the available gather buffer size using gather.enable=1, then setting Gather.MaxSamples = Gather.MaxLines. For what I would like to do, based on the Manual, I'm most interested in using gather.enable=3. I guess the question becomes: When using gather.enable=3, is the gather_csv utility applicable or usable? And if so, how do you use it in combination with gather.enable=3?

 

[george.kontogiorgos]

Hi wehg,

On 2/1/2022 at 9:05 AM, wehg said:

I'm not having any issues with loosing samples

I did some tests with lower sample frequencies and I didn't had issues with loosing samples too. I was performing gathers using phase interrupt source with 10 kHz of phase frequency and Gather.PhasePeriod=1 which led me to a 10 kHz of sampefrequency. When I used Gather.PhasePeriod>=3 I didn't loose any sample, neither at begining nor ath the end. 

Maybe your sample frequency is sufficient slow to don't show this kind of problem (I would appreciate if you could test your setup with higher sample frequencies). 

On 2/1/2022 at 9:05 AM, wehg said:

I guess the question becomes: When using gather.enable=3, is the gather_csv utility applicable or usable? And if so, how do you use it in combination with gather.enable=3?

Unfortunately I don't know the exact answer to your question and I would like to know too. 

I was searching on the forum some stuff related to gather and Background Programs and I found this. Maybe the best option for us is writting our own gather_csv which manipulates the buffers and files in a way that fits well for our applications.

Regards

George