michaelthompson Posted December 18, 2012 Posted December 18, 2012 Has anyone set up the Power PMAC to send e-mail? We are looking to package our debug dumps during certain events and e-mail them directly to our support staff. We have used sendmail in other environments to do this but have not yet performed the required setup under Linux. We envision a few different scenarios such as when a watchdog or similar serious/unusual event occurs. For the watchdog case, there would be a C application running and monitoring the watchdog status. When detected, we would use gpascii to dump P and M variable tables as well as our "black box" gather buffer. The dump would then be sent as an attachment(s) to an e-mail.
KEJR Posted January 23, 2013 Posted January 23, 2013 I've never tripped the watchdog, do the linux apps continue to run after it trips? If you are familiar with Sendmail then I would think to use that. From what I've read you can open /bin/mail or /usr/bin/mail with a call to popen() then use fwrite() to send the message and attachment data. See the following: http://www.unixguide.net/unix/programming/5.2.1.shtml I think that you will need to specify the correct MIME type for the binary data or convert all of your data to ASCII before sending it. I found some discussions on attachments here: http://lists.freebsd.org/pipermail/freebsd-questions/2005-April/083518.html If you want ultimate control you could roll the following C code into your app. I'm not sure if it is any good or not, but you can check it out: http://libsmtp.berlios.de/ The latter is probably overkill. Let me know if you get something working, I'm curious for my own work. KEJR
michaelthompson Posted January 23, 2013 Author Posted January 23, 2013 KEJR, Thanks for the info. Based on your other postings and the amount of experience you have with the Power PMAC, I'm a little surprised that you have yet to watchdog it! We have done so with bad pointers and locked PLCs on a few occasions (both accidentally and intentionally). Our experience is that Linux applications continue to run when you watchdog the PMAC. This is unrelated to the thread, but it is worth noting that we don't get any benefit from the built-in watchdog relay. The problem is that we don't get to specify the conditions under which the Power PMAC watchdog trips. To address safety issues, we need the watchdog to trip if a particular safety PLC fails to run its periodic scan (regardless of why). This includes the case where someone disables the PLC (or it disables itself deliberately or accidentally). The normal PMAC watchdog will not trip if we disable our safety PLC because our safety PLC is not required to "kick" it to keep it happy. As far as I know, Delta Tau's watchdog system does not allow us to operate it in that fashion (it would be nice if it did). Nor does it allow us to reset it after a watchdog event occurs. Once we watchdog the Power PMAC, we are forced to reboot. Consequently, we implement our own external watchdog relay with an ABB device. It is the only relay in our system (5-axis custom coordinate measuring machine). Our safety PLC "kicks" the ABB timer relay every 50-100 ms to keep it happy. If we fail to do so, it trips, triggering abort of all moves followed by a safe-torque-off (and disabled) state on all of our amplifiers. It works great, but we wouldn't need the ABB relay if the PMAC watchdog system supported a "user-kick" requirement and reset capability. Anyway, regarding the e-mail support, I think we need to install a mail package first. Our earlier investigation led us to believe that no mail package is installed out of the box (or at least not set up). It's low priority so we haven't done much other investigation since my last posting. I was sort of hoping that you might have already cracked it since, from past postings, you seem to be working with the Power PMAC in a way that most closely resembles our own use. The one difference is that we are using the PLC system plus external Linux applications and it sounded like you were doing all of your development as external applications. Thanks again for your reply. Hopefully, Delta Tau's Linux experts will give us some guidance and save us from investigating further on our own.
michaelthompson Posted January 23, 2013 Author Posted January 23, 2013 KEJR, Now that I think about it, if you are doing all of your development in external Linux applications, then perhaps that is why you have never watchdogged the Power PMAC. I think the watchdog occurs because the necessary "kick" doesn't occur within Delta Tau's PLC scheduling system (due to PLC lockup, exception, etc.) You can trap external applications all you like and never watchdog the PMAC. Mike
KEJR Posted January 24, 2013 Posted January 24, 2013 KEJR, Now that I think about it, if you are doing all of your development in external Linux applications, then perhaps that is why you have never watchdogged the Power PMAC. Hi Mike, I do prefer to work mainly outside of the PLC scripts and "C language" PLCs. I think there is more power in using the xenomai threading and keeping most of your machine logic in one process. I don't know if that helps with watchdogging, but I suppose it is likely. Its too bad the IDE won't let us do C++ because the pass by reference feature would eliminate most of the pointer usage (at least for me) and make for a safer experience. I think that your external watchdog relay is probably going to be the most reliable. I'm curious as to how you are shutting down your drives safely and gracefully from this device? Does it pull an enable line on the drives and some number of ms later it kills your mains contactors to the drives? I am working on the mail thing on the PPMAC now. It looks like both mail and sendmail programs are installed. I'm going to do some testing because I see that as something that I would like to use in the future. I'll let you know if I get anywhere. KEJR
shansen Posted January 24, 2013 Posted January 24, 2013 KEJR, I have used Eclipse for a little bit of C++ development (and much more C development) and I love it. One of the big benefits is that Eclipse lets me organize my projects however I see fit. It has its quirks like any software, but I highly recommend it if you are interested in C++ (or C) development for Power PMAC. Plus, it's free. Can't get better than that!
KEJR Posted January 24, 2013 Posted January 24, 2013 I have used Eclipse for a little bit of C++ development (and much more C development) and I love it. One of the big benefits is that Eclipse lets me organize my projects however I see fit. I've thought of using an external IDE or standalone make system but I've been struggling to keep everything in the IDE to get the shared global variable labels and have the motion prog editing be under one roof. Have you found any ways around these issues? Thanks, KEJR
KEJR Posted January 24, 2013 Posted January 24, 2013 I got the mail system to work and I'm sending mail with the command line. Your setup will vary depending on your network setup, but I ended up having to run the following debian config: dpkg-reconfigure exim4-config and choose "mail sent by smarthost" as the option for sending mail. Then later in the setup it will ask you for the mail server, which in my case auto detected somehow as our company mail server. This is all that was needed. Keep in mind that you will probably want to do a fresh reboot, unlock the filesystem, run the config, etc etc very similar to what you do when installing debian packages (I haven't done this, I'm just testing before committing my system). I can't speak to attachments, I haven't done anything with that. KEJR
michaelthompson Posted January 24, 2013 Author Posted January 24, 2013 KEJR, I think there is more power in using the xenomai threading and keeping most of your machine logic in one process. We considered that approach (and your other posts/discussions were helpful), but we decided to stay within the PLC model for now (which I believe is single process, multiple thread). That said, we have a couple of external (built and run outside of the context of the IDE) C applications for communications over sockets, logging, etc. What's nice is that Delta Tau gave us the flexibility to develop/operate in either way, or any combination. Its too bad the IDE won't let us do C++ because the pass by reference feature would eliminate most of the pointer usage (at least for me) and make for a safer experience. Agreed 100%, not to mention all of the other benefits C++ provides with classes, encapsulation, inheritance, polymorphism, etc. I understand Delta Tau needing to focus on the broader audience, but when they get C++ capability within the IDE, it will take the product to another level. I'm curious as to how you are shutting down your drives safely and gracefully from this device? The watchdog trips after 1000 ms (configurable) of not receiving a trigger pulse. When it trips, it puts all of the drives in Safe-Torque-Off mode (MOSFETs disabled). We are using Copley Xenus Plus drives and they support Safe-Torque-Off as separate from the enable line. However, the drives are not being used to commutate the motors (the PMAC is) so the drive can't handle the deceleration itself. In order to handle E-STOP, we let our E-STOP switch break the trigger signal to the watchdog timer. This means that the the watchdog timer will keep the drives out of the Safe-Torque-Off state for 1000 ms, long enough for the PMAC to execute a powered abort deceleration (which we can do in about 500-600 ms from our maximum velocity and inertia states). After the 1000 ms, the drive goes into Safe-Torque-Off state because of the watchdog relay. Redundantly, the PMAC will individually disable each drive (enable line) after any aborted motion completes (actual velocity less than some small value) and/or after some timeout period (i.e. 1000 ms) as a catch-all. What we gain with this approach is: 1. Elimination of all relays and braking resistors 2. Watchdog is only relay we presently require and it only switches 24V, low current Safe-Torque-Off line 3. Controlled and rapid E-STOP, much faster decelerations than resistive braking approaches 4. Safety logic implemented in software and managed with version control 5. Failsafe and redundant The only thing we lose over our older relay logic approach is static braking. Our older relay logic used resistive braking in the E-STOP state. The motor was disconnected from the drive and connected to a resistive load. This resistive load makes the air bearing stage harder to move in an E-STOP state (like moving through molasses). Without resistive braking in the E-STOP state, the air bearing floats freely. Depending on one's expectations, the elimination of static resistive braking can be considered a drawback or a benefit. In either case, it doesn't impact safety. I'm going to do some testing because I see that as something that I would like to use in the future. I'll let you know if I get anywhere. Much appreciated! Mike
michaelthompson Posted January 24, 2013 Author Posted January 24, 2013 KEJR I got the mail system to work and I'm sending mail with the command line. Great! Thanks. We will give it a try and post our experience. Mike
shansen Posted January 24, 2013 Posted January 24, 2013 KEJR, I rarely need motion programs, so I haven't really looked into doing them without Delta Tau's IDE. However, if motion programs are interpreted instead of compiled, it would seem straightforward to use Eclipse as an editor (just create a custom syntax highlighting for motion programs). When you want to download it to the Power PMAC, Eclipse has built in functionality to do this over SSH. On a side note, Eclipse has a "Remote Project" functionality, which lets you store, compile, and debug a project remotely (e.g. over SSH). What makes it so cool is that you can put Eclipse on a flash-drive, and then go to any computer (Windows, Linux, etc) and develop C and C++ for Power PMAC without installing anything. The downside is that this approach requires hardware (a physical Power PMAC). Obviously the typical Power PMAC user should use DT's IDE as recommended, but if you are really looking for C++ code this is one possible way to do it.
michaelthompson Posted January 28, 2013 Author Posted January 28, 2013 KEJR, I got the mail system to work and I'm sending mail with the command line. Your setup will vary depending on your network setup, but I ended up having to run the following debian config: dpkg-reconfigure exim4-config and choose "mail sent by smarthost" as the option for sending mail. Then later in the setup it will ask you for the mail server, which in my case auto detected somehow as our company mail server. This is all that was needed. After running the reconfiguration and attempting to send mail, we get permission denied messages: root@xxx.xxx.xxx.xxx:/opt/ppmac# echo "this is a test" | mail -s "subject line" recipient@mydomain.com 2013-01-27 20:40:13 1Tzdhl-0000xF-3Q Cannot open main log file "/var/log/exim4/mainlog": Permission denied: euid=101 egid=103 2013-01-27 20:40:13 1Tzdhl-0000xF-3Q Failed to create spool file /var/spool/exim4/input//1Tzdhl-0000xF-3Q-D: Permission denied 2013-01-27 20:40:13 1Tzdhl-0000xF-3Q Cannot open main log file "/var/log/exim4/mainlog": Permission denied: euid=101 egid=103 exim: could not open panic log - aborting: see message(s) above Can't send mail: sendmail process failed with error code 1 We can resolve this by changing ownership of the directories and modifying the Debian-exim user to be part of the root group as follows: root@xxx.xxx.xxx.xxx:/opt/ppmac# usermod -a -G root Debian-exim root@xxx.xxx.xxx.xxx:/opt/ppmac# chown -R Debian-exim:root /var/log/exim4 root@xxx.xxx.xxx.xxx:/opt/ppmac# chown -R Debian-exim:root /var/spool/exim4 However, making Debian-exim part of the root group might be solving the permission denied problem with a hammer. Would you know the preferred solution to this sort of problem? There are a lot of posts online about exim4 problems with permissions, but it isn't clear what the recommended solution is. Also, I'm wondering why you didn't run into this problem. Mike
KEJR Posted January 28, 2013 Posted January 28, 2013 After running the reconfiguration and attempting to send mail, we get permission denied messages: Ouch. I definitely did not get that problem. Is your distribution based on Debian Lenny? I only ask because the power PMAC I based my standard image off of is at least 2.5 years old by now and I don't know if they have upgraded to newer distro yet. Another difference might be because I have the video chip on board with the 4GB flash which I believe has more debian packages installed than the regular distro (i.e. desktop stuff). Now this shouldn't make any difference, but I thought I'd throw it out there as to a potential difference. I wish I had a better answer, but I'm not an expert with linux mail.
michaelthompson Posted January 28, 2013 Author Posted January 28, 2013 KEJR, It seems that all of our problems so far have been with exim and permissions. However, even with the permission changes, we still haven't been able to get it configured properly to send the mail through our relay. I verified that our relay is set up to allow the e-mail from the Power PMAC node by the following telnet session (which produced a test e-mail): telnet mailrelay.mydomain.com 25 Connected to mailrelay.mydomain.com. Escape character is '^]'. 220 mailrelay.mydomain.com ESMTP Sendmail 99.99.99/99.99.99; Mon, 28 Jan 2013 14:38:34 -0500 (EST) helo user 250 mailrelay.mydomain.com Hello myhost.mydomain.com [147.185.28.219], pleased to meet you MAIL FROM: 250 2.1.0 ... Sender ok RCPT TO: 250 2.1.5 ... Recipient ok DATA 354 Enter mail, end with "." on a line by itself Test message . 250 2.0.0 r0SJd8q03075 Message accepted for delivery This technique is described here (although I'm sure there are better examples out there): http://www.febooti.com/products/automation-workshop/tutorials/test-smtp-connection-send-test-email.html So, as a last resort, we could talk directly to the mail relay as I do above, but that would cause us to lose the benefits of exim (queuing, error handling, standardization, etc.) Is your distribution based on Debian Lenny? I believe that our systems are all Lenny. I haven't checked versions or updated the exim package. I'm pretty sure the only thing we are presently doing above and beyond the Delta Tau install is setup for our network access, samba, and an ntp daemon. Thanks for all of the help you've provided. I'm sure we'll eventually crack it, but it may take some work since nobody on this end is a Linux e-mail expert either.
KEJR Posted January 28, 2013 Posted January 28, 2013 One thing I recommend doing is to get a PPMAC that has a virgin image and back up your image using WinImage (see other posts). I've had a couple cases where I screwed something up and had to go back to stock to try and see where things went wrong. It would be good to know if this also failed the same on the stock image after running the config. I think that I did do an update on all my packages at one point, but its been a while (obviously don't update the kernel). KEJR
Recommended Posts