Hey guys,
I've hit a bit of a wall and I'm coming up on a deadline for getting SAM setup to replace our current monitoring solution (Icinga). While I've been able to setup all of the basic monitors and alerts we had in the last environment, I'm having some struggles getting the email alerting to do what I expect, and I'm wondering if I'm just building all of our alerts completely backwards or if I'm missing something obvious.
I've set the trigger conditions in such a way that we have thresholds and severity (as well as HTML emails) which send as an issue increases in severity like such:
Informational CPU:
Trigger: 80 <= X < 90
Reset: X < 80
Warning CPU:
Trigger: 90 <= X < 95
Reset: X < 80
Critical CPU:
Trigger: 95 <= X
Reset: X < 80
My reasoning is that I wanted the alert to clear (and another alert email to trigger) if the CPU moved into the next threshold, but I didn't want the reset (a Green "all clear" email) to fire unless the issue was resolved, not just moving from Warning to Critical. I also wanted to prevent the NOC view from seeing an Informational, Warning, and Critical alert for the same machine, as we've already had issues with the team ignoring a Warning as they also saw the Informational alert in the view.
As a note, I have tried removing the upper thresholds and only setting a reset on the lowest threshold. This worked, but the additional alerts in the NOC view confused our team. I've also tried adding the thresholds with only the lower reset trigger, but if we get too large of a jump between polls (Moving right from OK to Warning back to OK, we never get the reset/all clear email. Lastly, I've considered adding an "OK" alert, however this added clutter to both the Node/Object views (since they always had "triggered alerts") and added additional confusion to the NOC team when they went to the "All Alerts" view.
Am I coming at this backwards or is there a simple setting I'm missing? The NOC view works perfectly now, however the emails aren't behaving as I want (definitely a configuration issue on my side). Any advice or pointers would be greatly appreciated.
Thank you,
-JD