BlogFileMaker

Monitoring & Auto-Healing FileMaker Server with Zabbix

By June 10, 2019 September 25th, 2019 3 Comments

Our Zabbix FileMaker Server Monitoring Session at DevCon 2019

At FileMaker DevCon 2019, Mislav Kos and I presented a session on Server Monitoring with Zabbix. The FileMaker community has good information around this topic but most of it relates to how to install Zabbix.

In this short preview, we fast-forwarded a bit and showcased what the end result looks like. Specifically, we demonstrated the power of Zabbix. It can keep an eye on your FileMaker Server, show you its vitals, and send notifications when things go wrong. However, more importantly, it can also proactively step in and fix important issues.

Restarting the Data API Process with Zabbix

For this article, we will show how Zabbix can restart the Data API process if it stops. You can apply this same logic to the other processes that make up all of FileMaker Server: the server-side scripting engine, the web publishing engine, the ODBC/JDBC process, etc…

When any of these components crash or stop, Zabbix can step in and take corrective action.

How Zabbix Works with FileMaker Server

Zabbix works as follows:

  1. It collects data on items that you have defined. In the Zabbix template released with our DevCon presentation (get them here), we include over 150 relevant items related to FileMaker Server’s health.
  2. For those items that you deem important, you define a trigger that evaluates the data for a particular item. Examples include: process running or not running, cache hit % below 95%, number of Remote Calls in Progress higher than 10, etc…
  3. When Zabbix finds that the data for anfdevcon item matches one of your triggers, it raises a problem event. Zabbix automatically tracks these problems and their durations, which you can see in the Zabbix dashboard.
  4. Additionally, you can have Zabbix take action. Most typically choose to send out a notification by email or SMS. However, Zabbix can also execute a command on the host machine where the problem happens.
FileMaker Zabbix Workflow - Item: Collect data, then Trigger: Evaluate collected data, then Problem: Track problems, then Action: Send email or text or perform remote action

Figure 1 – Zabbix workflow

FileMaker Server Monitoring with Zabbix

One of the servers we track is a FileMaker Server 17 running on macOS. Using the macOS Activity Monitor we can see the various FileMaker Server processes including “fmwipd,” which is the Data API process.

Screenshot of the macOS Activity Monitor in Zabbix FileMaker

Figure 2 – macOS Activity Monitor

There are a few different ways that we can track if it stops working; for instance, we could rely on the FileMaker Server event log where an event with ID 701 will be raised when FileMaker Server detects that its Data API has terminated abnormally. Zabbix can certainly (and easily) keep an eye on the event log.

But we will work with the operating system and use the OS tools to track the process. On Windows, we would do this by using the built-in Windows Perfmon and ask for the Thread Count on the process. If it is 0 or there is no data for that counter, then we would know that the process is not running.

But this is macOS, so we will just use the built-in “ps” command to get a list of the running processes and their state.

In our Zabbix template we have an item for just that purpose (note that it filters the output of the ‘ps’ command to just processes owned by ‘fmserver’):

Screenshot of Zabbix template showing 'ps' command

Figure 3 – Zabbix Template

This item will list all of the fms processes every 30 seconds. We have a dependent item that uses this output to look specifically for the ‘fmwipd’ process. For this we make use of Zabbix’s JavaScript pre-processing functionality:

Screenshot of the FMS Processes FileMaker Zabbix

Figure 4 – FMS Processes

This item will return a simple ‘1’ if the process is running and ‘0’ if it is not.

As you can tell from the screenshot, we have a trigger defined for the item to raise a ‘Disaster’ problem when the last value returned for the item is ‘0’. We check this every 30 seconds.

Screenshot of trigger defined

Figure 5 – Trigger defined

At this point, the problem visibility is limited to the Zabbix dashboard. We need an action to be notified automatically and for Zabbix to take action:

Screenshot of Actions in Zabbix for FileMaker

Figure 6 – Zabbix Actions

We have configured the action to do two things: email us immediately to make us aware of the problem and also immediately try to restart the Data API through the FileMaker Server admin command line:

fmsadmin restart fmdapi

What happens if we simulate a Data API crash? From the macOS activity monitor, we force quit the Data API process:

FileMaker Zabbix Terminate

Figure 7 – Using the Activity Monitory to force quit the Data API process

When we turn to Zabbix and look at the data for the Data API item, we see that it was showing ‘1’ (aka ‘running just fine’). It then dipped to ‘0’ (aka ‘not running) and fairly immediately went back to the running state:

Screeshot of the latest data for the Data API item in Zabbix for FileMaker

Figure 8 – Viewing latest data for the Data API item

A quick look at our FileMaker Server shows that the process has indeed returned. Note that it has a different process ID (PID) than it had before, an indication that the process has restarted.

Screenshot of macOS Activity Monitor showing Data API process with a different ID (PID) in Zabbix

Figure 9 – Different process ID (PID) indicates the process has restarted

In Zabbix, the problem event was updated to ‘RESOLVED’, and we can see the actions that Zabbix took and how fast it did it:

  • At 8:53 and 47 seconds it caught the problem.
  • One second later it executed the command.
  • 30 seconds later (the next data collection interval for the item), it determined that the process was running and the problem was closed.
Screenshot of the problem event and actions that Zabbix took

Figure 10 – Problem event updated

Other Zabbix Actions

Zabbix triggers and actions have much more power than what we are showing here, including escalation procedures. However, we hope this shows how Zabbix can give us both monitoring and healing capabilities beyond anything we ever had with just the FileMaker Server Admin Console.

Next Steps with Zabbix FileMaker Server Monitoring

The functionality to allow Zabbix to execute commands on your FileMaker Server is ready for you. We hope this convinces you to consider adding Zabbix to your toolbelt and fortifying your FileMaker Server deployments. Please contact our team if you have any questions!

Wim Decorte

Wim Decorte

Wim is a Senior Technical Solution Architect at Soliant. He is a FileMaker 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17 and 18 Certified FileMaker Developer and the author of numerous Tech Briefs and articles on FileMaker Server. Wim is one of the very few multiple FileMaker Excellence Award winners and was most recently awarded the FileMaker Community Leader of the Year award at the 2015 FileMaker Developer Conference. He is also a frequent speaker at the FileMaker Developer Conference and at FileMaker Developer groups throughout the world. In addition to being a renowned expert on FileMaker Server, Wim also specializes in integrating FileMaker with other applications and systems. His pet project is the open source fmDotNet connector class that he created.

3 Comments

Leave a Reply