BlogFileMaker

How to Fully and Correctly Use FileMaker’s Recovery Tool

By May 4, 2016 September 5th, 2019 13 Comments

File corruption. It happens. It’s part of software, and sometimes no matter how hard we try to prevent it, it still happens. This guide will detail out how to determine if your FileMaker file is corrupted, what to next, and how to prevent it from corrupting again in the future.

How do I know if my file might be corrupted?

You may suspect your file is corrupted if you are experiencing and of the following:

  • Repeated crashing of the FileMaker file.
  • Unexpected or unexplained behavior (for example an odd behavior will start happening once in a while and the only way to fix it is to restart FileMaker).
  • Any time you shut down your server machine without properly closing down the files in FileMaker Server. If that ever happens, you should verify there is no corruption. For example, if there was a power outage.
  • If your OS level backup software on your server (i.e. Time Machine, Crashplan, etc.) is backing up the hosted FileMaker files, they could get corrupted. You should always back up your backup copies, not the hosted files located inside the  FileMaker Server > Data > Databases.
  • If you are working in Manage Database on a hosted file and you lose connectivity to the file while Manage Database is open, your file may be corrupted.
  • If the file size increases dramatically from one backup to the next, without a large data increase to explain it, it’s possible that corruption is the cause.
  • Generally, if something seems fishy, it’s worth ruling out corruption!

How can I verify if my file is corrupted?

FileMaker has a Recovery feature that allows you to determine if your file is safe to use or not. Here is a step by step guide to running a recovery process:

  • Grab a recent backup of the file you think may be corrupted.  Ideally, move this file to a non-server machine (it’s best not to take up resources on the server machine)
  • Make a compacted copy or clone of the backup. A clone is nice because it will make the recovery faster, but it will not tell you if the corruption is in your data and not your schema.
  • Open FileMaker, and from the menu select File > Recover…
  • Select the compacted copy/clone and run the recovery (you can optionally check the consistency of the file first, but no matter the result of the consistency check, you should continue with the recovery process)
  • You will be asked if you want to rename the old file and select a location of the new recovered file
  • Once the recovery is complete, FileMaker will alert you if there were any issues found. If there were issues, your file is corrupted.

It is important to note that you should never use the file generated during the recovery process as your production file going forward. The recovery process is a diagnostic tool. Once you have performed the recovery, you will be using the log. The recovered file can really go in the trash, since it should not be used (an exception would be if you need to export the most recent data).

My file is corrupted. Now what?

Option 1: Revert to a Backup

  • Gather your recent backups and run them through the recovery process until you find the most recent backup that passes the recovery check
  • Create a clone of the good backup
  • Create a compacted copy of the clone

Note: Be careful, if there has been any code or layout or schema changes in production since the last good backup, you will need to replicate those changes.

  • Perform a data import into this new file with most recent data
  • Host your updated file as a replacement of your corrupted file

Option 2: Perform Surgery

Before going into these steps, I want to really stress that you need to be saving copies of the file as you go along! This is really important! At every step of the way, you make a copy. You don’t want to start the surgery only to half way through discovering that you will have to redo all the changes you made for whatever reason, or even worse, you don’t have an original copy anymore because you have been making changes to the original. I recommend creating a new folder for each stage you are at and have the folder name be a very specific description of where you are in the surgery.

For example your folders may look like: Original, CF_Removed, CF_And_Relationship_Removed, and so on.

Picture of plant growing
  • Open the log. When you perform a recovery, a log file gets generated. You can look through this file to get a little more detail on what has failed.
  • The 2nd column is an error code column. For most of the rows, this will be a “0”. Scroll until you find a non-zero number; this is where your corruption is. There may be more than one location.
  • Look for the location of the error, and determine if you can “perform surgery” to take it out. For example, if the log determines it’s a specific layout, try deleting all the contents of that layout and running the recovery again (making backup copies of each step along the way!). If it is a script, try deleting the contents of that script. If it’s a custom function, delete the contents of the custom function, etc. I recommend not deleting the script, layout, custom function, etc. in its entirety (at first) because you will then lose the references throughout your file. If the recovery still fails after deleting the contents, and the log says it’s failing at that layout/script/custom function/etc, then I think it’s appropriate to delete the item and run the recovery again.

Tip: If you’d like to be more conservative, look at your failed item and see if there’s anything complicated/unusal about it and then remove just that piece and run the recover. For example, if your layout is failing and you have non-native icon or image, try deleting that first instead of deleting all the layout contents.

  • Repeat the surgery until you’ve removed all failed items and your file passes a recovery check.
  • You will then have to rebuild those items you have modified or deleted. Ideally, do not cut and past anything; rebuild from scratch. You don’t want to risk re-introducing corruption. You may want to copy the corrupt objects into myFMButler’s Clip Manager in order to inspect and repair the XML, which you can then past back into your file.
  • If you think it’s the data that might be corrupted, try running recovery on a clone and see if it passes. If you determine the data is corrupted, export the data as .merge files, but rename those files to have a .csv suffix. This should take care of the bad data. Then import that data back into a clone of your file.
  • If the surgery fails, you will have to look into other options, such as option 1 above: Revert to a Backup.

What Can I Do to Prevent Corruption?

There’s not a whole lot you can do to prevent corruption, but there are things you can do to minimize the damage:

  • If possible, get a backup generator for your server machine in case there’s a power outage. This will prevent unexpected shutdowns of your server machine without proper closing down of the files
  • Make sure your OS-level backup software on your server (i.e. Time Machine, Crashplan, etc.) is not backing up the hosted FileMaker files. You should always be backing up your FileMaker backup files, not your hosted files.
  • If you are working in Manage Database on a hosted file, try to be as quick as possible. Don’t leave Manage Database open while you go get a cup of coffee. You should be in and out really fast.
  • Backup your files as frequently as you can (within reason) and try to keep your backups for as long as you can. When corruption happens, the more backup files you have to work with, the better the options you will have.
  • If you have files that seem to get corrupted often, check them on a regular basis. Put a reminder in your calendar if you need to. Doesn’t hurt to be proactive!
  • Compact the file on a regular basis or whenever you get the chance. The smaller the file, the fewer blocks available for corruption.

My Experience with Corruption

I have had some interesting corruption causes; I’m sure some of you can relate. Here is a list of issues I have determined to cause corruption in the files I worked on, along with what some of my colleagues have experienced:

  • Corrupted text/image on a layout (deleting this one object on all my layouts allowed the file to pass the recovery)
  • Corrupted Fonts on a Mac (going to Font Book > Restore Standard Fonts… fixed this)
  • Unexplainable constant crashing on a specific layout. I was creating a new report, and whenever I went into browse mode, the table assignment for that layout would disappear and FileMaker would get very slow and eventually crash. When comparing backups we realized this file also jumped drastically in file size, and it wasn’t due to a large record volume increase or container usage increase. Also this file was passing the recovery! I ended up reverting from a backup (before the large file size increase) and it has been working fine since.
  • I have had a file fail recovery because of an uninstalled plugin when I ran the recovery. I ran the recovery on the same file with a machine that had the plugin installed and it passed.

If you have any interesting recovery stories or tips I would love to hear them!

Makah Encarnacao

Makah Encarnacao

Makah is a Technical Project Lead that hails from Albuquerque, NM. She joined Soliant a few months after graduating from UCLA in 2007, and must like it a lot because she’s still here! She is certified in FileMaker 9, 10, 11, 12, 13, 14, 15, 16 and 17, and placed 4th in the 2015 FileMaker DevCon Developer Cup. She was a speaker at the 2016 and 2018 FileMaker Conference. When she is not coding, she is enjoying life with her family that recently increased by 1.

13 Comments

  • You say “Make a compacted copy or clone of the backup” first. This is actually a bad idea. Compacting a file re-indexes the File Blocks, and might fix a Consistency Problem, before the Recover, and a Clone can actually remove structure it does not like ( a nasty quirk ). It is best to Recover the original, to see all the problems

    More info is here: http://www.fileshoppe.com/recover.htm

    greg

  • It should be noted that sometimes FileMakerΓÇÖs Recover cannot run to completion and hence it is impossible to repair the database with FileMakerΓÇÖs tools at all. If the privileges section seems to be broken (a common error), FileMaker in many case is unable to recover the file. In such cases another mean e.g. a repair service is needed.

    FileMakerΓÇÖs Recover might also indicate some ΓÇ£minorΓÇ¥ problems that actually do no harm and are no indication of a broken database. So the list of problems reported by Recover needs be checked carefully to see if there are only problems reported that can be ignored safely. For instance recovering without a needed plug-in raises some ΓÇ£problemsΓÇ¥, but the original file is OK as just the plugin is missing.

    For recovering a database it is often useful to keep a clone of the most recent version of database around. This clone can then be used to import any saved dataΓÇöof course starting from a copy of the clone. Using FMDiff it is quickly possible to check whether a working copyΓÇÖs structure (layouts, scripts etc.) was modified and whether there are problems with the databaseΓÇÖs structure. The clone is also helpful when a repair service tries to attempt to put a file back into working condition as it may need parts from the clone (e.g. accounts information).

    • Again, it is better to keep regular backups, rather than clones. A clone can have pieces missing.

      This was the basis for the original FMDiff. Winfried noticed that you can detect “corruption” by comparing a file to it’s clone. They end up different

      greg

      > For recovering a database it is often useful to keep a clone of the most recent version

  • […] starters, read Makah Encarnacao's extensive guide to using the FileMaker Recovery Tool. ┬áBe sure to read the comments on her blog post, as […]

  • Avatar Dave Evans says:

    I ran the instructions and found a few problems. If “Surgery” is the way I decide to go, which file should I be working on to get started?
    On reading the log file, does the non-zero error code follow the problem area or does it come before it? Any resource for error codes and what they mean?

    • Avatar Makah Encarnacao says:

      Hi Dave,
      I recommend taking your original file, make a copy of it, do the surgery, run a recovery to see if the surgery worked, and if it did not work, I recommend taking the original file and making another copy. That way you try to isolate the problem and pinpoint the corrupted component. The recover log will give you an error code for each step of the way. So for example, you might get a result like this:
      0 Recovering: custom menu set ‘Contacts’ (5)
      0 means there was no error, but if there was an error, I know it would be with the custom menu set ‘Contacts’, so I would try deleting that and running the recovery again.

      Hope that helps. Let me know if this does not answer your question.
      Makah

  • Avatar Dave Evans says:

    Thank you so much Makah. I have got the Recovery process running now where it gives me an “Everything’s OK” (so to speak) message at the end. But when I open the Log file, near the end, when it analyzes the Themes, I have several log entries that look like this…

  • Avatar Dave Evans says:

    2019-04-16 13:18:59.792 -0500 EITD_Launchpad.fmp12 0 Recovering: theme ‘com.filemaker.theme.custom.7311F36F_7FE6_6748_9899_D1417865473A’ (1)
    2019-04-16 13:18:59.796 -0500 EITD_Launchpad.fmp12 8476 This item changed

    I have 10 of these entries (4 for my custom themes and 6 for FM themes).

    Should I worry about these since the Recovery tool says everything’s legit?

    • Avatar Makah Encarnacao says:

      That’s a great question. My guess is that if the recovery passes then you don’t need to worry about it. It may be some blocks in the backend or some indexing of some kind. But I will ask around and see if I can get a more knowledgeable response for you.

  • Avatar Dave Evans says:

    Thanks so much for your help and for this article.

Leave a Reply