BlogFileMaker

FileMaker 18: How to Send HTML Emails Using Insert from URL

By May 22, 2019 September 11th, 2019 18 Comments
8/15/19 update:

The original demo file used the TextEncode function to create the email message file. This function becomes excruciatingly slow when the email includes larger attachments; e.g. it takes longer than a minute to encode an email with a 3 MB attachment.

The demo file has been replaced with an updated version which uses the file manipulation script steps to write out the message file to a file in the temp folder and then read it back into a variable as a container value. This performs much better with larger attachments.

9/11/19 update:

The demo file has been updated to fix a bug (having to do with line endings) when sending non-MIME emails.

HTML Emails

FileMaker 18 added SMB, LDAP(S), and SMTP(S) to the list of protocols supported by the Insert From URL script step. The full list of supported protocols now is HTTP(S), FTP(S), FILE, SMB, LDAP(S), and SMTP(S). This blog post will focus on making use of cURL and the SMTP protocol to send HTML-formatted emails.

Prior to FileMaker 18, plain text emails could be sent using the Send Mail script step. FileMaker 17 introduced the ability to include multiple attachments. Now, fully stylized rich-text emails with attachments can be sent, but the process is not as simple as it was with the Send Mail step. This blog post breaks down much of what you need to know and supplements that with a demo file for some additional hands-on learning.

Insert From URL

To send an HTML email, the Insert From URL script step needs to have the following items specified:

  • Target – stores the result of the cURL command
  • URL – specifies the SMTP server URL where the cURL command will be directed to
  • Verify SSL Certificates – determines whether the SMTP server SSL certificate should be verified
  • cURL options – specifies the options used to fine tune the cURL command

URL

The URL specifies the protocol (SMTP or SMTPS), the fully qualified domain name of the SMTP server, and the port.

Here are some examples of SMTP URLs:

  • smtp://smtp.mydomain.com:25/
  • smtps://internal-smtp.mydomain.com:587/

cURL Options

The cURL options specify additional instructions for delivery of the email. Here are some of the options that you may use, along with some comments for each one:

–mail-from <address> This is the email address of the person sending the email. Specify the address only; e.g. llopez@email.com. Do not also include the name like this: Liam Lopez <llopez@email.com>.
–mail-rcpt <address> Use this option for the “To”, “Cc”, and “Bcc” addresses. Don’t place multiple recipients on a single line. Instead, repeat this option once for each recipient. For example:

--mail-rcpt cturner@email.com
--mail-rcpt awright@email.com
–user <account>:<password> Not needed if the SMTP server does not require authentication.
–upload-file <file> This option tells cURL where to find the message that is to be sent. In FileMaker, we can use a variable name, like $mailFile, in place of <file>. We’ll cover the message format of the mail file in more detail a bit later.
–trace <trace> This option causes all information from the command execution to be saved, which can be very helpful for debugging. In FileMaker, we can specify a variable, like $trace, for <trace>.
–show-error This option causes the Get(LastExternalErrorDetail) function to return useful troubleshooting information.
–ssl-reqd Require SSL/TLS. The SMTP URL should use the “smtps” protocol designation in this case.
–ssl Try to use (but do not require) SSL/TLS.

Message Format

The SMTP URL and cURL options tell cURL where to send the email, to whom, and how, but they do not directly specify what to send. Instead, the cURL options reference a mail file which includes the content of what is to be sent. This mail file, referred to as the message, has to be constructed following particular formatting rules.

The rules can get quite complex, and a big driver of the complexity is the fact that each new email standard aims to be backward-compatible with the older standard that it supersedes. Some examples of this are the rules that require content to use 7-bit US-ASCII characters with each line of text limited to 78 characters or less (more on that later).

In my testing, I’ve found variations in the extent to which some of these rules are enforced. My guess is this is due to differences in how the various SMTP servers and email clients have implemented the standards that define how internet email works, similar to how support for HTML and JavaScript is implemented differently across different web browsers. The lesson from this is that you may get something to work for a particular SMTP server or email client, but if one of those factors later changes, it may end up breaking the functionality.

MIME Formatting

In the simplest case, where the email message consists of only ASCII characters, none of the lines in the message is longer than 998 characters, and the email does not have any attachments, the message can be constructed using just one “part.” In more complex cases, the email message will use Multipurpose Internet Mail Extensions (MIME) formatting to handle different character encodings, to allow for attachments, or to include alternate (e.g. plain text) versions of the email message.

The message will always consist of a header and a body. In the case of MIME multipart messages, the body itself will be split up into multiple parts, each of which will have its own header and body.

Each part in a multipart message will specify a MIME type to indicate the type of content included in that part. Some examples of MIME types include:

  • Plain text: text/plain
  • HTML text: text/html
  • PNG image: image/png
  • PDF: application/pdf
  • Zip: application/zip

The full list of defined MIME types can be found here.

So that email clients can tell where one part ends and another begins, each part in a multipart message is separated by a boundary which has to be a string of printable 7-bit ASCII characters (codes 32 through 126), with a length of 70 characters or less, and it must not appear in any of the encapsulated parts. Here is an example of a valid boundary: gc0p4Jq0M2Yt08j34c0p.

Each time a boundary is used, it is prefixed with two hyphens. The last boundary is also suffixed with an additional two hyphens.

MIME Parts

There are three kinds of MIME parts, and therefore three kinds of part boundaries.

Alternative: The “alternative” boundary separates alternate versions of the message. For example, there could be a plain text version of the email to accompany the HTML version, so that old-school plain-text-only email clients can display the message correctly.

The other two types of MIME parts have to do with attachments. Attachments can be embedded within the email message, or they can be attached to the email message in such a way so that they have to be saved out of the email before they can be accessed.

Related: The “related” boundary separates embedded attachments from each other and from the email message.

Mixed: The “mixed” boundary separates non-embedded attachments from each other and from the rest of the message.

The various different parts are nested with the mixed parts forming the outer-most portion of the structure and the alternative parts forming the inner-most portion.

At this point, your head is probably spinning with all of the details, so hopefully, this visual explanation will help clarify your understanding:

Visual of sending email

Figure 1 – Visual explanation of a MIME-formatted email message

The main header section can contain fields such as From, To, Cc, Subject, and Date, but note that the To and Cc fields are not used to determine where the message should be delivered. That information is determined from the cURL options (covered above). The analogy is a paper letter that contains the destination address both on the envelope and on the letter itself. The address on the envelope is the one that is used for delivery. The address shown at the top of the letter is included for informational purposes only.

MIME Format Example

Here is an example of a simple, non-MIME formatted message:

From: Liam Lopez <llopez@email.com>
To: Cameron Turner <cturner@email.com>, April Wright <awright@email.com>,
 Theodore Hall <thall@email.com>
Cc: Zara Allen <zallen@email.com>
Subject: Test
Date: Wed, 14 May 2019 19:34:21
Content-Type: text/html; charset="utf8"

<html><body>Hello, here is my <b>bold</b> email message.</body></html>

A couple of things to point out:

  • The message body is separated from the header section with an empty line.
  • Each line should be separated by CRLF (carriage return and line feed) characters.
  • Each line should be kept to a length of no more than 78 characters. The technical limit is actually 998 characters, but the recommended limit is 78. Longer lines should be broken out into multiple lines with a whitespace character beginning each new line. This process is called “folding.”
  • The message (both the header and the body) must be comprised of 7-bit ASCII characters.
  • The Bcc addresses, if there are any, are typically not included in the message header.

For MIME-formatted messages, non-ASCII data can be converted to 7-bit ASCII using an encoding scheme such as “quoted-printable” or base64 (RFC 2045). When encoding content that consists of mostly Latin-based text, the “quoted-printable” encoding is preferable, since it is human-readable. This makes troubleshooting easier. Both of these encoding schemes automatically fold lines at lengths of 76 characters.

Quoted-printable Encoded Text

Here’s an example of what quoted-printable encoded text looks like:

Original:

Congratulations to Luka Modrić 🇭🇷⚽ for winning the Ballon d'Or!

Quoted-printable:

Congratulations to Luka Modri=C4=87 =F0=9F=87=AD=F0=9F=87=B7=E2=9A=BD for w=
inning the Ballon d'Or!

Unfortunately, FileMaker doesn’t provide a native quoted-printable encoding function, so I used base64 encoding in the demo that accompanies this blog post.

Additional MIME Examples

Here are some additional visual guides that show how email messages (both MIME and non-MIME) are put together.

The pilcrow character (¶) is displayed to denote places where an empty line should be inserted.

To simplify things, some of the text is represented using placeholders. This is denoted with angled brackets; e.g. <<EMAIL_MESSAGE_HTML_TEXT>>.

HTML Email Only

The first example is of a simple non-MIME HTML email.

When I tested sending out this kind of email, I was able to include non-ASCII characters, but the standard specifies that only printable 7-bit ASCII characters should be used. This is another example of the point made earlier, which is that different SMTP servers and email clients implement the standards in different ways, including building in support that allows for stretching the rules. However, if you want to ensure that your email implementation will continue to work regardless of what server or client is used, better adhere to the rules defined in the standards.

Screenshot of a simple non-MIME HTML email

Figure 2 – Simple non-MIME HTML email

HTML Email + Attached Files

This second example shows two attachments included along with the HTML email. In this case, the attachments are not embedded within the email message.

Screenshot of MIME example of two attachments included with the HTML email

Figure 3 – Two attachments included with the HTML email

The preamble message is there so that non-MIME compliant email readers can have some intelligible message to show to the user. This message can be anything you want. For example:

This is a message in MIME format.  If you see this, your mail reader does not support this format.

HTML Email + Embedded Files + Attached Files

The following example adds two additional attachments, but unlike the previous two, these new ones are embedded within the email. This is done by setting the content disposition to “inline”.

Screenshot of MIME example of two attachments embedded within the email

Figure 4 – Two attachments embedded within the email

Embedding attachments is commonly done with images. When the HTML for the email is constructed, these attachments can be referred to by their content ID, which is defined in the part header. For example, suppose that the content ID for one of the attachments is defined as follows:

Content-ID: <image001.png@01D4FC56.1590DFB0>

Then the image can be referenced inside the HTML email like this:

<img src="cid:image001.png@01D4FC56.1590DFB0">

Plain Text Email + HTML Email + Embedded Files + Attached Files

The last example uses “alternative” boundaries to add a plain text version of the email message.

Screenshot of MIME example for adding a plain text version of the email

Figure 5 – Adding plain text version of the email

Using Encoded-Words in Message Headers

The preceding examples describe how a message body can be encoded to account for non-ASCII characters, but what if the message header also contains these unsupported characters?

For instance, the name of a recipient or the subject of the email could contain characters that fall outside of the 7-bit ASCII range. In that case, you can use encoded-words to convert sequences of unsupported text using the following syntax:

=?charset?encoding?encoded-text?=

For example, suppose the email subject consists simply of the following smiley face emoji: 😋

As a base64 encoded-word, the subject message header would appear as:

Subject: =?UTF-8?B?8J+Yiw==?=

In the example above, the character set is specified as “UTF-8”, the encoding is specified as “B” (for base64), and the encoded text is specified as “8J+Yiw==”, which is the base64 equivalent of 😋

The two encoding options are “B” for base64 and “Q” for quoted-printable.

Similarly, the following recipient message header…

To: 老子 <laotzu@email.com>

…could be encoded as follows:

To: =?UTF-8?B?6ICB5a2Q?= <laotzu@email.com>

You can read more about encoded-word syntax in RFC 2047.

Conclusions and Demo File

We now have the ability to send HTML emails with attachments, but the step up in complexity is significant compared to the Send Mail script step.

Get the demo file

The accompanying demo file includes a script which does a lot of the heavy lifting for you. The script was written to be portable so that you can copy it into your solution without having to set up any tables or fields. (The script does rely on the “GetMIMEType” custom function though, so you will need to copy that before copying the script.)

The demo file supports fully stylized emails which can be achieved through HTML or through FileMaker field rich text formatting (using the GetAsCSS function). Attachments can be designated as embedded (“Inline”) or as regular, non-embedded (“Attachment”).

The demo file does not, however, support alternative MIME parts, which are used to provide alternate (e.g. plain text) versions of the email message. And it does not do any folding or encoding of message headers. So if your message headers (e.g. subject or recipient email addresses) use non-ASCII characters or are longer than 998 characters in length, the email might not get sent out.

If you do not have an SMTP server readily available for testing, you can create a Gmail account and use the Gmail SMTP server. If you have two-factor authentication enabled (and you should), you will need to create an application-specific password. You can follow these instructions to do so.

As a final word, I’d like to thank my colleagues Brian Engert and Marcelo Piñeyro for their assistance with preparing this blog post and the demo file.

References

Mislav Kos

Mislav Kos

Mislav is a FileMaker developer and a Senior Technical Project Lead at Soliant Consulting.

18 Comments

  • Avatar Will Bullard says:

    Really great article Mislave. Thank you for sharing.

  • Avatar Da says:

    Thanks, Mislav – this saves me a lot of research and head-scratching!

  • Avatar Teo says:

    Wonderful compilation of knowledge about HTML emails!
    Cannot wait to implement it in our solutions.

  • Hi Mislav,

    Thx for the great blog post!

    ( small errata in figure 5: first “text/html” should be “test/plain” I think… )

    Kind regards,

    Jan

  • oops, I now made a typo myself ;-( not “test/plain”, but “text/plain” of-course…

    • Mislav Kos Mislav Kos says:

      Jan, thank you for the correction. You’re right – that should say “text/plain”. I’ll work on getting that fixed shortly.

  • Avatar Shawn says:

    I am finding that when sending emails with FileMaker html as fmp protocol hyperlinks that Gmail seems to strip out the hyperlink so that they are not clickable? The links work in ios mail though but not Gmail? Can you confirm the same behavior or if there is a workaround?

    For example the View FileMaker link below does not work in Gmail but the other website http link does but both of them work in ios mail:

    From: My Name
    To: Sales
    Subject: HTML Email Test
    Content-Type: text/html; charset=”utf8″

    Body of <b>email</b> <a href="//$/mydatabase.fmp12?script=FindPOList&amp;$param=list&amp;$po=p60429,p60448,p60449" rel="nofollow">View FileMaker Records</a> <a href="https://www.mywebsite.com&#x22; rel="nofollow">My Website</a> qty: <strong>1</strong>

    Thank you,
    Shawn

    • Mislav Kos Mislav Kos says:

      Hi Shawn, I haven’t encountered this situation yet. If I have a chance to look into it, I’ll be sure to post back here with my findings.

    • Mislav Kos Mislav Kos says:

      Shawn,
      I looked into this a bit further, and as it turns out, Gmail strips out protocol anchors from all non-http/https links. See https://stackoverflow.com/questions/23575553/ios-deep-linking-is-stripped-out-in-gmail. A possible workaround could be to link to a web page that you create that then redirects the user to the file protocol link. This would open a new browser tab and then open FileMaker.
      — Mislav

    • Mislav Kos Mislav Kos says:

      Something like the following might work. (Courtesy of my colleague Brian Engert.) This will take any parameters passed to the php script and send them on to FileMaker. Save it as a php file – e.g. fmp.php – and host it on a server somewhere.


      <html>
      <head>
      <meta http-equiv="Refresh" content="0; url=fmp://$/mydatabase.fmp12?<?= http_build_query($_GET) ?>" />
      </head>
      <body>
      Opening filemaker
      </body>
      </html>

      Your email would then include a link like this:
      https://my.server.com/fmp.php?script=FindPOList&$param=list&$po=p60429,p60448,p60449

  • Avatar Paolo di Mauro says:

    Hi,
    Thanks a lot for your sharing.

    I find twi drawbacks in sending mail using direct smtp instead of email client:

    – I do not have a copy in sent folder
    – i cannot take advantage from email-client contact auto completion (using fm field as address container)

  • Avatar Paolo di mauro says:

    Hello,
    is there a way to upload email to sent folder via IMAP ?
    Thanks,

    • Mislav Kos Mislav Kos says:

      Hi Paolo, I might not be understanding your question correctly, but if you looking to send mail via IMAP, that won’t work. Mail is sent out using SMTP and retrieved using IMAP or POP3. FileMaker’s Insert from URL script step supports using cURL with SMTP but not IMAP or POP3. Hope this helps.

      • Avatar Paolo di mauro says:

        Hi Mislav,
        i was not that clear, sorry, I would like to have a copy of the email sent, in the sent mail folder.
        Thanks again

        • Avatar Paolo di mauro says:

          up

        • Mislav Kos Mislav Kos says:

          As I understand it, SMTP is used to send emails only. It doesn’t know about email folders. To have the sent email stored in your Sent folder, you would have to make an IMAP call separate from the SMTP call. But unfortunately IMAP is not a supported FileMaker cURL protocol.

          Possible workarounds:
          (1) If you are sending email via SMTP using an Exchange server, the email will be added to the Sent folder automatically, so long as you use the username and password for the appropriate user account.
          (2) You could make a separate IMAP call using some non-native method (e.g. a plugin) to save the email in the Sent folder – see the IMAP APPEND command.
          (3) BCC yourself on the email (but note that the email will be received in your Inbox, not your Sent folder).

  • Avatar Jason Young says:

    REALLY clever solution with encoding the subject with base64! I was all set to come up with some nutty custom function to solve this and am glad I took a closer look!
    Thank you,
    Jason

Leave a Reply