FileMaker 16: cURL and Container Fields

FileMaker 16 introduces newly available options to the “Insert from URL” script step that utilize the populate cURL library. If you are not familiar with cURL, it has long been a standard for command line interaction with all manner of internet protocols and applications. The introduction of these options heralds a host of new options for FileMaker developers looking to interact and integrate with all kinds of web services.

In this post, we will focus on one particularly useful feature: the ability to upload a file stored in a container field directly from the field. That means you do not need to write the data out to the file system before referencing it with your script.

It makes good sense. The location of the data, the contents of the container field, is known, so you should be able to reference it directly without having to save it somewhere locally first. Now we have a way to do just that, even if it is not apparent at first. Admittedly, the syntax was not intuitive for me when I first tried to get this working.

The Need For This Functionality

There are times when you need to upload a file to a web page, and in fact, most web scripting languages have built-in functions to handle file uploads. After all, the HTML form input type “file” has been around forever to allow exactly this sort of functionality. We know we can handle file uploaded on the web server side of things. We will assume that the file is being transferred as binary data and not Base64 encoded. Some web services may require it and there may be limits on the amount of Base64 text you can transfer.

What we do with the file once it is uploaded is irrelevant to this blog post. Our focus is on how to make that request from a FileMaker 16 script. We will assume that the web scripting side of things works as expected.

Enter cURL

Luckily for us, the cURL functionality needed for this is included with FileMaker 16. External scripting or third party plugins are no longer required. Since “file” is a valid input type for HTML forms, the cURL library includes support for submitting this data as part of an HTTP request, most commonly transferred with the POST method. This means that the payload of the request is not sent as part of the URL string like you would have with a GET request. Consequently, an arbitrarily large request can be made without running into URL string limits.

cURL syntax with FileMaker Variables

Let’s review the relevant steps. First, we set a variable to the value of our container field.

Set Var: ["$myFile" ; Table::UploadField_r ]

How can we use this variable in our cURL expression? The “-F” flag, which is short for “form”, allows cURL to emulate a filled in form in which the user has pressed the submit button.

We can use this flag with the new “Insert from URL” cURL options like so:

-F "file=@$myFile"

Note that when we include the variable in the cURL options, it appears inside the quotation marks. This is the correct syntax; placing it outside the quotes will not work.

This is analogous to how a file name is specified inside of quotation marks in an HTML form element:

<input type="file" name="myFile" />

Similarly, on the server you can access the data the same way any uploaded files are received. For example, with PHP, you can use the $_FILE variable to handle and save the file on the server.

Side Effects

A side effect of this syntax is that, while it allows you to pass the binary data directly from a container, the file name that appears on the receiving end will be the name of the variable you use. In the above example, the file name would be “myFile” which may not be what you want.

A possible solution will be to pass the file name as a separate parameter and then rename the file on the server. However, if you are not in control of the web scripting, that won’t be an option.

To make it easier for web services to consume, we can set the variable name to the file name of the container data. The trick here is to do this dynamically and to avoid illegal characters and reserved words. Note that variable names are subject to the same rules as field names.

A simplified version of this can be done with a few variables and using the Evaluate and Let functions to dynamically set a variable.

Set Variable [$myFileName ;
Substitute ( GetAsText ( Detail::Upload_File_r ) ; [" " ; "_"] ; ["(";""] ; [")";""] ;  ["+";""] ; ["-";""] ; ["=";""] ; ["≠";""] ; ["&";""] ; ["<";""] ; [">";""] ; [",";""] ; [";";""] ; ["[";""] ; ["]";""] ; [":";""] ; [""";""] ; ["$";""] ; ["/";""] ; ["*";""] ; ["^";""] ; ["}";""] ; ["{";""] )  ]
Set Variable [ $null ; Evaluate ( "Let ( $" & $myFileName & " = " & "Detail::Upload_File_r" & " ; "" )" ) ] 
Insert From URL [ Select ; With dialog: Off ; $your_result ; $your_url ; cURL options: Evaluate ( ""-F file=@$" & $myFileName & """ ) ]

You may also note that the target of the result of the insert can now be a variable. This is a new change in FileMaker 16. You can now call this function without requiring a field to be placed on a layout to hold the result, as was the case before.

Testing

As mentioned above, you can write your own web script and host it on your web server. You can also use a service like Request Bin to inspect HTTP requests that are made. If we do that, we can see that the file gets submitted as part of the HTTP request and that it is sent as binary data.

Feel free to try it out yourself from the sample file.

The sample file also includes a sample script to download a file directly to a container field. Although the functionality shown is not an entirely new feature to FileMaker, the part that is new in 16 is the ability to verify SSL before downloading data. The sample file has this new feature enabled.

Of course, this just scratches the surface of what you can do with FileMaker 16 and the newly introduced cURL options. I look forward to seeing what can be done with it myself and seeing what others can come up with.

References

20 thoughts on “FileMaker 16: cURL and Container Fields”

  1. Hi Mr. Duncan, thank you for this post; I found your solution really useful. Quick question though – what about multiple files, is that achievable in a single “InsertFromURL” command?

  2. Hi Mike,

    Thank you very, very much for this post. Specifically the fact that the file variable has to be in between the quotes. For an old fashioned FileMaker developer who knows little of http Post and cURL this was not imaginable without your help.

    Some remarks or questions.
    – Should the -F in: -F “file=@$myFile” not be within the quotes, as in: “-F file=@$myFIle” ?
    – In my case I had to post data in csv format. The data are gathered in a script in a loop into the variable $data, just before the Insert from URL. As all the help texts and explanations talk about uploading ‘a file’, I thought that I should give the file path in the Insert from URL. After a lot of trial and error I understood that in my case the ‘file’ is the data itself. So, in the script the variable $data is used in the Insert from URL as in:
    “-F file=@$data”.
    Do I misunderstand the use of the word ‘file’ here? Could it be a path to a file, or should it always be the content of a file? (If you understand what I mean.)

    1. In this example, “file” is just the name of the parameter that gets passed. Similar to “name=Martin” where “name” is the key and “Martin” is the value. Of course you would need to adjust according to the requirements of the web service or script you were submitting to.

  3. Hi Mike,

    I’m looking to hopefully take this solution one step further. Is it possible to use FileMaker’s new cURL functionality to upload say a jpg file in the container field up to a bucket on my Amazon S3 account? and if so, do you have any pointers on the calculation formula to use when specifying the cURL options?

    1. Possible? I think so, but AWS uses a header signature that is non-trivial to create, and is different for each service. I am actually working on that now, and can share when done. Usually you might use a SDK to do the heavy lifting for this part, but with cURL options, have to build it all yourself.

  4. Mike,
    Thank you for your great insights into avoiding the issues of uploading files, via FMP’s Insert From URL command.
    Your solution was quite creative!
    I however found that your coding of literal terms might cause a problem, should the container file name or its table name change.

    I added the following lines to keep the names as references rather than as strings.
    …
    Go to Field [Table::UploadField_r]
    Set Variable [$TableName_DoubleColon_ContainerFieldName ; Value:Get ( ActiveFieldTableName ) & “::” & Get ( ActiveFieldName )]
    Set Variable [$null; Evaluate ( “Let ( $” & $myFileName & ” = ” & $TableName_DoubleColon_ContainerFieldName & ” ; \”\” )” )]
    …

    After that the command to Insert From URL will have an active reference to post, and less likely to error on not finding a document or other data to upload, due to a static name.
    I’d appreciate your thoughts on this, considering that your creativity has brought us all this far.

    1. Sure, and I have seen others use custom functions to abstract table and field names. In presenting the technique, I hope it is easy to understand, and if this abstraction is needed it can be added. Thanks for you input!

  5. Think you can use the filename to tell the remote server what you named the file, from the curl manual

    “You can also explicitly change the name field of a file upload part by setting filename=, like this:
    curl -F “file=@localfile;filename=nameinpost” url.com”

    1. This is right, but may be dependent on the server, and what is handling the file on that end. You mileage may vary 🙂

    1. That would be true, since creating variables dynamically will also be constrained to field naming restrictions, so no leading numbers. Also no dashes or other characters not allowed in field names.

  6. Your script works well where the variable, $myFile refers to a container field such as “Table::UploadField_r”, with no repetitions, or to the first repetition in that container field.
    The variable, however, fails when a repeating variable $myFile[$i] is set to any repetition greater than 1, ($i>1), in that container field.
    What might be an appropriate syntax to refer to a container field with repeating fields?

    1. Not exactly sure how your code works, but if you are doing a loop and performing the curl, it may be possible to set another variable from the repetition and use that instead. Would that work?

  7. The problem arises in a loop, after $i has incremented above 1.
    In the following example, after $i has been incremented to 2, although the variables $myFileName and Detail::Upload_File_r[$i]” are both assigned string values, the expression

    [Evaluate ( “Let ( $” & $myFileName & ” = ” & “Detail::Upload_File_r”[$i] & ” ; “” )” ) ]

    results in a question mark ? when monitored in the Edit Expression pane under Data Viewer’s Watch tab.

    1. Looks like the quoting is off a bit, you might try to test outside of the “Evaluate” to get the string right, then Evaluate it. Should be more like this I think…
      “Let ( $” & $myFileName & ” = ” & “Detail::Upload_File_r[” & $i & “] ; \”\” )”

      1. Mike, thank you for your great help. Your corrections fixed my script, quite well.
        I have since attempted to remotely store the container field contents using Field Options “Options for Field”, Storage tab, store container externally, relative to: [hosted location]/Files/…, with the Secure Storage option.

        My FM script contains the following lines:
        1. Set Variable [ $PDF_Container_Field ; Value: GetContainerAttribute ( GetRepetition (Appointments::PDFContainer ; 1) ;”filename” ) ]
        2. Set Variable [ $myFileName : Value: Substitute ( GetAsText ( $PDF_Container_Field ) ; [” ” ; “_”] ; [“(“;””] ; [“)”;””] ; … ] per your earlier blog instructions
        3. Set Variable [ $null ; Value: Evaluate ( “Let ( $” & $myFileName & ” =” & “($PDF_Container_Field)”& “; \”\” )” ) ]
        4. Set Variable [ $POST_Options_Attachment ; Value: ” ” -s -X POST -F \”attachment[document]=@” & Evaluate ( “\”$” & $myFileName & “\”” ) & “;type=application/pdf\” -F \”attachment[report_type]=09\””]
        5. Set Variable [ $DB_Full_Path ; Value: “https://… ]
        6. Insert from URL [ Select ; With dialog: Off|; Target: $DB_URL_Result ; $DB_Full_Path ; cURL options: $POST_Options_Attachment ]

        Although the FM Insert from URL script still works and returns no cURL error, its uploaded PDF document now registers errors, when viewed from the recipient url’s perspective.

        Questions:
        a. Can externally storing a container field be contributing to this error of PDF transmission?
        b. If so, what might be some possible corrections?

        Thank you for your great instructions, in the past, as well as any insights you might share in regard to my new conundrum.

        1. Does it work if you do not use external stored container data? You might also try to open the uploaded pdf file with a text editor to see if there is any error text in it.

          1. I was able to correct the container field storage error, by assigning different variables to the PDF FM container field repetition, and the PDF file name,
            => Set variable pointing to the PDF Container Field Repetition
            1. Set Variable [ $PDF_Container_Field ; Value: GetRepetition (Appointments::PDFContainer ; 1) ]
            => Set variable pointing to the PDF file name:
            2. Set Variable [ $PDF_File_Name ; Value: GetContainerAttribute( $PDF_Container_Field ;”filename” )
            3. Set Variable [ $myFileName : Value: Substitute ( GetAsText ( $PDF_File_Name ) ; [” ” ; “_”] ; … ]
            4. Set Variable [ $null ; Value: Evaluate ( “Let ( $” & $myFileName & ” =” & “($PDF_Container_Field)”& “; \”\” )” ) ]

Leave a Comment

Your email address will not be published. Required fields are marked *

We're celebrating 20 years! Read about our journey here.

Party horn and confetti
Scroll to Top