Dynamically Consuming APIs

Developers may want to consume external RESTful APIs to bring data into the Arc Flow and retain full control over the parameters and specifications of the request. This guide details how to consume RESTful APIs via the REST Connector when the API requests should be dynamically populated.

• Dynamic Request Body
  • Raw Request Body
  • Form Data
    • Example: Creating a New Connector
• Dynamic URLs
  • AllowArcScriptInURL
  • Best Practice: The Message Item
  • ArcScript in the URL

The REST Connector builds dynamic REST requests by parsing data from input files. Each file processed by the connector results in a new request with parameters that reflect the values in the file.

The REST Connector supports two types of dynamic request bodies:

• Raw request body
• Form data

When the Body Type field of the REST Connector is set to ‘raw,’ the entire contents of input files will be used as the request body. Any steps necessary to curate and manipulate the body of the REST request should occur prior to the file being processed by the REST Connector.

This option provides the most flexibility for generating arbitrary API requests, but may be more work intensive than using the form data approach described below.

When Body Type is set to ‘form-data’ or ‘x-www-form-urlencoded,’ the REST Connector is configured with a set of Fields in the Body section of the settings panel. Each Field configured as ‘Dynamic’ in the connector will be populated with values from the input file.

Input files should be XML files with the following structure:

<Items>
  <FormData>
    <FieldName1>value1</FieldName1>
    <FieldName2>value2</FieldName2>
  </FormData>
</Items>

The REST Connector will parse the input file looking for elements that match each dynamic Field name (note that these elements must be inside a FormData element as shown above). When a matching element is found, the value within that XML element is used as the Form value for the corresponding Field.

Since the XML structure for input files is specifically defined, it is strongly recommended to use the XML Map Connector prior to the REST connector to transform any arbitrary XML data into the appropriate structure.

Below is an example of consuming Arc’s Admin API to create a new connector via the REST Connector. The type and name of the connector will be dynamically read from input XML files. The same principles for consuming Arc’s Admin API extend to dynamically consuming external RESTful APIs as well.

The API request for creating a new connector in Arc has the following required parameters:

• ConnectorID (i.e. the name of the connector)
• ConnectorType (e.g. REST, Script, AS2)
• Workspace (e.g. Default)

Thus, the REST Connector will have these three Fields configured in the Body section of the Settings tab. For this example, the Workspace parameter will be statically set to ‘Default,’ and the remaining parameters will be dynamically populated from input files:

Files processed by a REST Connector with this configuration should have XML contents like the following:

<Items>
  <FormData>
    <ConnectorType>SFTP</ConnectorType>
    <ConnectorID>newSFTPConnector</ConnectorID>
  </FormData>
</Items>

A static URL may not always be feasible when consuming a REST API, for example query parameters may need to be added to the URL depending on data from an incoming Arc message. The REST Connector supports writing ArcScript in the URL field to generate a dynamic URL.

In order to enable dynamic URLs, a configuration setting must be enabled directly in the REST Connector’s port.cfg file. This file can be found at the following filepaths by default:

Windows: C:\Program Files\CData\Arc\data\[connector_name]\port.cfg
Java: ~/arc/data/[connector_name]/port.cfg

Note that if the REST connector is in a non-default Workspace, the name of the Workspace will replace the ‘data’ folder in the above path.

Within this file, the following setting must be set to true:

AllowArcScriptInURL = true

Once this value is set in the config file, the connector will interpret ArcScript within the URL field.

The recommended way to utilize ArcScript in the URL field is to reference the _message item. This special item represents an Arc message (a message is a file passing through an Arc Flow along with headers/metadata related to that file), and can be accessed like any other ArcScript item.

The steps are as follows:

• Use a Script Connector to write a script that fetches/generates/calculates any desired dynamic values (values that should be used later in the URL)
• Promote these values as custom headers of the Script Connector’s output file
• Pass the file (along with the custom headers) to the REST Connector
• Use the _message item to reference these custom headers in the URL field of the REST Connector

For example, imagine the need to fetch a query parameter from a database prior to making a GET request. The first step is to write a script that queries the database and promotes the results as a message header:

<!-- configure database query -->
<arc:set attr= "myDb.conn" value="Server=hostName;Port=3306;Database=mydb;User=test;Password=test" />
<arc:set attr="myDb.driver" value="System.Data.CData.MySQL" />
<arc:set attr="myDb.query" value="SELECT targetColumn FROM my_table WHERE otherColumn = 'value' "/>

<!-- execute query and store output in myDb.output -->
<arc:call op="dbQuery" in="myDb" out="tempOut">
  <arc:set attr="myDb.output" value="[tempOut.db:targetColumn]" />
</arc:call>

<!-- generate output file -->
<arc:set attr="outputItem.filename" value="myOutputFile.txt" />
<arc:set attr="outputItem.data" value="" /> <!-- no file contents required in this example -->

<!-- promote a custom header in the output file -->
<arc:set attr="outputItem.header:myCustomHeader" value="[myDb.output]" />

<!-- push the output -->
<arc:push item="outputItem" />

This script should be placed in a dedicated Script Connector, and that Script Connector should be connected to the REST Connector in the Flow.

After a message is passed from the Script Connector to the REST Connector, the syntax for referencing this custom header in the URL field is [_message.header:header_name]. For example:

http://localhost:8001/api.rsc/connectors(ConnectorID='[_message.header:myCustomHeader]')

Multiple dynamic values can be passed via this _message item by giving each value a unique header name and referencing each header name in the URL.

In addition to the _message item approach described above, custom logic can be implemented in ArcScript directly within the URL field of the REST Connector. Script contained within <arc:*> elements will not be included in the resulting URL and will instead be executed as script. ArcScript can be written directly in the port.cfg file for the REST Connector.

The following is a simple example of including an ArcScript variable in the URL via the port.cfg file:

Url = "<arc:set attr='myParam' value='myConnectorName' />http://localhost:8001/api.rsc/connectors(ConnectorID='[myParam]')"

A GET request to this URL would return the connector details for myConnectorName via the Admin API provided with Arc. The full scope of ArcScript’s ability to fetch and calculate dynamic values is available using this approach.