Maintaining Visibility into your Data Automation: Logging, Auditing, and Error Handling in CData Arc

Automated workflow solutions like CData Arc require very little active maintenance as they hum quietly in the background. While convenient, this might also be a cause for concern: will you lose visibility into your data now that you no longer need to perform day-to-day intervention? What if something goes wrong and you're not paying attention?

Sophisticated automation tools address these concerns with a bevy of visibility features to ensure that your data is transparent as it passes through your automation pipeline. Maintaining visibility into your data allows you to generate meaningful reports or audits, respond to failures, and feel confident that you are still in control of your data even after automating away the inconvenience of manual processes.

When Does Visibility Become Important?

Alongside the general desire to stay attentive to your data flow, there are a few specific reasons to require that your automation solution provides a high degree of visibility:

• Generating audits and reports for cross-checking and understanding data processing
• Non-repudiation for your trading partners
• Responding to errors and failures in intelligent ways

This article addresses each of the above challenges and the features within a data movement platform that ensure visibility does not become a problem for users.

Audits and Reports

Audits and reports help ensure that data is being managed and processed correctly. Auditing can have two separate meanings within a data automation platform: auditing the transactions processed by the application (e.g. generating an audit for data sent by a specific trading partner), and auditing changes to the automation platform itself.

Transaction Audits

Transaction audits provide a record of the data processed by automated workflows. These audits can be cross-checked against external records to ensure agreement between parties (e.g. trading partners) regarding the correct processing of critical data sets.

Generating intelligible transaction audits requires the ability to filter out irrelevant transactions. Highlighting specific transactions that are associated together (e.g. from the same trading partner) within a specified time frame helps ensure that data has not been missed or incorrectly processed.

Platform Audits

Audits may also be necessary to review configuration changes to data automation platforms. These audits help address errors and failures that arise after configuration changes were made, either to prepare for configuration rollback or to better understand and debug the issues.

Useful platform audits require the automation platform to have a user scheme that ties configuration changes to specific users.

Non-Repudiation

Non-repudiation refers to the assurance that your trading partners cannot dispute that your organization correctly transferred, received, or otherwise processed their data. Non-repudiation helps protects an organization from potential liability in the rare case where important data goes unaccounted for.

Automation solutions need to be able to provide proof to external parties that their data has been correctly sent or received.

Errors and Alerts

System administrators need the ability to know about and react to failures that occur during data processing. The most common form of error response is in the form of alerts -- any time something goes wrong in an automated data pipeline, an administrator is alerted to this fact via email or text.

More sophisticated error responses help relieve system administrators from the burden of monitoring and responding to alerts, and can be designed to address or eliminate possible emergencies from processing failures. Together, error responses and alerts ensure that there is no danger in automating away the task of day-to-day data management.

How CData Arc Maintains Data Visibility

CData Arc is designed to keep data visible even in the midst of automated pipelines. CData Arc supports intelligent auditing, non-repudiation, and error management via a suite of enterprise features detailed below.

Auditing in CData Arc

CData Arc simplifies the auditing process for both transaction Audits and platform audits. The Transaction Log in CData Arc provides fully searchable transaction records that allow for grouping transactions together in ways that make sense for generating meaningful audits. CData Arc uses the metadata associated with each transaction to guarantee that similar transactions can always be grouped together via simple search terms in the Transaction Log. Additionally, CData Arc supports adding custom metadata to transactions to maintain granular control over how transaction groupings without having to resort to ad-hoc approaches.

CData Arc also provides a convenient Audit Log for easy platform audits. Each individual CData Arc user is associated with the changes they make within the application so that managers and other team members can check behind other users when necessary. Combined with the Roles system that ensures users do not have more access than they need, this helps prevent any difficulty in recovering from configuration problems.

CData Arc's Automatic Non-Repudiation

CData Arc automatically generates and requires receipts (e.g. MDNs) when exchanging data with trading partners. Proof of delivery is required before CData Arc marks a transaction as successful. In other words, CData Arc does not allow the possibility that data was processed but unacknowledged, since this will result in a top-level error that requires a resolution.

Since the proof of delivery is included as part of the transaction process itself, CData Arc's users do not need to take on any further responsibility to guarantee non-repudiation.

Configuring Alerts and Custom Error Responses in CData Arc

CData Arc supports both global alerts and specific custom error responses. Global alerts merely requires specifying the email address for the appropriate system administrators to ensure that data processing failures never go unaddressed.

CData Arc's custom workflows also support intelligent processing of specific errors -- each element in an automated data pipeline can invoke custom workflows when an error occurs during processing. As a result, the full flexibility of CData Arc's workflow canvas is available for system administrators to configure their own error handling logic.

In combination, CData Arc's error handling features ensure that administrators can resolve data processing failures either with personal intervention or with intelligently automated error recovery.

Download a Free Trial of CData Arc and Request a Demo

To get hands-on with CData Arc's data visibility support, download the CData Arc Free Trial or request a personalized demo with one of our product specialists.