Skip to main content

Data Upload & Archiving

Introduction​

The Data Upload & Archiving feature provides a comprehensive solution for exporting Business Central data to Azure Table Storage with advanced synchronisation capabilities. This feature enables organisations to create scheduled data exports for integration purposes, analytics, or data archival, while optionally cleaning up source data after successful upload.

Feature Details​

The Data Upload & Archiving tool allows administrators to configure automated data exports from Business Central tables to Azure Table Storage. The main benefits of this tool include:

  • Flexible Data Export: Export specific fields from any Business Central table to Azure Table Storage
  • Incremental Synchronisation: Support for both full and incremental data synchronisation modes
  • Data Cleanup: Optional deletion of source records after successful upload
  • Integration Ready: JSON-formatted exports suitable for integration with external systems
  • Scheduled Operations: Job queue integration for automated, recurring exports
  • Audit Trail: Comprehensive logging of all export activities and errors

The tool consists of two main components:

  • Azure Storage Table Endpoints: Configuration of Azure Storage connections
  • Data Export Mappings: Definition of what data to export and how
note

You can only use this feature with an Premium license.

Important

When using the "Delete After Upload" option, records are permanently deleted from Business Central after successful export. Ensure you have proper backups and understand the implications before enabling this feature.

Setting Up Azure Storage Table Endpoints​

Before creating data export mappings, you must configure at least one Azure Storage Table Endpoint.

You access Azure Storage Table Endpoints by searching for this term in Business Central.

Configuring an Endpoint​

  1. Code: Unique identifier for the endpoint
  2. Name: Descriptive name for the endpoint
  3. Account URL: Your Azure Storage account URL (e.g., https://mystorageaccount.table.core.windows.net/)
  4. SAS Token: SAS token with appropriate permissions for table operations
  5. Overlap Time (Seconds): Default overlap time for incremental synchronisation (typically 5 seconds)

Creating Data Export Mappings​

You access Data Export Mappings by searching for this term in Business Central.

Setting Up a Data Export Mapping​

  1. Code: Unique identifier for the export mapping
  2. Table No.: Select the Business Central table to export from
  3. Azure Table Endpoint: Choose the configured endpoint
  4. Data Sync Mode:
  • Full Sync: Exports all records matching the filter
  • Incremental Sync: Only exports records modified since the last export
  1. Table Filter: Optional filter to limit which records are exported
  2. Delete After Upload: Enable to delete records after successful export
  3. Batch to Update: Number of records to commit after each deletion batch

Configuring Field Selection​

After creating a mapping, you need to specify which fields to include in the export:

  1. Use the Add Fields action to populate all available fields
  2. Use the Set Include All Fields action to select all fields at once
  3. Manually select individual fields using the Include Field checkbox

Table Filter Configuration​

The Table Filter field lets you specify which records to include in the export. Click the drill-down button to open the filter page builder for the selected table.

Examples of useful filters:

  • Customer table: Exclude blocked customers
  • Item table: Only include active items
  • Ledger entries: Only include open entries

Data Synchronisation Modes​

Full Sync​

  • Exports all records that match the table filter
  • Suitable for initial data loads or when a complete data refresh is needed
  • Each run processes the entire dataset

Incremental Sync​

  • Only exports records modified since the last successful export
  • Uses the system's "SystemModifiedAt" field to identify changed records
  • Includes an overlap time to ensure no records are missed
  • Much more efficient for regular synchronisation

Scheduling and Automation​

Data exports can be automated using Business Central's Job Queue functionality:

  1. Create a Job Queue Entry
  2. Set the Object Type to Codeunit
  3. Set the Object ID to 70254671 (DataExportJobQueue_IMP_TSL)
  4. Enter the Data Export Mapping Code in the Parameter String
  5. Configure the recurrence pattern as needed

Monitoring and Logging​

The system provides comprehensive logging of all export activities:

Data Export Log​

Access the Data Export Log to monitor:

  • Export start and end times
  • Error messages for failed exports
  • Activity tracking for each mapping

Log Entries Include:​

  • Start: Export process initiated
  • End: Export process completed successfully
  • Error: Details of any errors encountered during export

Key Information Tracked:​

  • Data Export Code
  • Table Number
  • Activity Type
  • Error Messages (if applicable)
  • Timestamp of each activity

Best Practices​

Data Export Strategy​

  • Start with Full Sync for initial data load
  • Switch to Incremental Sync for ongoing synchronisation
  • Test filters thoroughly before enabling

Security Considerations​

  • Use SAS tokens with minimal required permissions
  • Regularly rotate access keys
  • Monitor access logs in Azure

Performance Optimisation​

  • Use appropriate table filters to limit data volume
  • Schedule exports during off-peak hours
  • Consider batch sizes for deletion operations

Data Cleanup​

  • Only enable "Delete After Upload" after thorough testing
  • Ensure proper backup procedures are in place
  • Start with small batches to validate the process

Supported Tables for Deletion​

caution

Deleting records after uploading is intended for scenarios where you want to archive data in Azure Storage and then delete the records to free up space in your database. Only enable this feature after consulting with your partner.

The "Delete After Upload" feature can work on all tables that a user would usually be allowed to delete. Some tables (e.g. G/L Entries) cannot be deleted because they contain posted financial history. Some tables can be deleted with the necessary permissions using a specific function (e.g. Posted Sales Invoices, Change Log Entry).

Supported Tables that require Special Permission to delete​

The app supports deletion for the following tables with special permissions:

  • Change Log Entry
  • Sales Shipment Header
  • Sales Shipment Line
  • Sales Invoice Header
  • Sales Invoice Line
  • Sales Credit Memo Header
  • Sales Credit Memo Line
  • Return Shipment Header
  • Return Shipment Line
  • Return Receipt Header
  • Return Receipt Line
  • Purchase Receipt Header
  • Purchase Receipt Line
  • Purchase Invoice Header
  • Purchase Invoice Line
  • Purchase Credit Memo Header
  • Purchase Credit Memo Line

If you try to delete tables not specified in this list, it will only be possible if the user can delete the records from the front end.

Extending Supported Tables​

Business Central partners can extend the list of supported tables by subscribing to the OnAfterAddSupportedTablesForDeletion business event in their extension. The DeleteRecordsUploaded_IMP_TSL codeunit publishes this event.

For Partners: Implementation Example​

[EventSubscriber(ObjectType::Codeunit, Codeunit::"DeleteRecordsUploaded_IMP_TSL", 'OnAfterAddSupportedTablesForDeletion', '', false, false)]
local procedure OnAfterAddSupportedTablesForDeletion(var SupportTableList: Dictionary of [Integer, Text])
begin
 // Add custom tables that are safe for deletion
 SupportTableList.Add(Database::"My Custom Table", 'My Custom Table');
 SupportTableList.Add(Database::"Another Safe Table", 'Another Safe Table');
end;

Implementation Guidelines for Partners​

When extending the supported tables list, partners should:

  1. Evaluate Data Criticality: Only add tables where data deletion will not affect business operations.
  2. Consider Dependencies: Ensure deleted records will not break referential integrity.
  3. Test Thoroughly: Validate the deletion process in a test environment.
  4. Document Changes: Inform clients about which additional tables are supported.
  5. Review Regularly: Reassess supported tables with each extension update.

Safety Considerations​

  • Tables with master data (e.g. G/L Accounts, Chart of Accounts setup) should generally not be added.
  • Posted document tables require careful consideration.
  • Setup and configuration tables should typically be excluded.
  • Consider the impact on reporting and audit trails.

Contact your Business Central partner if you need additional tables added to the supported list. Partners can implement this functionality as part of their extension or as a separate customisation.

Troubleshooting​

Common Issues​

Export Fails to Start

  • Verify Azure Storage endpoint configuration
  • Check SAS token permissions and expiration
  • Ensure the mapping is enabled

No Records Exported

  • Review table filter settings
  • Check if incremental sync has a valid last export timestamp
  • Verify field selection includes required fields

Deletion Errors

  • Confirm the table is supported for deletion
  • Check user permissions for delete operations
  • Review any related table constraints

Error Resolution​

All errors are logged in the Data Export Log with detailed error messages to help identify and resolve issues quickly.