September 2023 Release Notes

What's New? 🚀

Features

CDC Streaming Flow Activation - When activating a CDC (Change Data Capture) River, users will navigate through the following:

• Schedule: Configure the schedule for your CDC River.
• Point In Time (Sync Definition): Establish the synchronization of specific data streams by defining your sync parameters.
• Initial Migration: If there are tables with a 'Waiting for Migration' status, this section will guide you through the initial migration process.

Sources Updates

• We've implemented comprehensive improvements to our Oracle, PostgreSQL, and MySQL Sources, bringing enhancements in terms of efficiency and scalability.
  Key highlights include:

  • Improved Efficiency: We have fine-tuned multiple aspects of our Oracle, PostgreSQL, and MySQL databases to optimize query performance for more efficient interactions with your data.

  • Scalability Enhancements: Major adjustments have been made to our Oracle, PostgreSQL, and MySQL configuration to ensure seamless scalability as your data requirements expand.

'SQL Server' is set to receive these identical improvements in the near future.

• SFTP - We have completed a thorough SFTP enhancement process that includes the following significant improvements:
  • Issue Resolution for Client Extracting and Incremental Load Mechanisms: We have diligently addressed various issues reported by our clients related to the extracting and incremental load mechanisms, ensuring smoother and more reliable functionality.
  • Expanded ZIP File Support: we now support ZIP files across all our supported file types, including CSV, JSON, and Excel. This enhancement enables you to seamlessly load ZIP files with ease.

'Amazon S3' and 'FTP' are set to receive these identical improvements in the near future.

• Twitter - We've introduced the Custom Application authentication method to align with the most recent updates in Twitter's pricing plan for the Tweets and Cards entities.

  

Resolved Issues

• ElasticSearch - Now supports incremental fields with special characters, improving data compatibility and accuracy in indexes. This ensures correct handling of incremental updates, reducing errors and maintaining data integrity.

• SAP SuccessFactors- Now features an improved pagination method to prevent duplicated data, ensuring data consistency and reliability during synchronization and reducing downstream inaccuracies and redundancy.

• PostgreSQL - This enhancement includes refinement to the schema and table mapping process, resulting in the accurate presentation of Partitioned tables.

• Logic Python scripts - Python scripts now support importing and using additional libraries, enhancing flexibility and extensibility in data processing workflows.

• Amazon S3 as a Source - Enhanced performance for data loading from buckets containing a substantial number of files.

• From Amazon S3 to Databricks - Improved permission setup between Amazon S3 and Databricks when using IAM Role authentication.

Kits ⭐

The Sharepoint Excel File Load Kits provide a solution that utilizes the Graph API for connecting, fetching an Excel file, and transferring its contents into your data warehouse. This kit encompasses a multi-action workflow for authenticating with the Graph API and fetching the Excel file data, as well as it enables you to conveniently load this data into your data warehouses, including:

• Sharepoint Excel File Load to Snowflake
• Sharepoint Excel File Load to Databricks.