- 2 Minutes to read
- Print
- DarkLight
- PDF
Databases Overview
- 2 Minutes to read
- Print
- DarkLight
- PDF
Rivery enables migrating data from legacy and cloud databases to one of the DWHs we support.
Our database connectors are built with correctness, safety, and security in mind. The term "correctness" refers to the fact that your destination should always be an exact copy of your database source. Rivery should never cause difficulties in your source database, and clients can use a custom file zone to ensure that data does not leave their environment during transit.
We have simple methods to obtain data from databases without needing access to Binary Logs, in addition to connecting with Binary Log capture tools. We can handle more customers and databases thanks to our customizable approach.
Compatibility of Versions
Database | Modes | Versions Supported | Support for Managing Schema Drift |
ElasticSearch | Standard Extraction | Scroll API Version 7.10 | ![]() |
Indexes API Version 7.10 | |||
Mapping API Version 7.10 | |||
MariaDB | Standard Extraction | All Versions | ![]() |
System Versioning | |||
Custom Query | |||
Microsoft SQL Server | Standard Extraction | All Versions | ![]() |
Change Data Capture | Sp1 2016 and Above (Standard or Enterprise Editions) | ||
Change Tracking | All Versions | ||
MongoDB | Standard Extraction | Versions 3.4 and Above | ![]() |
Change Streams | Versions 4.0 and Above | ||
MySQL | Standard Extraction | All Versions | ![]() |
Change Data Capture | MySQL 5.6 and Above, or MySQL 8.0 and Above | ||
Oracle | Standard Extraction | Versions 11g and Above | ![]() |
Change Data Capture | |||
PostgreSQL | Standard Extraction | All Versions | ![]() |
Write Ahead Logging (WAL) | Versions 10 and Above | ||
Redshift | Standard Extraction | All Versions | ![]() |
SAP S4/Hana | Standard Extraction | Versions 2020 and Above | ![]() |
Teradata | Standard Extraction | Version 15.10 | ![]() |
Vertica | Standard Extraction | All Versions | ![]() |
Supported Features:
Parquet File Conversion
It is possible to convert CSV/JSON files to Parquet files when using Amazon S3 as the Target.
Consult our Quick Guide for an illustration of how to do it.
Please keep in mind that while the Quick Guide uses Amazon S3 as a Source, this is possible with any Database and Google Cloud Storage.
Incremental Behavior
Rivery works in millisecond increments, but users can store data in more granular microsecond or nanosecond increments.
When operating incremental runs based on timestamp, it may appear that the most recent record is missing.
Rivery reads three numbers after the dot when there are six, so '2022-08-14 10:26:52.895132' becomes '2022-08-14 10:26:52.895' and the last record is missing from the current run when fetching the data.
Please note that this record will be retrieved in the next run with updated data.
Soft Delete
A "Soft Delete" is a method of deleting data from a database table while still maintaining a record of the deleted data. This is typically done by adding a "__deleted" flag to the rows of data that are marked for deletion, rather than physically deleting the rows from the table.
Using CDC, this process can be automated and tracked in real-time, allowing for a more efficient and accurate way to manage data deletion and updates. It also allows for the possibility of "undeleting" data if needed, as the deleted rows are still present in the table with the "__deleted" flag set to true.
Here is an example of how you might use the DELETE statement: If you have a Source table, and you use the DELETE statement in your database to mark a row with an ID of 22 as deleted, the outcome of the Target table will be as follows:
Please note:
Soft Delete does not actually delete the record from the table, so it will still take up space in the database and will be included in backups unless you take additional steps to exclude it.
Add a Calculated Column
One of the key features of Rivery is the ability to add calculated columns using expressions. This powerful functionality allows the user to customize their output table beyond the limits of the raw metadata extracted from the Source by including expressions, mathematical or string operations, or simple functions to enrich and tailor the output to their use-case.
For more detailed information, please refer to our Targets document.
Connection
Click the sections below for details on how to connect to our supported databases: