BryteFlow uses a micro services architecture where each source and target data replication pipeline is independent and can be decoupled. The source and destination pair can run on premises or on the cloud or on another server as per the requirements. This prevents a single point of failure, and allows concurrency and independent scaling. One large source does not affect the performance of the other sources. Each Ingest instance comes along with the companion software modules – XL Ingest, TruData and Blend. These borrow the configuration from the Ingest instance. It is recommended that these are located with Ingest, but again, these could be decoupled for performance or security considerations. BryteFlow ControlRoom provides a consolidated interface across all the data pipelines.
The following describes the hardware configuration for a Windows server, assuming that there are a few sources and target combinations (3 medium ideally). It also depends on how intensively the data is being replicated from these sources, so this is a guide, but will need extra resources depending on the amount of data being replicated. The amount of disk space will also be dependent on the amount of data being replicated.
Processor: 4 core
Memory: 16GB
Disk requirements: Depend on the data being extracted, but a minimum of 300GB
Network performance: High
The following softwares are required to be installed on the server:
Setting up BryteFlow suite of products on an On premises windows server can be done in few easy steps. Please ensure there is connectivity to the sources and targets from this server.
Setting up BryteFlow Ingest in few easy steps:
Please follow below steps for installation on LINUX platform.
Creating new pipeline in BryteFlow Ingest:
In order to capture the database changes the software needs access to database logs. To make these available there are certain permissions required for each databases. Depending on the source database please perform the prerequisite steps. For change data capture via timestamps or if only full extracts are required, BryteFlow will support any database that can connect via a jdbc connection string, and these prerequisites are not required.
SQL Server prerequisites depend on the replication option chosen, Change Tracking or Change Data Capture. Prerequisites for each option are described in detail below.
This section applies to MS SQL Server, the version should be higher than 2008.
ALTER DATABASE databasename SET CHANGE_TRACKING = ON (CHANGE_RETENTION = 7 DAYS, AUTO_CLEANUP = ON)
ALTER TABLE tablename ENABLE CHANGE_TRACKING WITH (TRACK_COLUMNS_UPDATED = ON)
3. Grants and Permission
GRANT VIEW CHANGE TRACKING ON schema.tablename TO BryteFlow user;
*** Please Note: If you are configuring BryteFlow Ingest for a completely new SQL Server database, Please make sure to perform at least ‘One Transaction’ on the database to generate the log sequence number for BryteFlow to start with.
This section applies to MS SQL Server, the version should be higher than 2008.
Use <databasename>; EXEC sys.sp_cdc_enable_db;
USE <databasename> GO EXEC sys.sp_cdc_enable_table @source_schema = ‘<schema_name>’, @source_name = ‘<table_name>’, @role_name = null, @supports_net_changes = 0;
To modify the default retention run the below SQL
--Update minutes in the CDC job
EXECUTE
sys.sp_cdc_change_job
@job_type = N
'cleanup'
,
@retention = @New_retention_period_in_minutes;
GO
3. Check if the SQL Server Agent has been started
Use Microsoft configuration Manager to turn on the SQL Server Agent.
The SQL Server Agent should be running.
Supported Oracle versions – Oracle 9i and above.
Execute the following queries on Oracle Server to enable change tracking. For RDS Oracle refer to the user guide here.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; ALTER DATABASE FORCE LOGGING;
ALTER TABLE <schema>.<tablename> ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
exec rdsadmin.rdsadmin_util.alter_supplemental_logging('ADD','ALL');
exec rdsadmin.rdsadmin_util.set_configuration('archivelog retention hours',24);
ALTER TABLE <schema>.<tablename> ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
Grants and Permission required for Oracle database as a source connector
CREATE SESSION
SELECT access on V_$LOGMNR_CONTENTS
SELECT access on V_$LOGMNR_LOGS
SELECT access on ANY TRANSACTION
SELECT access on DBA_OBJECTS
EXECUTE access on DBMS_LOGMNR
GRANT SELECT ON V_$ARCHIVED_LOG TO <user>;
GRANT SELECT ON V_$LOGMNR_CONTENTS TO <user>;
GRANT EXECUTE ON DBMS_LOGMNR TO <user>;
GRANT SELECT ON V_$LOGMNR_LOGS TO <user>;
GRANT SELECT ANY TRANSACTION TO <user>;
GRANT SELECT ON DBA_OBJECTS TO <user>;
GRANT execute_catalog_role
TO <user>;
GRANT LOGMINING to <user>; -- This is required for Oracle 12c and higher versions
**For Oracle 10g only
CREATE PUBLIC SYNONYM DBMS_LOGMNR FOR SYS.DBMS_LOGMNR;
For Oracle 19c Continuous Logmining, Please provide additional grant as below:
GRANT SELECT ON V_$DATABASE TO <user>;
GRANT SELECT ON V_$Log TO <user>;
GRANT SELECT ON V_$Logfile TO <user>;
To verify if Oracle is setup correctly for change detection execute the following queries.
Condition to be checked | SQL to be executed | Result expected |
---|---|---|
Is ArchiveLog mode enabled? |
SELECT log_mode FROM V$DATABASE; |
ARCHIVELOG |
Is Supplemental logging turned on at database level? |
SELECT supplemental_log_data_min FROM V$DATABASE; |
YES |
Is Supplemental Logging turned on at table level? |
SELECT log_group_name, table_name, always, log_group_type FROM dba_log_groups; |
RESULT <log group name>, <table name>, ALWAYS, ALL COLUMN LOGGING |
To enable binary logging, the following parameters need to be configured as below in my.ini file on MySQL on Windows or in my.cnf file on MySQL on UNIX:
Parameter | Value |
---|---|
server_id | Any value from 1. E.g. server_id = 1 |
log_bin=<path> | Path to the binary log file. E.g. log_bin = D:\MySQLLogs\BinLog |
binlog_format | binlog_format=row |
expire_logs_days | To avoid disk space issues it is strongly recommended not to use the default value (0). E.g. expire_log_days = 4 |
binlog_checksum | This parameter can be set to binlog_checksum=none. BryteFlow does support CRC32 as well |
binlog_row_image | binlog_row_image=full |
2. Grants & Permissions
The Ingest user id must have the following privileges:
CREATE USER ‘bflow_ingest_user’ IDENTIFIED BY ‘*****’;
GRANT SELECT, REPLICATION CLIENT, SHOW DATABASES ON *.* TO bflow_ingest_user IDENTIFIED BY '******';
GRANT SELECT, REPLICATION slave, SHOW DATABASES ON *.* TO bflow_ingest_user IDENTIFIED BY '*****';
To enable change tracking MySQL on Amazon RDS perform the following steps.
binlog_format: binlog_format=row binlog_checksum : binlog_checksum=none OR CRC32.
CREATE USER 'bflow_ingest_user' IDENTIFIED BY '*****';
GRANT SELECT, REPLICATION CLIENT, SHOW DATABASES ON *.* TO bflow_ingest_user;
GRANT SELECT, REPLICATION slave, SHOW DATABASES ON *.* TO bflow_ingest_user; P.S. If the source DB type is Amazon RDS MySQL DB, please download 'mysqlbinlog.exe' and add its directory path in Windows 'Environment variable' 'PATH' on the Client machine(BryteFlow Server)
binlog_format: binlog_format=row
binlog_checksum : binlog_checksum=none OR CRC32.
3. Please have an User account for BryteFlow that has the following privileges:
REPLICATION CLIENT
REPLICATION SLAVE
SUPER – (Required only in MySQL versions before 5.6.6)
The user must also have SELECT privileges for the source tables selected for replication.
Note:- After changing these parameters, a restart is needed for PostgreSQL
2. Grant Replication role to BryteFlow User
3. Grant ‘Read-only’ access for all the replicating tables to BryteFlow user.
On Salesforce Change Data Capture is turned on by default, please do not turn it off.
You would need to generate a security token to be used with Bryteflow Ingest.
A security token is a case-sensitive alphanumeric key that appended to your Salesforce password.
eg. Your Salesforce password to be used with Ingest will be “<your Salesforce password ><security_token>”
A token can be generated by following these steps:
1. Log in to your salesforce account and go to My Setting > Personal > Reset my security token.
2. Click on Reset Security Token button. The token will be emailed to the email account associated with your salesforce account.
For getting all the required permissions for data replication please refer to the Salesforce Documentation.
SAP S/4HANA System:
For ODP Based Extraction or CDS view Based Extraction below components are needed:
Pre-Requisites for SAP HANA (Change tracking) :
1.Create a user account for BryteFlow.
CREATE USER <USERNAME> PASSWORD <PASSWORD>;
2.BryteFlow replication user should have ‘select‘ access on the tables to be replicated.
BryteFlow replication user should have access to ‘create triggers’ to be replicated
Grant below privileges to BryteFlow user created above.
GRANT SELECT, TRIGGER ON SCHEMA <YOURSCHEMA> TO <USERNAME>;
3. BryteFlow replication user should have access to a schema where it can create a table on the source database.
This is used to store transactions for restart and recoverability
Grant below privileges to BryteFlow user created above.
GRANT CREATE ANY ON SCHEMA <YOURSCHEMA> TO <USERNAME>;
4. Provide ABAP_READ role to the BryteFlow replication user to provide access to the application layer objects
SAP ECC 6.0 System:
For ODP Based Extraction or CDS view Based Extraction below components are needed for ECC6.0:
For ODP Based Extraction on SAP BW System, below components are needed:
2. In transaction SPRO check System Alias. At a minimum, the LOCAL System alias is required.
3. Check that SAP Gateway is activated.
4. Check for ODP Release check mark for ODP extractors in transaction RSA6.
BryteFlow XL Ingest is a companion software to BryteFlow Ingest. It does the initial loads for huge volume tables.
The setup is easy as it reads through the Ingest config for source and destination databases.
Follow below steps to do the install:
BryteFlow Trudata is a companion software to BryteFlow Ingest. It reconciles the source and destination tables in near real-time.
The setup is easy as it reads through the Ingest config for source and destination database details.
Follow below steps to do the install:
BryteFlow Blend is a companion software to BryteFlow Ingest. Its used to perform data transformations of AWS S3.
The setup is easy as it reads through the Ingest config for AWS details.
Follow below steps to do the install:
BryteFlow Suite of products when setup in Non-AMI environment comes with a validity which is specific to each customers agreement. Make sure you get the valid license keys in order to begin with software usages. Each software comes with its own product id when setup individually.
NOTE: Licensing is not applicable when sourced from the AWS Marketplace.
Here is how to apply license keys for each:
To get a valid license go to Configuration tab, then to the License tab and email the “Product ID” to the BryteFlow support team – support@bryteflow.com
To get a valid license go to Configuration tab and email the “Product ID” to the BryteFlow support team – support@bryteflow.com
To get a valid license go to Configuration tab and email the “Product ID” to the BryteFlow support team – support@bryteflow.com
To get a valid license go to ‘Setup‘ tab then select ‘AWS Location‘ in the left menu. Please email the “Product ID” to the BryteFlow support team – support@bryteflow.com
Setting up BryteFlow Control Room on an On premises windows server can be done in few easy steps.