User Guide

Overview

BryteFlow uses a micro services architecture where each source and target data replication pipeline is independent and can be decoupled. The source and destination pair can run on premises or on the cloud or on another server as per the requirements. This prevents a single point of failure, and allows concurrency and independent scaling. One large source does not affect the performance of the other sources. Each Ingest instance comes along with the companion software modules – XL Ingest, TruData and Blend. These borrow the configuration from the Ingest instance. It is recommended that these are located with Ingest, but again, these could be decoupled for performance or security considerations. ControlRoom provides a consolidated interface across all the data pipelines.

Prerequisites for software on the server

The following softwares are required to be installed on the server:

  1. The server should have 64-bit Java 8 or above.
  2. Google Chrome
  3. For SQL Server sources,  Install bcp – Microsoft Utility and below drivers:

Setup BryteFlow Suite

Setting up BryteFlow suite of products on an On premises windows server can be done in few easy steps. Please ensure there is connectivity to the sources and targets from this server.

  1. Download the software from the link provided on the server where the software needs to be installed.
  2. Unzip it to the installation directory of your choice .
  3. This will have entire BryteFlow suite under the main directory named ‘bryteflow’.
  4. If you are planning to setup more than one BryteFlow instances, its good to rename the main install directory ‘bryteflow’ to ‘bryteflow-<source1>’ to avoid any conflict.

 

Setup BryteFlow Ingest

Setting up BryteFlow Ingest in few easy steps:

  1. Once the software is downloaded and unzipped.
  2. Go to ‘ingest’ directory under the main directory ‘bryteflow’.
  3. In order to install the service open windows command prompt with ‘Run as Administrator’
  4. Change the directory to ‘ingest’
  5. Execute  “Install-BryteFlow Ingest Service.bat”
  6. ‘BryteFlow Ingest’ service is now installed on the server.
  7. Go to Windows ‘services’ and start the service for Ingest.
  8. Launch the user interface from google chrome browser with the URL: localhost:8081 
  9. Proceed to complete Ingest setup by configuring the Source db and Destination db.

Prerequisites for Source Databases

In order to capture the database changes the software needs access to database logs. To make these available there are certain permissions required for each databases. Depending on the source database please perform the prerequisite steps,

Prerequisite for MS SQL Server source

1. bcp – Microsoft Utility has been installed for MS SQL Server as source
As per prerequisites please check that bcp has been installed on the server by opening a command window and typing “bcp -v” . If not please refer to the section above.
 2. Enable Change Tracking for a database in MS SQL Server

This section applies to MS SQL Server, the version should be higher than 2008.

  • To enable Change Tracking at the database level execute the following query:
    ALTER DATABASE databasename
    SET CHANGE_TRACKING = ON
    (CHANGE_RETENTION = 7 DAYS, AUTO_CLEANUP = ON)
  • Enable Change Tracking at the table level as below execute the following query:
    ALTER TABLE tablename
    ENABLE CHANGE_TRACKING
    WITH (TRACK_COLUMNS_UPDATED = ON)
  • Enable Change Tracking at the database and table level for all the databases and tables to be replicated.
  • To enable view permission to event viewer logs execute the following query:

3. Grants and Permission

  • GRANT VIEW SERVER STATE TO “AccountName”
  • The BryteFlow Ingest database replication login  user should have VIEW CHANGE TRACKING permission to view the Change Tracking information.
    GRANT VIEW CHANGE TRACKING ON schema.tablename TO BryteFlow user;

*** Please Note:  If you are configuring BryteFlow Ingest for a completely new SQL Server database, Please make sure to perform at least ‘One Transaction’ on the database to generate the log sequence number for BryteFlow to start with.

Prerequisite for Oracle source

 

1. Enable Change Tracking for an On-Premises Oracle Server

Execute the following queries on Oracle Server to enable change tracking.

  • Oracle database should be in ARCHIVELOG mode.
  • The supplemental logging has to be turned on at the database level. Supplemental logging is required so that additional details are logged in the archive logs.
    To turn on supplemental logging at the database level, execute the following statements:

    ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
  • Alternatively to turn on minimal database supplemental logging execute the following statements:
    ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; 
    ALTER DATABASE FORCE LOGGING;
  • In Oracle, ensure that supplemental logging is turned on at the table level. To turn on supplemental logging at the table level, execute the following statement:
    ALTER TABLE <schema>.<tablename> ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
  • SELECT access on all tables to be replicated
  • SELECT access on V$ARCHIVED_LOG
  • The following security permissions should be assigned to the userCREATE SESSION
    SELECT access on V_$LOGMNR_CONTENTS
    SELECT access on V_$LOGMNR_LOGS
    SELECT access on ANY TRANSACTION
    SELECT access on DBA_OBJECTS
    EXECUTE access on DBMS_LOGMNRRun the following grant statements for <user> for the above requirements:

    GRANT SELECT ON V_$ARCHIVED_LOG TO <user>;
    GRANT SELECT ON V_$LOGMNR_CONTENTS TO <user>;
    GRANT EXECUTE ON DBMS_LOGMNR TO <user>;
    GRANT SELECT ON V_$LOGMNR_LOGS TO <user>;
    GRANT SELECT ANY TRANSACTION TO <user>;
    GRANT SELECT ON DBA_OBJECTS TO <user>;

Preparing On-premises MySQL

To prepare MySQL for change tracking perform the following steps.

  1. Enable Change Tracking

To enable binary logging, the following parameters need to be configured as below in my.ini file on MySQL on Windows or in my.cnf file on MySQL on UNIX:

Parameter Value
server_id Any value from 1.
E.g. server_id = 1
 log_bin=<path> Path to the binary log file.
E.g. log_bin = D:\MySQLLogs\BinLog
binlog_format binlog_format=row
expire_logs_days To avoid disk space issues it is strongly recommended not to use the default value (0).
E.g. expire_log_days = 4
 binlog_checksum This parameter can be set to binlog_checksum=none.
BryteFlow does support CRC32 as well
binlog_row_image binlog_row_image=full

Preparing Salesforce account for BryteFlow Ingest

On Salesforce Change Data Capture is turned on by default, please do not turn it off.

You would need to generate a security token to be used with Bryteflow Ingest.
A security token is a case-sensitive alphanumeric key that appended to your Salesforce password.

eg. Your Salesforce password to be used with Ingest will be “<your Salesforce password ><security_token>”

A token can be generated by following these steps:
1. Log in to your salesforce account and go to My Setting > Personal > Reset my security token.
2. Click on Reset Security Token button. The token will be emailed to the email account associated with your salesforce account.

Preparing for BryteFlow Trigger solution

For the databases such as DB2, Firebird or for any RDBMS where there are no access to archive logs to get the change data, BryteFlow has the trigger option to get the change data.

For this solution there are certain prerequisites which needs to be implemented:

  1. BryteFlow replication user should have ‘select‘ access on the tables to be replicated
  2. BryteFlow replication user should have access to ‘create triggers’ to be replicated
  3. BryteFlow replication user should have access to ‘create tables’ on the source database. The solution requires tables to be created on the source by BryteFlow, so please create a schema where these tables can reside, so that it can be configured in the software.

Please provide relevant grants to BryteFlow replication user in order to proceed with the Trigger Solution.

 

Setup BryteFlow XL Ingest

BryteFlow XL Ingest is a companion software to BryteFlow Ingest. It does the initial loads for huge volume tables.

The setup is easy as it reads through the Ingest config for source and destination databases.

Follow below steps to do the install:

  1. Once BryteFlow Ingest is setup and running.
  2. Go to the main directory ‘bryteflow’->’xlingest’
  3. In order to install the service open windows command prompt with ‘Run as Administrator’
  4. Change the path to ‘xlingest’ directory
  5. Execute  “Install-BryteFlow XL Ingest Service.bat”
  6. ‘BryteFlow XL Ingest’ service is now installed on the server.
  7. Go to Windows ‘services’ and start the service for Ingest.
  8. Launch the user interface from google chrome browser with the URL: localhost:8084

Setup BryteFlow Trudata

BryteFlow Trudata is a companion software to BryteFlow Ingest. It reconciles the source and destination tables in near real-time.

The setup is easy as it reads through the Ingest config for source and destination database details. 

Follow below steps to do the install:

  1. Once BryteFlow Ingest is setup and running.
  2. Go to the main directory ‘bryteflow’->’trudata’
  3. In order to install the service open windows command prompt with ‘Run as Administrator’
  4. Change the path to ‘trudata’ path
  5. Execute  “Install-BryteFlow Trudata Service.bat”
  6. ‘BryteFlow Trudata’ service is now installed on the server.
  7. Go to Windows ‘services’ and start the service for Ingest.
  8. Launch the user interface from google chrome browser with the URL: localhost:8085

Setup BryteFlow Blend

BryteFlow Blend is a companion software to BryteFlow Ingest. Its used to perform data transformations of AWS S3.

The setup is easy as it reads through the Ingest config for AWS details. 

Follow below steps to do the install:

  1. Once BryteFlow Ingest is setup and running.
  2. Go to the main directory ‘bryteflow’->’blend’
  3. In order to install the service open windows command prompt with ‘Run as Administrator’
  4. Change the path to ‘blend’ path
  5. Execute  “Install-BryteFlow Blend Service.bat”
  6. ‘BryteFlow Blend’ service is now installed on the server.
  7. Go to Windows ‘services’ and start the service for Ingest.
  8. Launch the user interface from google chrome browser with the URL: localhost:8082

Setup BryteFlow Control Room

Setting up BryteFlow Control Room on an On premises windows server can be done in few easy steps.

  1. Download the software from the link provided on the server where the software needs to be installed.
  2. Unzip it to the installation directory preferably at the same level of ‘bryteflow’
  3. In order to install the service open windows command prompt with ‘Run as Administrator’
  4. Change the directory to the path of ‘controlroom’
  5. Execute ‘Install-BryteFlow controlroom Service.bat’
  6. ‘BryteFlow Controlroom’ service is now installed on the server.
  7. Go to Windows ‘services’ and start the service for Ingest.
  8. Launch the user interface from google chrome browser with the URL: localhost:8091

 

Suggest Edit