Steps to perform on how to restart Ingest.

1. Turn off Ingest Schedule and wait for load to complete, then stop the BryteFlow Windows service.
2. Restart BryteFlow Windows service and wait for the Ingest dashboard to appear (refresh the page).
3. Go to Schedule tab, then turn on the schedule.
4. BryteFlow Ingest will pick up the correct sequence for processing, but it is good to do a sanity check and ensure that the last sequence number loaded correctly and the new sequence number started correctly, otherwise perform “Rollback” to the correct sequence number for processing.

Steps to perform Rollback in Ingest.

1. Turn off Ingest Schedule and wait for the load to complete.
2. From the Schedule tab:
   1. Click Rollback
   2. In the pop-up screen, select sequence number from where you want to replay the log files. Please note Log files should be available on the source server.
   3. Click Select
   4. Turn on the Schedule.

Steps to perform Full Extract when an Initial Sync needs to be performed for all tables configured in a pipeline.

1. Turn off Ingest Schedule and wait for the load to complete.
2. From the Data tab:
   1. Select the tables (one or more) where you want the Full extract.
   2. Ensure Primary key (PK) column(s) are applied
   3. Select Delta extract (PK or PK Hist).
   4. Click Apply.
3. Goto Schedule tab:
   1. Click Full Extract.
   2. Check logs and ensure that selected tables started full extract.

Steps to perform Redo Initial Extract for existing tables.

1. Turn off Ingest Schedule and wait for the load to complete.
2. From the Data tab:
   1. Select the table(s) where you want to redo the initial extract.
   2. Ensure Primary key (PK) column(s) are applied.
   3. Select Delta extract (PK or PK Hist).
   4. Enable Redo Initial Extract toggle button
   5. Click Apply.
3. Go to Schedule tab:
   1. Click Sync New Tables.
   2. Check logs and ensure the selected table(s) started full extract.

Steps to perform deltas without an Initial Sync of the data or for XL-Ingest.

1. Turn off Ingest Schedule and wait for the load to complete.
2. From the Data tab:
   1. Select the table(s) where you want to perform deltas without an Initial Sync of the data.
   2. Ensure Primary key (PK) column(s) are applied
   3. Select Delta extract (PK or PK Hist).
   4. Enable Skip Initial Extract toggle button
   5. Click Apply.
3. Goto Schedule tab:
   1. Click Full Extract.
   2. Check logs and ensure the selected table(s) skipped full extract.

Steps to perform adding only new tables into existing pipelines.

1. Turn off Ingest Schedule and wait for the load to complete.
2. From the Data tab:
   1. Select new tables (one or more) as per requirement.
   2. Ensure Primary key (PK) column(s) are applied
   3. Select Delta extract (PK or PK Hist).
   4. Click Apply.
3. Go to Schedule tab:
   1. Click Sync New Tables.
   2. Check logs and ensure that existing tables were ignored and new table(s) started to sync.

Steps to perform Sync Struct for existing tables when source structure changes.

1. Turn off Ingest Schedule and wait for the load to complete.
2. Go to Schedule tab:
   1. Click Sync Struct.
   2. Check logs, Ingest will compare the structures and synchronize structures that are different.

Steps to perform on how to stop Ingest.

1. Turn off Ingest Schedule and wait for the load to complete.
2. Open Window services and stop the BryteFlow Windows service.
3. If BryteFlow Windows service have stopped successfully but the Ingest page is still loading  then you might need to kill the non responding processes.
4. To terminate the non responding Ingest process, perform below steps.
   1. Open command prompt as Administrator.
   2. Execute the command – netstat -ano | find “8081”
   3. Replace 8081 with the port number on which Ingest is running.
   4. In the example below, Ingest is running on port 9090. The process id of ingest is 4044 as highlighted in the snapshot.
   5. Open Windows Task Manager, under the Details tab, find the process with PID 4044.
   6. Right click on the process and click the End task.
   7. This will kill the runaway Ingest Java process.

 BryteFlow setup guide details on how to enable supplemental logging for source table(s) for Delta replication. Please refer to the setup guide.

https://docs.bryteflow.com/bryteflow-setup-guide#prerequisite-for-oracle-source

Important note:

• If Database supplemental logging is enabled then only inserts and deletes will come through and updates will be missed.
• If Table level supplemental logging is enabled only updates come through and inserts and deletes will be missed. 

Hence, both Database and Table level supplemental loggings are required.

Steps to create an additional Ingest pipeline.

1. Unzip the provided zip file to a folder. Preferably something like c:\bryteflow\pipeline2\
2. Edit the ingest.xml file in the c:\bryteflow\pipeline2\ folder.
3. Change the  <id>Bryteflow-Ingest</id> and  <name>BryteFlow Ingest</name> xml tags with new pipeline reference names.
4. Check if port 8081 is free and not used by any other processes.
5. Open a command prompt as Administrator, and navigate to the new pipeline folder and execute Install-service.bat. Close the command prompt.
6. Open the Windows services console and start the newly created service.
7. Launch the pipeline on localhost:8081
8. Its recommended to change the port from the ‘Configuration’->’Source Database’-> ‘Web Port’ to any other preferred web port, in order to use 8081 from creating new pipelines.
9. Restrat the windows service and launch the BryteFlow appliccation on the new web port for eg: localhost:8082
10. Make sure to avail a valid license key in order to get started.

Latency is calculated based on time from the record is committed to source till the record is available on the destination.

S3 Delta files contains the incremental data for a particular run from the source database.

The file has 2 types of fields:

1. Constant fields : These are present in all files by default and are meant to provide additional details about the record in file.
2. Variable fields : These varies for each file and aligns with the table structure. The columns on between ‘op_’ and ‘seq_no’ are the table columns as per the table definition at the source.

Different Constant fields and their description:

1. op_       –  This field determines the type of operation for each record in the file. There are different values for this field. Each operation code value is described below
   1. 1. A – The record is extracted as part of FULL INITIAL Load.
      2. I  – The record is an incremental record and the operation is INSERT.
      3. D – The record is an incremental record and the operation is DELETE.
      4. i   – The record is an incremental record and the operation is UPDATE. The values for this record are the latest updated as of the source.
      5. d  – The record is an incremental record and the operation is UPDATE. The values for this record are the previous value(prior to update).

2. seq_no –  This field determines the order of each record. The value is a sequence NUMBER and is incremented by 1 for each record within a file. For incremental files, the value determines the order of DML operations performed on the source.

3. eff_dt – The column is present in the files for which the table Delta replication is set to ‘Primary Key with History‘ . This determines the commit time for the record on the source. Each record will have its commit time as ‘EFF_DT’ in the delta files.

BryteFlow Ingest delivers data to the S3 data lake from relational databases like SAP, Oracle, SQL Server, Postgres, and MySQL in real-time or changed data in batches (as per configuration) using log-based CDC.

The upsert on the S3 data lake is automated and requires no coding nor integration with any third-party applications.

It prepares a manifest file on AmazonS3 with the details of the files loaded onto the data bucket.

Location: [data directory path]/manifest/[table_name_{partitionname}].manifest

File Type: manifest

File format: JSON

File Details : File contains the list of S3 URL’s for all the data files created for each table respectively.

Sample entry for the manifest file:

{
“entries”: [
{“url”:”s3://samplebucket/data/SAMPLETABLE_A/20230615130423/part-r-00000.snappy.parquet”, “meta”: { “content_length”: 26267},”mandatory”:true}
]
}

BryteFlow Ingest delivers data to the S3 data lake from relational databases like SAP, Oracle, SQL Server, Postgres, and MySQL in real-time or changed data in batches (as per configuration) using log-based CDC.

The upsert on the S3 data lake is automated and requires no coding nor integration with any third-party application.

It prepares a .struct file on AmazonS3 with the details of the table structure of the loaded tables.

Location: [data directory path]/manifest/[table_name_{partitionname}].struct

File Type: .struct

File format: JSON

File Details : File contains the structure of the table with column list and its data types.

• manifest : Path and name of manifest file
• srcTable : Table name of the source database
• dstFile    : Filename created at destination
• dstCleanFile : Filename created at destination
• outputFormat: Format of the data file created
• structure  : List of columns and its datatypes in order of the file created.

Sample entry for the STRUCT file:

{
“manifest”: “s3://samplebucket/data/manifest/SAMPLETABLE_A.manifest”,
“srcTable”: “default:SAMPLETABLE_A”,
“dstFile”: “SAMPLETABLE_A”,
“dstCleanFile”: “SAMPLETABLE_A”,
“outputFormat”: “parquet.snappy”,
“structure”: [
{
“name”: “COLUMN1”,
“type”: “VARCHAR”
},
{
“name”: “COLUMN2”,
“type”: “INTEGER”
},
{
“name”: “COLUMN3”,
“type”: “VARCHAR”
},
{
“name”: “COLUMN4”,
“type”: “INTEGER”
}
]
}

Steps for enabling Windows authentication in BryteFlow Ingest: 

1. Stop the BryteFlow Ingest service.

2. Download and Copy the sqljdbc_auth.dll to the bin directory of your java. You can download the DLL from “https://bryteflow.com/release/files/sqljdbc_auth.zip” or you can obtain it from the Microsoft site.

3. Start Ingest service and add the following to the JDBC option in the  Source database connection settings :
integratedSecurity=true

BryteFlow Ingest can be setup to Rollover in a parallel pipeline.

Below are the steps to rollover, Please note this can be implemented using REST API calls of BryteFlow Ingest, details below:

Prerequisites for Bryteflow Ingest:

For kicking off Data Replication from Pipeline 2 :