User Guide
  1. Get Ingest Instance Summary
  2. Configure Source Database
  3. Configure Destination Database
  4. Configure Credentials
  5. Set Scheduler switch
  6. Configure Table Settings
  7. Skip Initial Extract
  8. Redo Initial Extract
  9. Schedule Full Extract
  10. Manual Delta
  11. Schedule Sync New Tables
  12. Schedule Sync Struct
  13. Save scheduler frequency settings
  14. Get loaded records count
  15. Get the list of Selected Tables
  16. Select, Deselect a table for extraction
  17. Configure Recovery
  18. Get Recovery Status
  19. Get licence information for Ingest
  20. Save new Licence key for Ingest
  21. Provide data for recovery purposes
  22. BryteFlow Enterprise Manager API's
    1. Enterprise Manager - Generate Authorization Bearer Token
    2. Enterprise Manager - Get list of available templates
    3. Enterprise Manager - Create a new Instance
    4. Enterprise Manager - Start or Stop an instance
    5. Enterprise Manager - Delete an Instance
    6. Enterprise Manager - List all Instances

Get Ingest Instance Summary

Get complete summary of the current Ingest instance:

URL: http://<host>:<port>/ingest/api/ingest?cmd=rstat&func=get&_=<current timestamp in ms>

Method: GET

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved or fetched.
    • Values – To save connection settings, value is ‘conn’, to save configuration settings, value is ‘cop’, for other purposes it is ‘rstat’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=rstat&func=get&_=1631083520032
  • func –
    • Description – Describes whether the API call is to turn the scheduler on or off, or to get the ingest summary.
    • Values – on, off, get
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=rstat&func=get&_=1631083520032

Response: The response of the above API is a text response as shown below:

  • Sample response –                                                                                                                                                                                                                    Y
    SRC|I|Full Extract requested
    DST|I|Source configuration error
    Status|Licence invalid
    StepList|Get a valid licence;Configure Source Database;Configure Destination Database;Configure File System;
    Dates|2021-09-13 18:29:26|2021-09-13 23:59:31|2021-09-13 23:59:34|2021-09-13 18:29:26|2022-01-25                                                        17:43:17|2021-09-13 18:29:31|
  • Explanation:
    • The first line indicates the status of the scheduler. ‘Y’ implies that the scheduler is on and ‘N’ implies that the scheduler is off.
    • The second line provides the status of the source connection. ‘I’ means that the source is in idle state and no activity is ongoing, ‘R’ implies a running state meaning some activity is in process and ‘E’ means that there is some error on the source side. The actual status message is also available following ‘I’, ‘R’ or ‘E’ (separated by a | character).
    • The third line provides the status of the destination connection. ‘I’ means that the destination is in idle state and no activity is ongoing, ‘R’ implies a running state meaning some activity is in process and ‘E’ means that there is some error on the destination side. The actual status message is also available following ‘I’, ‘R’ or ‘E’ (separated by a | character).
    • The fourth line provides the license status of the Ingest instance.
    • The fifth line talks about the next series of steps to be undertaken by the user.

Configure Source Database

Configure Source Databases in BryteFlow Ingest using below API :

BryteFlow’s API allows you to do the source database configuration.

URL: http://<host>:<port>/ingest/api/ingest?cmd=conn&conn=s

Method: POST

Request Body:

func=save&src-db=<database name>&src-host=<database host>&src-options=&src-port=<database port>&src-pwd=<database password>&src-pwd2=<database password>&src-type=rds.oracle11lm&src-uid=<database user id>&type=src

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved.
    • Values – To save connection settings, value is ‘conn’, to save configuration settings, value is ‘cop’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=conn&conn=s
  • conn –
    • Description – Short for connection. It is used to describe the kind of connection settings being saved.
    • Values – Value is ‘s’ for source database settings, ‘d’ for destination database settings, ‘f’ for credentials, ‘m’ for email settings.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=conn&conn=s

Payload Params:

  • func –
    • Description – Describes whether the API call is just to save the settings or save and test the connection.
    • Values – test, save
    • Sample –
    • Example – func=save&src-db=abc&src-host=localhost&src-options=&src-port=1521&src-pwd=def&src-pwd2=def&src-type=rds.mscdc&src-uid=userId&type=src
  • src-db –
    • Description – It is the database name.
    • Values – The name of the database to connect to.
    • Sample –
    • Example – func=save&src-db=abc&src-host=localhost&src-options=&src-port=1521&src-pwd=def&src-pwd2=def&src-type=rds.mscdc&src-uid=userId&type=src
  • src-host –
    • Description – It provides the hostname or url of the database server.
    • Values – The hostname or url of the database server to connect to.
    • Sample –
    • Example – func=save&src-db=abc&src-host=localhost&src-options=&src-port=1521&src-pwd=def&src-pwd2=def&src-type=rds.mscdc&src-uid=userId&type=src
  • src-options –
  • src-port –
    • Description – This is the port on which the database server is running.
    • Values – The port of the database server to connect to.
    • Sample –
    • Example – func=save&src-db=abc&src-host=localhost&src-options=&src-port=1521&src-pwd=def&src-pwd2=def&src-type=rds.mscdc&src-uid=userId&type=src
  • src-pwd –
    • Description – This is the password of the database.
    • Values – The password of the database to connect to.
    • Sample –
    • Example – func=save&src-db=abc&src-host=localhost&src-options=&src-port=1521&src-pwd=def&src-pwd2=def&src-type=rds.mscdc&src-uid=userId&type=src
  • src-pwd2 –
    • Description – This again is the password of the database.
    • Values – This field has the same value as the src-pwd field.
    • Sample –
    • Example – func=save&src-db=abc&src-host=localhost&src-options=&src-port=1521&src-pwd=def&src-pwd2=def&src-type=rds.mscdc&src-uid=userId&type=src
  • src-type –
    • Description – This field describes the type of source database from which extraction is to be done.
    • Values – It can take one out of several values available for the type of source database to connect to such as rds.mscdc, rds.oracle11lm, rds.oracle11lmc, rds.msfull, rds.maria, rds.hanafe, rds.jdbc, etc.
    • Sample –
    • Example – func=save&src-db=abc&src-host=localhost&src-options=&src-port=1521&src-pwd=def&src-pwd2=def&src-type=rds.mscdc&src-uid=userId&type=src
  • src-uid –
    • Description – This is the user-id part of the credentials to connect to the source database.
    • Values – Mention the valid user-id to connect to the database.
    • Sample –
    • Example – func=save&src-db=abc&src-host=localhost&src-options=&src-port=1521&src-pwd=def&src-pwd2=def&src-type=rds.mscdc&src-uid=userId&type=src
  • type –
    • Description – Describes the type of connection settings being saved.
    • Values – src, dst, file, mail
    • Sample –
    • Example – func=save&src-db=abc&src-host=localhost&src-options=&src-port=1521&src-pwd=def&src-pwd2=def&src-type=rds.mscdc&src-uid=userId&type=src

Configure Destination Database

Configure Destination Databases using below API :

URL : http://host:port/ingest/api/ingest?cmd=conn&conn=d

Method: POST

Request Body:

func=save&dst-bucket=<S3 bucket>&dst-db=<database name>&dst-dir=<S3 work directory>&dst-host=<Redshift host>&dst-iam_role=<IAM Role>&dst-options=&dst-port=<Redshift port>&dst-pwd=<Redshift password>&dst-pwd2=<Redshift password>&dst-type=rds.redmulti&dst-uid=<Redshift user id>&type=dst

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved.
    • Values – To save connection settings, value is ‘conn’, to save configuration settings, value is ‘cop’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=conn&conn=d
  • conn –
    • Description – Short for connection. It is used to describe the kind of connection settings being saved.
    • Values – Value is ‘s’ for source database settings, ‘d’ for destination database settings, ‘f’ for credentials, ‘m’ for email settings.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=conn&conn=d

Payload Params:

  • func –
    • Description – Describes whether the API call is just to save the settings or save and test the connection.
    • Values – test, save
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • dst-bucket –
    • Description – It is the destination S3 bucket name.
    • Values – The name of the S3 bucket in which the data is to be loaded.
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • dst-db –
    • Description – It is the destination database name.
    • Values – The name of the destination database to load the data on.
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • dst-options –
  • dst-dir –
    • Description – The name of the directory in the S3 bucket mentioned above.
    • Values – The directory name.
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • dst-host –
    • Description – It provides the hostname or url of the destination database server.
    • Values – The hostname or url of the destination database server.
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • dst-iam_role –
    • Description – This field mentions the IAM role.
    • Values – IAM role.
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • dst-port –
    • Description – This is the port on which the destination database server is running.
    • Values – The port of the destination database server.
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • dst-pwd –
    • Description – This is the password of the destination database.
    • Values – The password of the destination database.
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • dst-pwd2 –
    • Description – This again is the password of the destination database.
    • Values – This field has the same value as the dst-pwd field.
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • dst-type –
    • Description – This field describes the type of destination where the data is to be loaded.
    • Values – It can take one out of several values available for the type of destination such as rds.redmulti.
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • dst-uid –
    • Description – This is the user-id part of the credentials to connect to the destination database.
    • Values – Mention the valid user-id to connect to the destination database.
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst
  • type –
    • Description – Describes the type of connection settings being saved.
    • Values – src, dst, file, mail
    • Sample –
    • Example – func=save&dst-bucket=testBucket&dst-db=abcDb&dst-dir=s3WorkDirectory&dst-host=abc.def&dst-iam_role=IAM_Role&dst-options=&dst-port=9999&dst-pwd=password&dst-pwd2=password&dst-type=rds.redmulti&dst-uid=userId&type=dst

Configure Credentials

Configure AWS login credentials.

URL: http://<host>:<port>/ingest/api/ingest?cmd=conn&conn=f

Method: POST

Request body: 

file-kms=<KMS id>&file-reg=<AWS region>&file-s3key=<Access Key Id>&file-s3skey=<Secrect Access Key>&file-type=<aws.cred|aws.iam>&func=save&type=file

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved.
    • Values – To save connection settings, value is ‘conn’, to save configuration settings, value is ‘cop’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=conn&conn=f
  • conn –
    • Description – Short for connection. It is used to describe the kind of connection settings being saved.
    • Values – Value is ‘s’ for source database settings, ‘d’ for destination database settings, ‘f’ for credentials, ‘m’ for email settings.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=conn&conn=f

Payload Params:

  • func –
    • Description – Describes whether the API call is just to save the settings or save and test the connection.
    • Values – test, save
    • Sample –
    • Example – func=save&file-kms=abc&file-reg=us-west-2&file-s3key=def&file-s3skey=ghi&file-type=aws.cred&type=file
  • file-kms –
    • Description – It is the AWS Key Management Service Id.
    • Values – The name of the AWS Key Management Service Id.
    • Sample –
    • Example – func=save&file-kms=abc&file-reg=us-west-2&file-s3key=def&file-s3skey=ghi&file-type=aws.cred&type=file
  • file-reg –
    • Description – It is the AWS region.
    • Values – The AWS region being used for the service.
    • Sample –
    • Example – func=save&file-kms=abc&file-reg=us-west-2&file-s3key=def&file-s3skey=ghi&file-type=aws.cred&type=file
  • file-s3key –
    • Description – It is the Access Key Id for the AWS user.
    • Values – The Access Key Id of the AWS user.
    • Sample –
    • Example – func=save&file-kms=abc&file-reg=us-west-2&file-s3key=def&file-s3skey=ghi&file-type=aws.cred&type=file
  • file-s3skey –
    • Description – It is the Secret Access Key for the AWS user.
    • Values – The Secret Access Key of the AWS user.
    • Sample –
    • Example – func=save&file-kms=abc&file-reg=us-west-2&file-s3key=def&file-s3skey=ghi&file-type=aws.cred&type=file
  • file-type –
    • Description – This describes the type of AWS login info to use to connect to AWS..
    • Values – Values are AWS credentials and AWS IAM Access.
    • Sample –
    • Example – func=save&file-kms=abc&file-reg=us-west-2&file-s3key=def&file-s3skey=ghi&file-type=aws.cred&type=file
  • type –
    • Description – Describes the type of connection settings being saved.
    • Values – src, dst, file, mail
    • Sample –
    • Example – func=save&file-kms=abc&file-reg=us-west-2&file-s3key=def&file-s3skey=ghi&file-type=aws.cred&type=file

Turn scheduler on or off.

Change the Extract scheduler to on or off.

URL: http://<host>:<port>/ingest/api/ingest?cmd=rstat&func=<on|off>&_=<current timestamp in ms>

Method: GET

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved.
    • Values – To save connection settings, value is ‘conn’, to save configuration settings, value is ‘cop’, for other purposes it is ‘rstat’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=rstat&func=on&_=1631083520032
  • func – 
    • Description – Describes whether the API call is to turn the scheduler on or off.
    • Values – on, off
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=rstat&func=on&_=1631083520032

Save table settings

Save extraction settings for a table.

URL: http://<host>:<port>/ingest/api/ingest?cmd=dbwc

Method: POST

Request body:

func=save&key=<table name>&tab-mask=<masked column>&tab-pfold=<partitioning folder>&tab-pkey=<primary key column>&tab-where=<where clause>&tab-xfer=<DH|DP>&type=tab

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved.
    • Values – conn, cop, rstat, dbwc.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=dbwc

Payload Params:

  • func –
    • Description – Describes whether the API call is just to save the settings or save and test the connection.
    • Values – test, save
    • Sample –
    • Example – func=save&key=abc&tab-mask=mn&tab-pfold=p&tab-pkey=idd&tab-where=pqr&tab-xfer=DH&type=tab
  • key –
    • Description – Name of the table.
    • Values – Table name.
    • Sample –
    • Example – func=save&key=abc&tab-mask=mn&tab-pfold=p&tab-pkey=idd&tab-where=pqr&tab-xfer=DH&type=tab
  • tab-mask –
    • Description – Id of the column to be masked.
    • Values – Id of the column to be masked.
    • Sample –
    • Example – func=save&key=abc&tab-mask=mn&tab-pfold=p&tab-pkey=idd&tab-where=pqr&tab-xfer=DH&type=tab
  • tab-pfold –
    • Description – Describes the Partition folder.
    • Values – Name of the partition folder.
    • Sample –
    • Example – func=save&key=abc&tab-mask=mn&tab-pfold=p&tab-pkey=idd&tab-where=pqr&tab-xfer=DH&type=tab
  • tab-pkey –
    • Description – Id of the column to be set as primary key.
    • Values – Id of the column to be set as primary key.
    • Sample –
    • Example – func=save&key=abc&tab-mask=mn&tab-pfold=p&tab-pkey=idd&tab-where=pqr&tab-xfer=DH&type=tab
  • tab-where –
    • Description – In this field, the where clause can be specified.
    • Values – A ‘Where clause’ string.
    • Sample –
    • Example – func=save&key=abc&tab-mask=mn&tab-pfold=p&tab-pkey=idd&tab-where=pqr&tab-xfer=DH&type=tab
  • tab-xfer –
    • Description – Specifies the delta extract type, primary key or primary key with history.
    • Values – DH or DP.
    • Sample –
    • Example – func=save&key=abc&tab-mask=mn&tab-pfold=p&tab-pkey=idd&tab-where=pqr&tab-xfer=DH&type=tab
  • type –
    • Description – Describes the type of settings being saved.
    • Values – tab
    • Sample –
    • Example – func=save&key=abc&tab-mask=mn&tab-pfold=p&tab-pkey=idd&tab-where=pqr&tab-xfer=DH&type=tab

Skip Initial Extract

API to set Skip Initial Extract status to on or off.

URL: http://<host>:<port>/ingest/api/ingest?cmd=dbwc

Method: POST

Request body:

func=save&key=<table name>&tab-skip=<Y|N>&type=tab

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved.
    • Values – conn, cop, rstat, dbwc.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=dbwc

Payload Params:

  • func –
    • Description – Describes whether the API call is just to save the settings or save and test the connection.
    • Values – test, save
    • Sample –
    • Example – func=save&key=abc&tab-skip=Y&type=tab
  • key –
    • Description – Name of the table.
    • Values – Table name.
    • Sample –
    • Example – func=save&key=abc&tab-skip=Y&type=tab
  • tab-skip –
    • Description – Specifies whether the initial extact is to be skipped or not.
    • Values – Y, N.
    • Sample –
    • Example – func=save&key=abc&tab-skip=Y&type=tab
  • type –
    • Description – Describes the type of settings being saved.
    • Values – tab.
    • Sample –
    • Example – func=save&key=abc&tab-skip=Y&type=tab

Redo Initial Extract

API to set Redo Initial Extract status to on or off.

URL: http://<host>:<port>/ingest/api/ingest?cmd=dbwc

Method: POST

Request body:

func=save&key=<table name>&tab-redo=<Y|N>&type=tab

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved.
    • Values – conn, cop, rstat, dbwc.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=dbwc

Payload Params:

  • func –
    • Description – Describes whether the API call is just to save the settings or save and test the connection.
    • Values – test, save
    • Sample –
    • Example – func=save&key=abc&tab-redo=Y&type=tab
  • key –
    • Description – Name of the table.
    • Values – Table name.
    • Sample –
    • Example – func=save&key=abc&tab-redo=Y&type=tab
  • tab-redo –
    • Description – Specifies whether the initial extact is to be redone or not.
    • Values – Y, N.
    • Sample –
    • Example – func=save&key=abc&tab-redo=Y&type=tab
  • type –
    • Description – Describes the type of settings being saved.
    • Values – tab.
    • Sample –
    • Example – func=save&key=abc&tab-redo=Y&type=tab

Schedule Full Extract

API to schedule full extract of selected tables from the Schedule page.

URL: http://<host>:<port>/ingest/api/ingest?cmd=sched

Method: POST

Request body:

delta-daily-value=<hh:mm>&delta-periodic-value=<DdHHhMMm DdHHhMMm>&delta-select=<auto|periodic|daily|weekly>&delta-weekly-value=<hh:mm>&func=fulldata&type=schedule

  • The value of delta-select param in request body is either auto, periodic daily, weekly depending on whether automatic scheduler is selected or one of periodic, daily, weekly.
  • The values of delta-daily-value and delta-weekly-value will be in the form of hh:mm where hh is the number of hours and mm is the number of minutes. For eg: 18:46.
  • The delta-periodic-value has it’s value in the form DdHHhMMm DdHHhMMm. For eg: 2d02h14m 1d01h01m.
  • Optionally, the request body may also have additional params if any of the days of the week are selected. Only those days which are selected will be added with values set to on. For eg, if sunday and tuesday are selected, then request body will have additional params: &weekly-day1-value=on&weekly-day3-value=on

Manual Delta

Initiate delta run manually in BryteFlow Ingest

 

API to manually initiate Delta Extract of selected tables.

URL: http://<host>:<port>/ingest/api/ingest?cmd=sched&func=delta

Method: POST

Schedule Sync New Tables

API to schedule full extract of new tables only from the Schedule page.

URL: http://<host>:<port>/ingest/api/ingest?cmd=sched

Method: POST

Request body:

delta-daily-value=<hh:mm>&delta-periodic-value=<DdHHhMMm DdHHhMMm>&delta-select=<auto|periodic|daily|weekly>&delta-weekly-value=<hh:mm>&func=syncdata&type=schedule

  • The value of delta-select param in request body is either auto, periodic daily, weekly depending on whether automatic scheduler is selected or one of periodic, daily, weekly.
  • The values of delta-daily-value and delta-weekly-value will be in the form of hh:mm where hh is the number of hours and mm is the number of minutes. For eg: 18:46.
  • The delta-periodic-value has it’s value in the form DdHHhMMm DdHHhMMm. For eg: 2d02h14m 1d01h01m.
  • Optionally, the request body may also have additional params if any of the days of the week are selected. Only those days which are selected will be added with values set to on. For eg, if sunday and tuesday are selected, then request body will have additional params: &weekly-day1-value=on&weekly-day3-value=on

Schedule Sync Struct

API to sync the table structures on the source with that on the destination from the Schedule page.

URL: http://<host>:<port>/ingest/api/ingest?cmd=sched

Method: POST

Request body:

delta-daily-value=<hh:mm>&delta-periodic-value=<DdHHhMMm DdHHhMMm>&delta-select=<auto|periodic|daily|weekly>&delta-weekly-value=<hh:mm>&func=syncstruct&type=schedule

  • The value of delta-select param in request body is either auto, periodic daily, weekly depending on whether automatic scheduler is selected or one of periodic, daily, weekly.
  • The values of delta-daily-value and delta-weekly-value will be in the form of hh:mm where hh is the number of hours and mm is the number of minutes. For eg: 18:46.
  • The delta-periodic-value has it’s value in the form DdHHhMMm DdHHhMMm. For eg: 2d02h14m 1d01h01m.
  • Optionally, the request body may also have additional params if any of the days of the week are selected. Only those days which are selected will be added with values set to on. For eg, if sunday and tuesday are selected, then request body will have additional params: &weekly-day1-value=on&weekly-day3-value=on

Save scheduler frequency settings

API to save the delta extract frequency from the Schedule page.

URL: http://<host>:<port>/ingest/api/ingest?cmd=sched

Method: POST

Request body:

delta-daily-value=<hh:mm>&delta-periodic-value=<DdHHhMMm DdHHhMMm>&delta-select=<auto|periodic|daily|weekly>&delta-weekly-value=<hh:mm>&func=save&type=schedule

  • The value of delta-select param in request body is either auto, periodic daily, weekly depending on whether automatic scheduler is selected or one of periodic, daily, weekly.
  • The values of delta-daily-value and delta-weekly-value will be in the form of hh:mm where hh is the number of hours and mm is the number of minutes. For eg: 18:46.
  • The delta-periodic-value has it’s value in the form DdHHhMMm DdHHhMMm. For eg: 2d02h14m 1d01h01m.
  • Optionally, the request body may also have additional params if any of the days of the week are selected. Only those days which are selected will be added with values set to on. For eg, if sunday and tuesday are selected, then request body will have additional params: &weekly-day1-value=on&weekly-day3-value=on

Get loaded records count

Get loaded records count for the current Ingest instance:

URL: http://<host>:<port>/ingest/api/ingest?cmd=grf&func=<d or m>

Method: GET

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved or fetched.
    • Values – To fetch loaded records count, value is ‘grf’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=grf&func=d
  • func –
    • Description – Describes whether the API call is to get the loaded records count on an hourly basis for the current running day or to get the loaded records count on a daily basis for the current running month.
    • Values – d, m
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=grf&func=m

Response:

  • Sample response for func = d – [
    [
    [“0:00”, 0],
    [“1:00”, 0],
    [“2:00”, 0],
    [“3:00”, 0],
    [“4:00”, 0],
    [“5:00”, 0],
    [“6:00”, 10231],
    [“7:00”, 0],
    [“8:00”, 10231],
    [“9:00”, 10231],
    [“10:00”, 10231],
    [“11:00”, 0],
    [“12:00”, 0],
    [“13:00”, 0],
    [“14:00”, 0],
    [“15:00”, 0],
    [“16:00”, 0],
    [“17:00”, 0],
    [“18:00”, 0],
    [“19:00”, 0],
    [“20:00”, 0],
    [“21:00”, 0],
    [“22:00”, 0],
    [“23:00”, 0]
    ],
    []
    ]
  • Sample response for func = m –
    [
    [
    [“1 Mar”, 7335627],
    [“2 Mar”, 7366320],
    [“3 Mar”, 2956759],
    [“4 Mar”, 5719129],
    [“5 Mar”, 7366320],
    [“6 Mar”, 7366320],
    [“7 Mar”, 7366320],
    [“8 Mar”, 7366503],
    [“9 Mar”, 7366503],
    [“10 Mar”, 7366320],
    [“11 Mar”, 7345858],
    [“12 Mar”, 7366320],
    [“13 Mar”, 7366320],
    [“14 Mar”, 3754777],
    [“15 Mar”, 10231],
    [“16 Mar”, 0],
    [“17 Mar”, 40924],
    [“18 Mar”, 0],
    [“19 Mar”, 0],
    [“20 Mar”, 0],
    [“21 Mar”, 0],
    [“22 Mar”, 0],
    [“23 Mar”, 0],
    [“24 Mar”, 0],
    [“25 Mar”, 0],
    [“26 Mar”, 0],
    [“27 Mar”, 0],
    [“28 Mar”, 0],
    [“29 Mar”, 0],
    [“30 Mar”, 0],
    [“31 Mar”, 0]
    ],
    []
    ]
  • Explanation:
    • The response is a collection of 24 values for func=d, i.e. hourly records, and 30 or 31 values for func=m, i.e. daily records.
    • Each element has the corresponding hour or date and the number of records loaded in that hour or day.

Get the list of Selected Tables

Get the list of selected Tables for the current Ingest instance:

URL: http://<host>:<port>/ingest/api/ingest?cmd=croom&func=list

Method: GET

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved or fetched.
    • Values – To fetch the list of selected Tables, value is ‘croom’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=croom&func=list
  • func –
    • Description – Describes the kind of Ingest Pipeline details being fetched, eg. – Ingest status, list of selected tables, etc.
    • Values – ‘list’
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=croom&func=list

Response:

  • Sample response –
    • [{“tableName”:”default:TABLE_1″,”transfer”:”DP”,”isPartitioned”:false},{“tableName”:”default:TABLE_2″,”transfer”:”DP”,”isPartitioned”:false},{“tableName”:”default:TABLE_3″,”transfer”:”DH”,”isPartitioned”:false}]
  • Explanation:
    • The response is an array of selected tables.
    • Each item in the array describes a single selected table.
    • The property ‘tableName’ provides the name of the table.
    • The property ‘transfer’ indicates the value of delta extract setting for that table. ‘DP’ implies Delta extract by primary key, ‘DH’ implies Delta extract by primary key with history.
    • The property ‘isPartitioned’ is set to true if the table is partitioned and false if the table is not partitioned.

Select, Deselect a table for extraction

API to add or remove a table from the list of tables selected to be extracted.

URL: http://<host>:<port>/ingest/api/ingest?

Method: POST

Request body:

cmd=sel&func=table&name=<table_name>&sel=<Y|N>

  • ‘cmd’ is short for command. It is used to describe the type of settings being saved or fetched. To select, deselect a table, it’s value will be ‘sel’.
  • ‘func’ will have the value of ‘table’, indicating the process of table selection, deselection.
  • ‘name’ contains complete name of the table being selected, deselected. It will be in the form ‘db_name:schema_name.table_name’. Eg. ‘default:masteruser.tab_1’ for a table called ‘tab_1’ in a schema called ‘masteruser’ in a database called ‘default’.
  • If the table is to be selected, then ‘sel’ will have it’s value set to ‘Y’, and if it is to be deselected, then ‘sel’ will have the value ‘N’.

Configure Recovery

Configure Recovery for the current Ingest pipeline using below API :

URL: http://<host>:<port>/ingest/api/ingest?cmd=cop&id=4

Method: PATCH

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved.
    • Values – To save configuration settings, value is ‘cop‘.
    • Example – http://localhost:3000/ingest/api/ingest?cmd=cop&id=4
  • id –
    • Description – It is used to specify the id of the configuration tab whose data is being saved.
    • Values – Value is ‘4’ for Recovery config.
    • Example – http://localhost:3000/ingest/api/ingest?cmd=cop&id=4

Request Payload:

The payload will be in the form of a JSON array with four objects, each corresponding to a different field and it’s value.

[
{ “name”: “src_db_name”, “value”: <Pipeline name> },
{ “name”: “rec_enable”, “value”: <Y or N> },
{ “name”: “rec_s3dir”, “value”: <Recovery S3 directory> },
{ “name”: “rec_kms_id”, “value”: <KMS Id> }
]
The value for ‘src_db_name’ is the name with which this Ingest Pipeline should be identified.
The ‘rec_enable’ field is used to enable or disable the recovery feature. It’s value will be ‘Y’ to enable and ‘N’ to disable.
The ‘rec_s3dir’ specifies the path to the S3 directory where all the data related to the recovery of current Ingest pipeline is saved.
The ‘rec_kms_id’ field holds the KMS id string.

Response:
If the above data is saved successfully, the response would be –
{ “success”: true }
If the data could not be saved due to some error, the response would be –
{ “success”: false, “error”: <Error message> }

Get Recovery Status

Get the Recovery status for the current Ingest instance:

URL: http://<host>:<port>/ingest?cmd=ha

Method: GET

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved or fetched.
    • Values – To get the current Recovery status, value is ‘ha’.
    • Example – http://localhost:3000/ingest?cmd=ha

Response:

  1. When a fresh installation of Ingest is launched, the response will be as follows –
    {
    “code“: “NEW_OR_RECOVER”,
    “description“: “New Instance or Recovery?”
    }
    Here, the code indicates the current state of recovery. “NEW_OR_RECOVER” means the user needs to choose between a fresh new instance and an old instance from which data needs to be recovered. Choosing ‘Restore’ will load the current instance with data from subsequently chosen instance (provided the chosen instance had the recovery feature configured).
  2. If ‘New Instance’ is selected, then no additional data needs to be provided and a fresh Ingest instance will be started. However, if ‘Restore’ is selected, then the response of the above API will be as follows –
    {
    “code“: “GET_AWS_PARAMS”,
    “description“: “Enter AWS Access Details”
    }
    Optionally, a ‘region’ field might also be present with the value of AWS region. The code “GET_AWS_PARAMS” means that certain AWS credentials need to be entered in order to fetch the list of Ingest instances with recovery configured. Here, you can enter values such as AWS Region, Access Key Id, Secret Access Key, Proxy Host, Proxy Port. Proxy Host and Proxy Port are optional fields while other fields are mandatory.
  3. Once the above details are entered and submitted, the response of the above API becomes –
    {
    “code“: “GET_INSTANCES”,
    “instanceList“: [
    {
    isActive: false,
    expiry: <The timestamp when the most recent state of the current instance was saved before it became inactive>,
    name: <Instance name>
    },
    {
    isActive: true,
    expiry: <The timestamp when the most recent state of the current instance was saved before it became inactive>,
    name: <Instance name>
    }
    ],
    “description“: “Select an Instance”
    }
    The code “GET_INSTANCES” means that a list of instances is provided to choose from for recovery. The “instanceList” property is an array of instances. Each instance has a name, ‘isActive’ field to tell whether it is active or inactive and an expiry timestamp corresponding to the time when the most recent state of the current instance was saved before it became inactive. Only inactive instances can be selected for recovery.
  4. Once any instance is selected, the response of the above API changes to –
    {
    “code“: “NORMAL”
    }
    This means that the ingest instance is being recovered and launched. Nothing needs to be done at this point.
  5. After some time, the response changes to-
    {
    “code“: “READY”
    }
    At this point, the ingest instance is recovered and ready to be launched.

Get licence information for Ingest

Get the licence information for Ingest instance:

URL: http://<host>:<port>/ingest/api/ingest?cmd=co&id=3

Method: GET

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved or fetched.
    • Values – To fetch the configuration settings, value is ‘co’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=c0&id=3
  • id –
    • Description – This is the id of the configuration tab whose details are being fetched.
    • Values – 0 for ‘Source Database’, 1 for ‘Destination Database’, 2 for ‘Credentials’, 3 for ‘Licence’, 4 for ‘Recovery’, 5 for ‘Remote Monitoring’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=c0&id=3

Response:

  • Sample response –
    • [{
      “description”:”Product Id”,
      “listValues”:””,
      “mandatory”:false,
      “name”:”_lic_prod”,
      “size”:”40″,
      “type”:”readonly”,
      “value”:”xxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx”
      },
      {
      “description”:”Licence Key”,
      “listValues”:””,
      “mandatory”:false,
      “name”:”_lic_key”,
      “size”:”40″,
      “type”:”text”,
      “value”:”xxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx-xxxxx”
      },
      {
      “description”:”Status”,
      “listValues”:””,
      “mandatory”:false,
      “name”:”_lic_status”,
      “size”:”25″,
      “type”:”readonly”,
      “value”:”Expires on 01 Nov 2023 (Standard Edition 10TB)”
      }]
  • Explanation:
    • The response is an array of objects.
    • Each item in the array describes a single input field on the UI.
    • The property ‘description’ provides the name of the field.
    • The property ‘listValues’ provides the options to be shown in the dropdown for the current field if the field has a dropdown.
    • The property ‘mandatory’ indicates whether providing the value for the current field is mandatory or not. If true, then this value needs to be provided before saving.
    • The property ‘name’ is the id of the field.
    • The property ‘size’ specifies the maximum number of characters that can be provided in the input value.
    • The property ‘type’ tells whether the field is readonly in which case it cannot be edited, or text, number, password, list, etc.
    • The property ‘value’ provides the current value of the field.

Save new Licence key for Ingest

Save new licence key for Ingest instance:

URL: http://<host>:<port>/ingest/api/ingest?cmd=cop&id=3

Method: PATCH

JSON Payload:
[
{
“name”: “_lic_prod”,
“value”: “xxx-xxxxx-xxxxx-xxxxx”
},
{
“name”: “_lic_key”,
“value”: “xxx-xxxxx-xxxxx-xxxxx”
},
{
“name”: “_lic_status”,
“value”: “Expires on 01 Nov 2023 (Standard Edition 10TB)”
}
]

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved or fetched.
    • Values – To save the configuration settings, value is ‘cop’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=c0p&id=3
  • id –
    • Description – This is the id of the configuration tab whose details are being saved.
    • Values – 0 for ‘Source Database’, 1 for ‘Destination Database’, 2 for ‘Credentials’, 3 for ‘Licence’, 4 for ‘Recovery’, 5 for ‘Remote Monitoring’.
    • Sample –
    • Example – http://localhost:3000/ingest/api/ingest?cmd=c0p&id=3

Payload:

  • The payload is a JSON array of objects.
  • Each object represents each individual input field on the current configuration tab whose value needs to be saved.
  • The object has just 2 keys. ‘name’ key specifies the id of the input field whose value is being saved in current object. ‘value’ field is the actual value of the field that needs to be saved.

Response:

  • Sample response –
    • { “success”: true }
  • Explanation:
    • The response is a simple JSON object with ‘success’ key. If config data was saved correctly then its value is true, else in case of errors its value is false.

Provide data for recovery purposes

Provide the data during the Recovery process for the current Ingest instance:

URL: http://<host>:<port>/ingest?cmd=hap

Method: POST

URL Params:

  • cmd –
    • Description – Short for command. It is used to describe the type of settings being saved or fetched.
    • Values – To update the data for Recovery process, value is ‘hap’.
    • Example – http://localhost:3000/ingest?cmd=hap

Request Payload and Response:

  1. When a fresh installation of Ingest is launched, the response of ‘/ingest?cmd=ha‘ API will be as follows –
    {
    “code“: “NEW_OR_RECOVER”,
    “description“: “New Instance or Recovery?”
    }
    At this point, the user needs to choose between a fresh new instance and an old instance from which data needs to be recovered.
  2. To launch a NEW instance, the payload for the above API ‘/ingest?cmd=hap‘ is –
    {
    “code“: “NEW_OR_RECOVER”,
    “type“: “new”
    }
    The response would be –
    {
    “code”: “normal”
    }
    This means that a new instance is being launched. Once the code changes to ‘ready’, the instance is ready.

    To recover an existing instance, the payload would be –
    {
    “code“: “NEW_OR_RECOVER”,
    “type”: “recover”
    }
    The response would be –
    {
    “code“: “GET_AWS_PARAMS”,
    “description“: “Enter AWS Access Details”
    }

  3. At this stage, the AWS credentials such as AWS Region, Access Key Id, Secret Access Key, Proxy Host, Proxy Port need to be entered in order to fetch the list of Ingest instances with recovery configured. These details are provided in the payload for API ‘/ingest?cmd=hap‘ as shown below –
    {
    “region“: <AWS region>,
    “key“: <Access Key Id>,
    “skey”: <Secret Access Key>,
    “proxyHost“: <Proxy Host>,
    “proxyPort“: <Proxy Port>
    }
    The response would be –
    {
    “code“: “GET_INSTANCES”,
    “instanceList“: [
    {
    isActive: false,
    expiry: <The timestamp when the most recent state of the current instance was saved before it became inactive>,
    name: <Instance name>
    },
    {
    isActive: true,
    expiry: <The timestamp when the most recent state of the current instance was saved before it became inactive>,
    name: <Instance name>
    }
    ],
    “description”: “Select an Instance”
    }
  4. After getting a list of instances, one instance needs to be selected to recover the data from (only an inactive instance can be chosen). The payload for the API ”/ingest?cmd=hap” to select an instance would be like –
    {
    “code“: “GET_INSTANCES”,
    “name“: <Name of the selected instance as obtained in the previous stage in the ‘instanceList‘ array>
    }
    The response would be –
    {
    “code“: “NORMAL”
    }
    This means that the ingest instance is being recovered and launched. Nothing needs to be done at this point.
  5. After some time, the response of the API ‘/ingest?cmd=ha‘ changes to –
    {
    “code“: “READY”
    }
    At this point, the ingest instance is recovered and ready to be launched.

BryteFlow Enterprise Manager API's

BryteFlow Enterprise Manager API details are as below: 

Enterprise Manager - Generate Authorization Bearer Token

Below API will generate the bearer token for authorization of further communication with the enterprise
manager APIs:

URL: https://<host>:<port>/authenticate

Method: POST

Request Payload: Raw(JSON)

{ “username”: <User Name>, “password”: <Password> }

 

Enterprise Manager - Get list of available templates

Below API will return the list of templates:

URL: https://<host>:<port>/api/templates

Method: POST

Authorization: Bearer Token

Request Headers:

Content-Type: application/json
Accept: */*
Authorization: Bearer <Token>

Enterprise Manager - Create a new Instance

Below API will create a new instance:

URL: https://<host>:<port>/api/instance

Method: POST

Authorization: Bearer Token

Request Headers:

Content-Type: application/json
Accept: */*
Authorization: Bearer <Token>

Request Payload: Raw(JSON)

{
“id”: “”,
“product”: “ingest”,
“template”: <templateName>,
“status”: “stopped”,
“config”: [
{
“name”: “webPort”,
“value”: <port>
},
{
“name”: “InstanceName”,
“value”: <name>
}
]
}

Enterprise Manager - Start or Stop an instance

Below API will start or stop the specified instance:

URL: https://<host>:<port>/api/instance/:id/:cmd

Method: POST

Authorization: Bearer Token

Request Headers:

Content-Type: application/json
Accept: */*
Authorization: Bearer <Token>

Path variables:

id: (Required) Id of the instance.
cmd: (Required) Values can be start or stop.

Enterprise Manager - Delete an Instance

Below API will delete the specified instance:

URL: https://<host>:<port>/api/:id

Method: DELETE

Authorization: Bearer Token

Request Headers:

Content-Type: application/json
Accept: */*
Authorization: Bearer <Token>

Path variables:

id: (Required) Id of the instance.

Enterprise Manager - List all Instances

Below API will return the list of all instances:

URL: https://<host>:<port>/api/instance

Method: GET

Authorization: Bearer Token

Request Headers:

Content-Type: application/json
Accept: */*
Authorization: Bearer <Token>

Suggest Edit
BryteFlow API Documentation