Ingest XL is a companion software to Ingest. When large tables (>200GB) are included in Ingest, a conventional full extract may not be feasible. Due to the long period it requires, it is highly likely to encounter a problem. In case of a problem, the entire initial extract has to be rerun. In a way, Ingest XL becomes indispensable when large tables are involved.
Ingest XL helps transfer large tables across. The normal use case is as below:
The subsequent steps are executed on Ingest XL. All tasks in Ingest XL are long running. Therefore, tasks are never directly executed but scheduled. Once run, the results are available. Ingest XL allows the tasks to be monitored and, if needed, cancelled before they start running. The steps are as below:
This section refers to the config location of Ingest along with the licensing information.
When using AMI, both fields are not required.
Configure the tables in this section, usually the list of tables on the left is directly populated from Ingest.
Additionally, a user may select one or more tables to be processed by Ingest XL by clicking on the checkbox.
For a selected table, selecting a slicing column is mandatory. Generally, the Primary key is used as a slicing column.
The Where field is rarely used except in SAP environments. The slices, if manually entered are likely to be one line. Any more than that and it is likely that users will use the Auto-slice function.
On clicking the Auto-slice button a dialog box is presented.
The user starts by selecting Schedule Count so that the number of records gets populated. This takes a while and the user may have to monitor the progress on the Task Status page. After entering the prefix size and slice count, Schedule Slice is selected. This also is a long process after which, the slices field will be populated in the background page in the slices field. This is likely to run into several lines.
The Schedule Slice button is disabled until the number of records is known. For partitioned tables, a Schedule Count is not required and the Schedule Slice button is enabled at all times.
Once tables have been selected, they appear on the Table Status page. A few other details about each table are also displayed. The bar graph on the right indicates the status of each table i.e. how much of the data has been transferred across.
Click on the table to show the Table Slicing Status
The Table Slicing Status page lists all slices in a table. This could be a single slice if the table is not sliced or a list of partitions in case of a partitioned table. There are two buttons for each slice. One is to schedule a quick compare of the slice between the source and destination and another is to schedule transfer the data across. Both buttons have the following states and colour in the current design:
Future versions may allow these actions to be done at the table level. The actions may be scheduled only for slices that need it.
This is where User gets to see the list of tasks in the last 24 hours.
Tasks have three statuses – Scheduled, Running, Completed and Failed.
There are checkboxes to filter based on the task status. There is pagination support, allowing users to navigate between pagefuls. The natural sort order is Scheduled, Running, Completed and Failed. Scheduled tasks will have a cancel button.
The dashboard gives an overview of the application.
The first panel provides information of the amount of data being looked after.
The second panel shows any work done over the last 24 hours. This includes the amount of time the application spent connected to the source and destination (Source and Dest. Duration).
A pie chart in the middle shows tasks in various states in the last 24 hours. The states are as follows:
A bar chart below shows the amount of data transferred over the last 30 days.