Data Flow - Data Stream Management
1. Overview
This article describes how to monitor the progress of a Data Stream and its associated files.
4 levels of information are available:
- Data Stream Overview, about the Data Stream metrics
- Data Stream Information, about the Data Stream characteristics
- Data Stream Files, to monitor the progress of each individual file of a Data Stream.
- Data Stream Assets, to monitor the assets coverage of a Data Stream.
To open the Data Stream monitoring page, click on a Data Stream from the Data Stream dashboard.
2. Data Stream status
A Data Stream can passes through the following states:
- Pending: For the Data Stream initialization.
- Listing: When the Data Stream connects to the data source and lists the files to import.
- Running: At the start of the import of files and of the processing steps..
- Paused: When the Data Stream is paused.
- Waiting for completion: When the files are all imported, contextualized, and transformed.
- Completion: Completion in progress
- Completed: When the Data Stream is completed and can be used by the other modules.
3. Data Stream actions
3.1. Complete a Data Stream
To complete a Data Stream, click on “Complete Data Stream” button.
Info
- Completing a Data Stream is only possible when the Data Stream is in “Waiting for completion” status
- In case of a problem during completion, the status will be updated to “Completion error”.
3.2. Delete a Data Stream
To delete a Data Stream, click on “Delete Data Stream” button.
Info
- Deletion is available at any status of the Data Stream.
3.3. Pause and Resume a Data Stream
To pause a Data Stream in Running status, click on the “Pause Data Stream” button.
Once paused, click on “Resume Data Stream” button to restart the processing.
4. Data Stream Metrics
The overview highlights key metrics and anomalies that need immediate attention.
The first block displays the number of files contained in the Data Stream and their global status:
- The number of completed files
- The number of files in progress
- The number of file with errors
The second block indicates the details of each status.
Shortcuts allow to quickly access related files (magnifying glass icon)
5. General information panel
5.1 Data Stream Information
From the Data Stream monitoring page, get the general information entered when creating the Data Stream:
- Template: Selected template
- Name: Data Stream name
- Description: Data Stream Description
- Data Stream source: Data source
- Company: Company associated with the Data Stream
- Data Capture - start date: Data capture start date
- Data Capture - end date: Data capture end date
5.2 Data Stream Quality checks
The list of Quality checks applied to the imported files is listed in the Quality checks section
5.3 Data Stream footprint on the map
The datatsream footprint can be visualized on the map by clicking on the target icon next to datastream name.
6. Data Stream files panel
6.1 Files panel overview
The "FILES" tab lists the files contained in the Data Stream as well as their file status, file contextualization status, quality check status and import date.
A data Stream files can pass through the following status:
- Pending: At Data Stream file initialization
- Importing: the Data Stream file is being imported
- Imported: the Data Stream file is imported
- Transforming: the Data Stream file is being transformed
- Complete: the Data Stream processing is completed
- Transforming Error: the Data Stream transformation is in error
- Error: the Data Stream processing is in error
The Data Stream contextualization status can passes through the following status:
- Pending: At Data Stream file initialization
- Contextualizing: the Data Stream file is being contextualized
- Contextualized: the Data Stream file is contextualized
- Contextualizing Error: the Data Stream contextualization is in error
The Data Stream quality checks status passes through the following status:
- Pending: quality check not applied yet
- Passed : all the checks are passed
- Failure: at least one check failed
Click on a data stream file to access its details panel.
6.2 Search
The search function allows users to easily find a Data stream file by its name or by its quality check status. The search can be done through the search field at the top right of the Data stream files list.
Warning
Please note that results will be obtained if the content of the search field is exactly the file name.
6.3 Files footprint visualization on the map
The file footprint of each file can be visualized on the map by clicking on the target.
6.4 Filter files panel
The filter function allows users to display only the Data Stream files that meet the selected status, contextualization status and quality check status.
Info
- For each filter, several values can be selected.
- Request between each field is and OR, that means it is status equals "xxx" OR "contextualization status" equals "yyy" OR “quality check status” equals “zzz”.
Apply filters by following these steps:
Step 1 - Click on the "FILTERS" button
Step 2 - Select one or several statuses, then click on "APPLY".
Step 3 - The Data Stream files list is updated accordingly.
7. Data Stream assets panel
7.1. Asset panel overview
The "ASSETS" tab lists the assets contained in the Data Stream as well as their coverage and coverage percentage.
7.2 Asset footprint visualization on the map
The asset footprint of each asset, the Data Stream footprint and the files covering the asset can be visualized on the map by clicking on the target.
Click on a data stream file to view its details.
7.3 Search an asset
The search function allows users to easily find a Data stream asset by its name. The search can be done through the search field at the top right of the Data stream assets list.
Please note that results will be obtained if the content of the search field is exactly the asset name.
7.4 Filter asset panel
It is possible to filter the view by asset type and cover percentage (min and max).
Several asset types can be selected and it can be cumulated with the search tool.
