1. Overview

This article describes how to monitor the progress of a Data Stream and its associated files.

4 levels of information are available:

• Data Stream Overview, about the Data Stream metrics
• Data Stream Information, about the Data Stream characteristics
• Data Stream Files, to monitor the progress of each individual file of a Data Stream.
• Data Stream Assets, to monitor the assets coverage of a Data Stream.

To open the Data Stream monitoring page, click on a Data Stream from the Data Stream dashboard.

2. Data Stream status

A Data Stream can passes through the following states: 

• Pending: For the Data Stream initialization.
• Listing: When the Data Stream connects to the data source and lists the files to import.
• Running: At the start of the import of files and of the processing steps..
• Paused: When the Data Stream is paused.
• Waiting for completion: When the files are all imported, contextualized, and transformed.
• Completion: Completion in progress
• Completed: When the Data Stream is completed and can be used by the other modules.

3. Data Stream actions

3.1. Complete a Data Stream

To complete a Data Stream, click on “Complete Data Stream” button. 

3.2. Delete a Data Stream

To delete a Data Stream, click on “Delete Data Stream” button. 

3.3. Pause and Resume a Data Stream

To pause a Data Stream in Running status, click on the “Pause Data Stream” button. 

Once paused, click on “Resume Data Stream” button to restart the processing. 

4. Data Stream Metrics

The overview highlights key metrics and anomalies that need immediate attention.

The first block displays the number of files contained in the Data Stream and their global status:

• The number of completed files 
• The number of files in progress
• The number of file with errors

The second block indicates the details of each status. 

Shortcuts allow to quickly access related files (magnifying glass icon)

5. General information panel

5.1 Data Stream Information

From the Data Stream monitoring page, get the general information entered when creating the Data Stream: 

• Template: Selected template 
• Name: Data Stream name
• Description: Data Stream Description
• Data Stream source: Data source 
• Company: Company associated with the Data Stream
• Data Capture - start date: Data capture start date
• Data Capture - end date: Data capture end date

5.2 Data Stream Quality checks

The list of Quality checks applied to the imported files is listed in the Quality checks section

5.3 Data Stream footprint on the map

The datatsream footprint can be visualized on the map by clicking on the target icon next to datastream name (available for some data streams only). 

6. Data Stream files panel

6.1 Files panel overview

The "FILES" tab lists the files contained in the Data Stream as well as their :

• Status, import date
• Contextualization status: if the contextualization step is activated
• Quality check status: if the quality check step is activated
• Export date: if the export step is activated

The data Stream files processing steps depend on the template's configuration. we have mandatory and optional statuses

A  data Stream files can pass through the following status:  

• Pending: At Data Stream file initialization
• Importing: the Data Stream file is being imported
• Imported: the Data Stream file is imported
• Transforming (optional): the Data Stream file is being transformed
• Transformed  (optional) : the Data Stream file is transformed
• Complete: the Data Stream processing is completed
• Importing Error: the Data Stream import is in error
• Transforming Error (optional): the Data Stream transformation is in error
• Export Error  (optional): the Data Stream export is in error

The contextualization is an optional step, if it activated the Data Stream files contextualization status can passes through the following status: 

• Pending: At Data Stream file initialization
• Contextualizing: the Data Stream file is being contextualized
• Contextualized: the Data Stream file is contextualized
• Contextualizing Error: the Data Stream contextualization is in error

The Quality checks are an optional step, if it configured the Data Stream files quality checks status can passes through the following status: 

• Pending: quality check not applied yet
• Passed: all the checks are passed
• Failure: at least one check failed

Click on a data stream file to access its details panel. 

6.2 Search a file

The search function allows users to easily find a Data stream file by its name or by its quality check status. The search can be done through the search field at the top right of the Data stream files list.

6.3 Files footprint visualization on the map

The file footprint of each file can be visualized on the map by clicking on the target (only available for some data streams). 

6.4 Filter files panel

The filter function allows users to display only the Data Stream files that meet the selected status, contextualization status and quality check status. 

Apply filters by following these steps:

Step 1 - Click on the "FILTERS" button

Step 2 - Select one or several statuses, then click on "APPLY".

Step 3 - The Data Stream files list is updated accordingly.

7. Data Stream assets panel

7.1. Asset panel overview

The "ASSETS" tab lists the assets contained in the Data Stream as well as their coverage, coverage percentage and asset version (only available for some data streams).

7.2 Asset footprint visualization on the map

The asset footprint of each asset, the Data Stream footprint and the files covering the asset can be visualized on the map by clicking on the target. 

Click on a data stream file to view its details. 

7.3 Search an asset

The search function allows users to easily find a Data stream asset by its name. The search can be done through the search field at the top right of the Data stream assets list.

Please note that results will be obtained if the content of the search field is exactly the asset name.

7.4 Filter asset panel

It is possible to filter the view by asset type and cover percentage (min and max). 

Several asset types can be selected and it can be cumulated with the search tool.