Noise reduction

Vista Data Vision includes a powerful Validation Toolkit that allows for automatic noise removal.  Validation simplifies the time consuming task of having to review data to find and remove out of range data, noise and spikes.  With Vista Data Vision there is no need for a specific standalone noise removal software since it is already included in VDV.

Validation is used to automatically or manually remove noise data in the database according to user defined boundaries. The user can define a maximum and minimum value, which if exceeded the validation toolkit will replace the value to a last known good value.

How does it work?

1. In db.robot.c, set out-of-range limits for one or more (all) sensors.
2. The Validation process checks the data in the database.  It runs every few minutes and checks latest data for errors.
3. If out-of-range noise data is found, it is replaced according to the defined bounderies.  If a change is made, the process is logged.

 

Validation Methods

1. Linear Interpolation over out of range values.
2. Out of range values replaced with NaN.
3. Replace out of range values with last known legal value

 

When validating data Vista Data Vision first checks if the value is within the legal data range, if it is not within the legal limit it next checks if the number of consecutive values out of the legal data range are lower or equal to the selected Gap Size.  If they are lower or equal to the gap size then the values are replaced by the validation method chosen, else no action is taken.

For example:  If the data values for a variable are "23.15; -100; 24.21; 23.98; -6999; -6999; -6999; 24.37; 25.14; 26.73;" and the lower limit for the variable is set to -50 and the upper limit is set to 150.

• If the Gap Size is set to 4 then after the validation process those same values would be: "23.15; 23.15; 24.21; 23.98; 23.98; 23.98; 23.98; 24.37; 25.14; 26.73;", all noise has been removed.
• If the Gap Size had been set to 2 then after the validation process those same values would be: "23.15; 23.15; 24.21; 23.98; -6999; -6999; -6999; 24.37; 25.14; 26.73;".

The Validation Process can be made Automatic or Users can perform Validation Manually for a selected time period.

Image showing a graph with 5 variables. 3 of the Variables have 2 NAN records. By Configuring the Validation Limits for each of those variables to -100 for the Lower Limit and -1 for the Upper Limit, the Validation Process automatically cleans up the data.

The same graph after the Validation Process has cleaned up the data.

All Validations are logged for later review.