appropriate correlation and amplitude thresholds

Posted by Ted Lewis at March 11. 2009

Hello,

We have data from a 2 MHz aquadopp profiler in high resolution mode

(other settings: 6 s average interval, 12 pings per burst, 98 x 3 cm cells, 0.3 m/s horizontal vel. range, 0.13 m/s vertical vel range).

I'm trying to filter 'bad' data out of our velocity files.

Are there appropriate amplitude and threshold values to use? Corr>50 and amp>25-35 seem like rules of thumb for other instruments... are these also appropriate for the 2MHz HR profiler? Are these values appropriate for all three beams at all depths?

Is there a minimum velocity value that should also be used as a threshold?

How do other users do this thresholding? Matlab?

Thanks,

Ted

Current state: Being created

Re: appropriate correlation and amplitude thresholds

Posted by Atle Lohrmann at March 12. 2009

Dear Ted, I hope some of the other HR users will share some of their experiences.

A couple of general comments:

a) The HR measures using two pings to measure velocity but we only store one amplitude profile (the first one). This means that you can have pulse-to-pulse interference without seeing it in the amplitude profile. The cut-off criterion of around 25 counts for the 2 MHz profiler is otherwise correct - although it is rare that it become an effective criterion unless one or more beams are covered with something.

b) The (de-)correlation seems to contain three separate elements:

Interference from the first pulse when you are trying to listen to the second pulse
Acoustic mechanism due to beam divergence, residence time, turbulence, etc (usually scales with velocity)
(Under)sampling mechanisms

The first one is easy to sort out with a fixed correlation limit. The second two criteria less so and I would recommend experimenting with a correlation value that scales with the mean velocity. If not, you risk throwing valid data that are collected close to (or exceeding) the ambiguity velocity.

Best regards, Atle Lohrmann

Current state: Being created

Re: appropriate correlation and amplitude thresholds

Posted by P.J. Rusello at June 06. 2009

Ted,

Here are a few thoughts on screening HR data and responses to your questions.

> Are there appropriate amplitude and threshold values to use? Corr>50 and amp>25-35 seem like rules of thumb for other instruments...

Throwing away obviously bad data (e.g. correlations less than 50) will work well to remove the really bad points as will an amplitude or SNR threshold. I recommend taking a look at histograms of velocity, correlation, and SNR to assist in setting appropriate threshold values for this initial screening.

To really remove outliers in a robust way, I recommend developing an adaptive outlier filter based on the spread of the data. There are a lot of ways to do this, with the simplest being to use the standard deviation (either from Student's T distribution of a normal distribution) and the mean and removing points outside some threshold based on these two values. Keep recalculating the standard deviation and removing points until you have reached some stopping criteria. Again, histograms will help with this process.

> are these also appropriate for the 2MHz HR profiler? Are these values appropriate for all three beams at all depths?

I unfortunately haven't looked at enough HR data to say for certain on the first question, but correlation thresholding should be fairly universal for most pulse coherent systems (there are always exceptions). I typically use the same thresholds for all beams and bins when tossing data based on quality. Each bin will be screened differently when I apply an adaptive filter.

> Is there a minimum velocity value that should also be used as a threshold?

This will be flow dependent. Again looking at your velocity histograms, you can do a first pass and set upper and lower limits and throw any data points outside of those away. This removes the outliers that are so far out they'll skew statistics used for an adaptive filter.

> How do other users do this thresholding? Matlab?

I've used Matlab quite extensively for this type of analysis, but because they seem to get buggier and buggier under OS X I've decided to switch to Python and it's numerical packages. Almost any scripting language will let you do this with varying degrees of ease based on your familiarity with the language. Just use what you know how to work in.

Good luck!

Sections

appropriate correlation and amplitude thresholds