Despiking Aquadopp dataset

We’re using a Nortek 600Khz AquaDopp - 2 meter bin in a water depth of 40m.  The general trend of the data (around 2 weeks worth) is lower current speeds at neap tide, then higher at spring, and then reduces again. But there are many unexplained spikes in the time series set.

We have attempted despiking using Goring-Nikamura phase space method (http://mulgor.co.nz/despiking.pdf ). Assuming it works for ADCPs, and that we got the method right – we have reduced the obvious spikes, but then we miss the peaks for the spring tide speeds, and still do not remove some unexpected peaks during the neap.

Any thoughts on how to process the attached dataset?

Thanks,

Lena

Hi Lena,

  I've never used the Nikora-Goring method you mention although I am aware of it. I've always wound up using an adaptive outlier filter, generally one based on removing outliers outside some set standard deviation range of the data.

That won't work for you because of the changing mean unless you remove the low frequency velocity components (the tides) first and are then screening the fluctuations, which should have a constant mean or pretty close to one. Try searching for "adaptive gaussian filter" and you should find something online.

An alternative would be to use a local median filter that compares the center point to a selection of points around it (say 5 or 9 so it's evenly centered) and removing the center point if it's significantly different than the local median. What "significantly different" actually is will depend on what works to remove the spikes.

Both methods should produce similar results, although I think the second one might be quicker to code initially. I haven't looked at your dataset, but assuming it's a profile, you could also filter in two dimensions (time and space) using either of the above methods.

P.J.

Thank you Peter! I'm sorry to have taken so long to answer... Busy as usual.

We didn't have so much more time trying to find different ways to do it, and sticked with the Goring Phase Space Threshold method of despiking. We eventually found out that Nobuhito Mori has uploaded a MATLAB toolbox that handles the calculations based on this method. It is not quite as effective as we had originally thought it would be, but the fact that the method doesn't require too many manual adjustments is seducing. We wanted a one-for-all method, and that is not so destructive (smoothing methods using median like Hanning for instance)!

I think we will therefore stick to this method for now and see if it can be improved. For those interested there are also some indication in a follow up paper about how to improve the robustness of the script, as the mean and standard deviation are not adequate estimators when dataset is badly corrupted. I will look at the adaptative Gaussian filter though.

PS: If you wonder why I suddenly changed name, it's just because my colleague was taking care of the post and related processing tasks, and I took over! No monkey business other than switching the user of the account that we shared for the task...