Successive Vectrino "mini-profiles" on a vertical don't line up

Hello,

I used a Nortek Vectrino II ADV to record velocity profiles in a small experimental channel but have noticed some suspicious trends in the data.  Our plan was to use the Vectrino II with a 3 cm sampling cylinder to collect a series of "mini-profiles" that we could then stack on top of one another to characterize flow structure throughout the water column, from the bed to the surface.  However, when I plot the data from individual data sets collected at different vertical positions, there is a very noticeable and abrupt offset from one mini-profile to the next as you move up away from the bed (please see first attachment).  The data look OK within a mini-profile, but the jumps from one to the next can't be right - is there some kind of correction I need to apply to account for this offset?  Both the temperature and speed of sound change a little between data files, but not by much.

Also, more generally, the data are very, very noisy.  I tried double averaging (in time over the 60 s, 50 Hz series and spatially over the 30 cm sampling cylinder of each mini-profile), but the flow field still does not appear very coherent, even when I use the velocity magnitude; the second attachment is  a plot of the cross-section we measured.  

I haven't attempted any filtering or de-spiking yet, but this seems like a more systematic problem that I need to address first.

Any advice will be greatly appreciated.

Thanks,

Carl

Here is the second attachment showing the velocity magnitude throughout the cross-section.

Hi Carl:

   What you're seeing appears to be phase wrapping within the velocity profiles.  You need to make sure that the velocity in the profile doesn't exceed the selected velocity range  (as shown in the Horizontal and Vertical velocity ranges).  Increasing the velocity range will  remove those discontinuities. 

  Hope this helps,

      Robert.

Hi Robert,

These data are from work I performed as a visitor at another lab, and I was not familiar with the instrument, so I fear that the velocity range I used might not have been sufficient.  Looking back at the Config structure in the MATLAB output, the velocity range was 300 mm/s and the horizontal and vertical components were 743 and 216 mm/s.  I can't go back and change those settings now, and re-collecting the data isn't really feasible either, so are you aware of any methods I might use to detect and correct for the phase wrapping at this point?  The correlation and SNR values are reasonably high, so hopefully the data can be salvaged.

Thanks for your help,

Carl

P.S.  Please let me know if attaching a sample data file would be useful.

Previously Robert Craig wrote:

Hi Carl:

   What you're seeing appears to be phase wrapping within the velocity profiles.  You need to make sure that the velocity in the profile doesn't exceed the selected velocity range  (as shown in the Horizontal and Vertical velocity ranges).  Increasing the velocity range will  remove those discontinuities. 

  Hope this helps,

      Robert.

Search for "ambiguity jumps" or "phase wrapping" and you'll find various posts on how to correct these. I think it is also covered in the Pulse Coherent Primer in the Technical Notes section of the website.

Developing an algorithm to detect these unfortunately has to be a sort of one off piece of analysis. How easy they are to identify and correct will depend entirely on how frequent they are and how much the velocity has wrapped. You might be able to identify them simply by a sign change, but there's no guarantee this will work.

P.J.

Hi Carl:

   I had another look at the data and it does look quite odd when I think about it.  Discrete jumps in velocities are usually a sign of phase wrapping, but if the flow hasn't changed in between profiles, then that would seem unlikely.   What sort of environment was this collected in?  Tuning things to work in a flume (for example) can be quite tricky because of the number of echoes generated in a small volume. What sort of configuration settings were used?  I've also been trying to track down a problem with the dynamic adpative mode of operation and was wondering if you had enabled this setting. 

     Thanks!

    Robert

Hello P.J.,

Thanks for your response.  I looked at the primer you mentioned and can try the approach you've outlined to determine if my data have indeed been subject to phase-wrapping.  However, the first step involves a transformation to beam coordinates and I'm not sure how to proceed using the information in the MATLAB export file.  The Config structure array has a field called Config.Probe_hBeamToXYZ that is a 16 X 1 vector, whereas the Vectrino II should be a 4 X 4 matrix.  I also noted that the values need to be rescaled, so this line of code gave me:

T   =   reshape(Config.Probe_hBeamToXYZ,4,4)/4096;

Does this look reasonable to you?  The zeros strike me as suspicious.

I then used b   =   T\u' to invert the matrix, where u is a matrix with the four velocity components in xyz, to get back to beam coordinates, but the data still look very spiky (see attachment), although they do tend to max out at about the nominal horizontal velocity maximum for the velocity range I used (0.3 m/s).  Please note that I'm not sure the beam numbers are labeled correctly in the plot.

Any further thoughts on this?

Thanks again,

Carl

Previously P.J. Rusello wrote:

Search for "ambiguity jumps" or "phase wrapping" and you'll find various posts on how to correct these. I think it is also covered in the Pulse Coherent Primer in the Technical Notes section of the website.

Developing an algorithm to detect these unfortunately has to be a sort of one off piece of analysis. How easy they are to identify and correct will depend entirely on how frequent they are and how much the velocity has wrapped. You might be able to identify them simply by a sign change, but there's no guarantee this will work.

P.J.

Hi Robert,

Thanks for your thoughts on this data set.  The measurements were made with a Vectrino II at the Saint Anthony Falls Outdoor Stream Lab facility, a small artificial but fairly natural channel with a real sediment bed and no flume side walls or anything like that.  The depths were up to about 0.4 m with a width of ~1.5 m.  We set the velocity range to 0.3 m, which might have been the fatal flaw.  I was a visitor at this lab and just used the instrument as it was configured, so I'm not certain about the dynamic adaptive mode or other settings I am attaching the Config structure array from one of our MATLAB export files, so hopefully that has the information you need.  Please let me know if the actual data would be useful, but I'm afraid the attachment might be too large.

Thanks again, and I'll appreciate any advice you have to offer.

Carl

Previously Robert Craig wrote:

Hi Carl:

   I had another look at the data and it does look quite odd when I think about it.  Discrete jumps in velocities are usually a sign of phase wrapping, but if the flow hasn't changed in between profiles, then that would seem unlikely.   What sort of environment was this collected in?  Tuning things to work in a flume (for example) can be quite tricky because of the number of echoes generated in a small volume. What sort of configuration settings were used?  I've also been trying to track down a problem with the dynamic adpative mode of operation and was wondering if you had enabled this setting. 

     Thanks!

    Robert

Hi Carl,

The transformation matrix for the Vectrino II range cells is stored in Config.ProbeCalibration_calibrationMatrix, one row per each cell. To get the calibration matrix in a more usable form for transforms I usually do this

T = reshape( Config.ProbeCalibration_calibrationMatrix( cell, : ), 4, 4 )'  <---- note the transpose

and to convert from XYZ to beam I usually do inv( T ) * [ x; y; z1; z2 ]

The matrix you are using is I think for the non-profiling versions of the Vectrino II. The zero entries are supposed to be there for that one because the four receivers form two independent sub-systems. 

You can actually usually spot phase wrapping without needing to transform to beam coordinates (that just makes the correction much simpler) by looking for discontinuities in the time series. Even with the wrong transformation matrix, the time series you provided suggest measurement quality needs some improvement.

In the post above you mentioned the velocity range was 0.3 m/s, which for the ping algorithm used yields a horizontal range of about 0.75 m/s and vertical range of 0.216 m/s. Do those sound reasonable for the flow you were working in? I've seen photos of this stream facility before, it is quite cool, but I suspect it actually has pretty high turbulence so the velocity range might have needed a little more tuning to produce the best quality.

I'd also recommend taking a look at the SNR and correlation time series and seeing how they look. They should be fairly consistent and above 20 dB SNR and above 90% correlation on average. I think if you spend a little time on data quality control to remove outliers, the profiles will look much better overall.

P.J.

Hi P.J.,

Thanks for the clarification on the transformation matrix.  With that correction, I obtained the attached plot, which shows all four beams maxing out at about 0.216 m/s, the max you mentioned for our velocity range setting.  This would seem to apply that the data are severely phase-wrapped, correct?

I'm glad you're familiar with the OSL - it is a nice facility.  We collected most of our data at bankfull, so the flow was quite turbulent and the velocity range was almost certainly too low (0.285 m^3/s with a cross-sectional area of about 0.4 m^2 gives a mean velocity right at the upper threshold for the 0.3 m/s velocity range).  I wish I'd known about that sooner, but can't re-collect the data now.  

I did a quick check on the correlation and SNR and for the one data record I inspected, only 4% of the time series had correlation less than 0.75, but 23% was less than 0.9.  For the SNR, only 2% were less than 25.  Would you recommend just discarding and interpolating over points with SNR < 20 and correlation < 0.9?  A plot of the correlation time series is attached as well.

Should I attempt the phase unwrapping correction prior to the SNR/correlation filtering, or what would you suggest?

Thanks again for all your help,

Carl

Previously P.J. Rusello wrote:

Hi Carl,

The transformation matrix for the Vectrino II range cells is stored in Config.ProbeCalibration_calibrationMatrix, one row per each cell. To get the calibration matrix in a more usable form for transforms I usually do this

 

T = reshape( Config.ProbeCalibration_calibrationMatrix( cell, : ), 4, 4 )'  <---- note the transpose

 

and to convert from XYZ to beam I usually do inv( T ) * [ x; y; z1; z2 ]

The matrix you are using is I think for the non-profiling versions of the Vectrino II. The zero entries are supposed to be there for that one because the four receivers form two independent sub-systems. 

You can actually usually spot phase wrapping without needing to transform to beam coordinates (that just makes the correction much simpler) by looking for discontinuities in the time series. Even with the wrong transformation matrix, the time series you provided suggest measurement quality needs some improvement.

In the post above you mentioned the velocity range was 0.3 m/s, which for the ping algorithm used yields a horizontal range of about 0.75 m/s and vertical range of 0.216 m/s. Do those sound reasonable for the flow you were working in? I've seen photos of this stream facility before, it is quite cool, but I suspect it actually has pretty high turbulence so the velocity range might have needed a little more tuning to produce the best quality.

I'd also recommend taking a look at the SNR and correlation time series and seeing how they look. They should be fairly consistent and above 20 dB SNR and above 90% correlation on average. I think if you spend a little time on data quality control to remove outliers, the profiles will look much better overall.

P.J.

 

Based on the beam velocity plot, the simple solution is to collect the data again. But, that's not really an option for you. Your correlation and SNR numbers sound reasonable and the data is likely okay in terms of quality. At this point though, screening on quality isn't going to help you and the beam velocities need to be unwrapped to make any sense of this data.

It's a little hard to tell from your plot, but you might be able to get away with a simple sign filter to identify the wrapped velocities. If that's the case, then it should be relatively easy to recover your data.

As an example, for the dark blue line in the plot (I can't quite tell which beam this is), it looks like the mean is negative and it's wrapping to positive values. In Matlab you could use the function sign() to identify the positive value indices (or something like positiveIndices = blueBeam > 0) and then unwrap those so everything is negative. The other beams would be similar, with the needed sign changing depending on how they were oriented to the flow.

I'm a little worried about the bluish-green beam (I think beam 4) as it seems to be swapping sign quite a bit, while the others look to be less erratic. It might be the most troublesome, but again it's a little hard to tell because of the way this is plotted.

P.J.

Thanks, P.J.  Trust me, I wish I could go back, choose a more appropriate velocity range setting, and make these measurements over again, but that's just not possible in this case.  Fortunately, the fundamental data quality appear to be OK, and your suggestions for the phase unwrapping seem reasonable, so I'm not going to give up on the data set.  I can tell that this is going to be a fairly serious undertaking, though, and I have nearly 500 individual records to process, so it's going to take some time.  I need to prepare for a conference next week (AGU) and put together a poster, so that has to be my priority now, but please check in on this thread periodically and I'll see what I can do with the data set as soon as I have a bit more time to focus on making these corrections.

Thanks again, you've been very helpful.

Carl

Previously P.J. Rusello wrote:

Based on the beam velocity plot, the simple solution is to collect the data again. But, that's not really an option for you. Your correlation and SNR numbers sound reasonable and the data is likely okay in terms of quality. At this point though, screening on quality isn't going to help you and the beam velocities need to be unwrapped to make any sense of this data.

It's a little hard to tell from your plot, but you might be able to get away with a simple sign filter to identify the wrapped velocities. If that's the case, then it should be relatively easy to recover your data.

As an example, for the dark blue line in the plot (I can't quite tell which beam this is), it looks like the mean is negative and it's wrapping to positive values. In Matlab you could use the function sign() to identify the positive value indices (or something like positiveIndices = blueBeam > 0) and then unwrap those so everything is negative. The other beams would be similar, with the needed sign changing depending on how they were oriented to the flow.

I'm a little worried about the bluish-green beam (I think beam 4) as it seems to be swapping sign quite a bit, while the others look to be less erratic. It might be the most troublesome, but again it's a little hard to tell because of the way this is plotted.

P.J.

The Matlab config information doesn't quite capture all of the configuration settings and how the ping interval was selected is one of them (the information is still in the raw data file, though).   Do you have any more than one velocity header record in your data?  That would be another way of telling if the dynamic adaptive mode was selected.  The bug is pretty obvious in that you get a discrete change in the velocity that corresponds with the time at which a velocity header is received.

   Given your description of the environment, phase wrapping is still an option, so it's definitely worth while exploring.  The actual velocity range configured for your data was 0.743 m/s (even though the nominal range is 0.3m/s),  so that gives you a little more lee way in terms of the phase wraps.

      Robert.

Previously Carl J. Legleiter wrote:

Hi Robert,

Thanks for your thoughts on this data set.  The measurements were made with a Vectrino II at the Saint Anthony Falls Outdoor Stream Lab facility, a small artificial but fairly natural channel with a real sediment bed and no flume side walls or anything like that.  The depths were up to about 0.4 m with a width of ~1.5 m.  We set the velocity range to 0.3 m, which might have been the fatal flaw.  I was a visitor at this lab and just used the instrument as it was configured, so I'm not certain about the dynamic adaptive mode or other settings I am attaching the Config structure array from one of our MATLAB export files, so hopefully that has the information you need.  Please let me know if the actual data would be useful, but I'm afraid the attachment might be too large.

Thanks again, and I'll appreciate any advice you have to offer.

Carl

 

Previously Robert Craig wrote:

Hi Carl:

   I had another look at the data and it does look quite odd when I think about it.  Discrete jumps in velocities are usually a sign of phase wrapping, but if the flow hasn't changed in between profiles, then that would seem unlikely.   What sort of environment was this collected in?  Tuning things to work in a flume (for example) can be quite tricky because of the number of echoes generated in a small volume. What sort of configuration settings were used?  I've also been trying to track down a problem with the dynamic adpative mode of operation and was wondering if you had enabled this setting. 

     Thanks!

    Robert

 

Hi Robert,

Thanks for your thoughts on this data set.  To clarify, I made the measurements as a series of separate data files from different positions and depths within the experimental channel, so each file has its own velocity header I believe.  Are you implying that there might be more than one header per file?  I am including the VelocityHeader fields from one of the Data structure arrays below, and there is a field indicating that the adaptive status was 0 in this case - is that what you were thinking of?  I have the .ntk raw files as well but they are too large to attach here (~3500 KB) - is there another means of sending you a file?  In any case, here's the info from that Data structure:

Thanks again,

Carl

 VelocityHeader_HostTime: 1.3179e+009

              VelocityHeader_CorBeam1: [1x31 single]

              VelocityHeader_CorBeam2: [1x31 single]

              VelocityHeader_CorBeam3: [1x31 single]

              VelocityHeader_CorBeam4: [1x31 single]

              VelocityHeader_AmpBeam1: [1x31 single]

              VelocityHeader_AmpBeam2: [1x31 single]

              VelocityHeader_AmpBeam3: [1x31 single]

              VelocityHeader_AmpBeam4: [1x31 single]

             VelocityHeader_TimeStamp: 0.0296

                VelocityHeader_Status: 4

           VelocityHeader_Temperature: 17.0800

          VelocityHeader_SpeedOfSound: 1473

     VelocityHeader_FirstPingInterval: 170

    VelocityHeader_SecondPingInterval: 170

    VelocityHeader_HorizontalVelRange: 0.7640

      VelocityHeader_VerticalVelRange: 0.2220

        VelocityHeader_AdaptiveStatus: 0

                 VelocityHeader_Range: [1x31 single]

           VelocityHeader_firstRecord: 1.3179e+009

             VelocityHeader_startDate: 'Oct 6, 2011 9:18:06.15 AM'

Previously Robert Craig wrote:

The Matlab config information doesn't quite capture all of the configuration settings and how the ping interval was selected is one of them (the information is still in the raw data file, though).   Do you have any more than one velocity header record in your data?  That would be another way of telling if the dynamic adaptive mode was selected.  The bug is pretty obvious in that you get a discrete change in the velocity that corresponds with the time at which a velocity header is received.

   Given your description of the environment, phase wrapping is still an option, so it's definitely worth while exploring.  The actual velocity range configured for your data was 0.743 m/s (even though the nominal range is 0.3m/s),  so that gives you a little more lee way in terms of the phase wraps.

      Robert.

 

 

Previously Carl J. Legleiter wrote:

Hi Robert,

Thanks for your thoughts on this data set.  The measurements were made with a Vectrino II at the Saint Anthony Falls Outdoor Stream Lab facility, a small artificial but fairly natural channel with a real sediment bed and no flume side walls or anything like that.  The depths were up to about 0.4 m with a width of ~1.5 m.  We set the velocity range to 0.3 m, which might have been the fatal flaw.  I was a visitor at this lab and just used the instrument as it was configured, so I'm not certain about the dynamic adaptive mode or other settings I am attaching the Config structure array from one of our MATLAB export files, so hopefully that has the information you need.  Please let me know if the actual data would be useful, but I'm afraid the attachment might be too large.

Thanks again, and I'll appreciate any advice you have to offer.

Carl

 

Previously Robert Craig wrote:

Hi Carl:

   I had another look at the data and it does look quite odd when I think about it.  Discrete jumps in velocities are usually a sign of phase wrapping, but if the flow hasn't changed in between profiles, then that would seem unlikely.   What sort of environment was this collected in?  Tuning things to work in a flume (for example) can be quite tricky because of the number of echoes generated in a small volume. What sort of configuration settings were used?  I've also been trying to track down a problem with the dynamic adpative mode of operation and was wondering if you had enabled this setting. 

     Thanks!

    Robert

 

 

Hi Carl:

  It looks like you're good.  There is always one velocity header record generated at the start of data collection and subsequent ones will be generated if the ping timing is changed (which would only happen if the dynamic adaptive mode was selected).  Given that there's only one record present, then the bug which would cause a velocity shift hasn't occured.

An adaptive status of 0 means that there was no error in the adaptive processing algorithm (but that's always there regardless of whether or not adaptive intervals is selected :>. 

This also shows that the ping intervals selected for this data file gave you an ambiguity velocity of 0.746 m/s for your horizontal velocity.

    Robert.

Previously Carl J. Legleiter wrote:

Hi Robert,

Thanks for your thoughts on this data set.  To clarify, I made the measurements as a series of separate data files from different positions and depths within the experimental channel, so each file has its own velocity header I believe.  Are you implying that there might be more than one header per file?  I am including the VelocityHeader fields from one of the Data structure arrays below, and there is a field indicating that the adaptive status was 0 in this case - is that what you were thinking of?  I have the .ntk raw files as well but they are too large to attach here (~3500 KB) - is there another means of sending you a file?  In any case, here's the info from that Data structure:

Thanks again,

 

Carl

 

 VelocityHeader_HostTime: 1.3179e+009

              VelocityHeader_CorBeam1: [1x31 single]

              VelocityHeader_CorBeam2: [1x31 single]

              VelocityHeader_CorBeam3: [1x31 single]

              VelocityHeader_CorBeam4: [1x31 single]

              VelocityHeader_AmpBeam1: [1x31 single]

              VelocityHeader_AmpBeam2: [1x31 single]

              VelocityHeader_AmpBeam3: [1x31 single]

              VelocityHeader_AmpBeam4: [1x31 single]

             VelocityHeader_TimeStamp: 0.0296

                VelocityHeader_Status: 4

           VelocityHeader_Temperature: 17.0800

          VelocityHeader_SpeedOfSound: 1473

     VelocityHeader_FirstPingInterval: 170

    VelocityHeader_SecondPingInterval: 170

    VelocityHeader_HorizontalVelRange: 0.7640

      VelocityHeader_VerticalVelRange: 0.2220

        VelocityHeader_AdaptiveStatus: 0

                 VelocityHeader_Range: [1x31 single]

           VelocityHeader_firstRecord: 1.3179e+009

             VelocityHeader_startDate: 'Oct 6, 2011 9:18:06.15 AM'

 

 

 

Previously Robert Craig wrote:

The Matlab config information doesn't quite capture all of the configuration settings and how the ping interval was selected is one of them (the information is still in the raw data file, though).   Do you have any more than one velocity header record in your data?  That would be another way of telling if the dynamic adaptive mode was selected.  The bug is pretty obvious in that you get a discrete change in the velocity that corresponds with the time at which a velocity header is received.

   Given your description of the environment, phase wrapping is still an option, so it's definitely worth while exploring.  The actual velocity range configured for your data was 0.743 m/s (even though the nominal range is 0.3m/s),  so that gives you a little more lee way in terms of the phase wraps.

      Robert.

 

 

Previously Carl J. Legleiter wrote:

Hi Robert,

Thanks for your thoughts on this data set.  The measurements were made with a Vectrino II at the Saint Anthony Falls Outdoor Stream Lab facility, a small artificial but fairly natural channel with a real sediment bed and no flume side walls or anything like that.  The depths were up to about 0.4 m with a width of ~1.5 m.  We set the velocity range to 0.3 m, which might have been the fatal flaw.  I was a visitor at this lab and just used the instrument as it was configured, so I'm not certain about the dynamic adaptive mode or other settings I am attaching the Config structure array from one of our MATLAB export files, so hopefully that has the information you need.  Please let me know if the actual data would be useful, but I'm afraid the attachment might be too large.

Thanks again, and I'll appreciate any advice you have to offer.

Carl

 

Previously Robert Craig wrote:

Hi Carl:

   I had another look at the data and it does look quite odd when I think about it.  Discrete jumps in velocities are usually a sign of phase wrapping, but if the flow hasn't changed in between profiles, then that would seem unlikely.   What sort of environment was this collected in?  Tuning things to work in a flume (for example) can be quite tricky because of the number of echoes generated in a small volume. What sort of configuration settings were used?  I've also been trying to track down a problem with the dynamic adpative mode of operation and was wondering if you had enabled this setting. 

     Thanks!

    Robert