Hydrographic Data Processing PDF Winter Semester 24/25

Hydrographic Data Processing Winter Semester 24/25 Ellen Heffner, M.Sc. & Annika L. Walter, M.Sc. Semester schedule Week HDP Topic Date/Time Room DoPaWD Topic 1 / 1...

Hydrographic Data Processing Winter Semester 24/25 Ellen Heffner, M.Sc. & Annika L. Walter, M.Sc. Semester schedule Week HDP Topic Date/Time Room DoPaWD Topic 1 / 17.10. @ 14:15 Introduction, history, basics 2 General & SBES & Sound Velocity 24.10. @ 14:15 3.103 SBES: principles 3 / (holiday) 31.10. MBES: principles 4 Individual Computer Time with Tutor 07.11. @ 16:15 / 5 MBES Processing I & Calibration 14.11. @ 14:15 3.103 / 6 Scientific Writing Seminar 21.11. @ 14:15 3.103 MBES: system configuration 7 / 28.11. / 8 MBES Processing II & Quality Control 05.12. @ 14:15 3.103 Error budget and IHO standards 9 Individual Computer Time 12.12. @ 16:15 Tides and chart datum 10 Product creation 19.12. @ 14:15 3.103 Reference Systems: horizontal / vertical 11 Backscatter Processing 09.01. @ 14:15 3.103 Introduction to other sonars 12 Guest lecture Philippe Wischow from BSH 16.01. @ 14:15 3.103 Survey operation / Navigation & Positioning 13 Individual Computer Time 23.01. @ 16:15 System configuration 14 Guest lecture: Thomas Thies and Frank 30.01. @ 14:15 online Numerical exercise Köster from Hamburg Port Authoriy Hydrographic Data Processing WiSe 24/25 2 Agenda Lecture 1 Single beam processing and Sound Velocity Profiles Theory Introduction to data processing Accuracy, precision, uncertainty and standard deviation Single beam processing Sound velocity profiles Bar check Practical Exercise CARIS HIPS & SIPS: SBES Data Processing Excel: SVP Plots Excel: Bar Check Hydrographic Data Processing WiSe 24/25 3 Agenda Lecture 1 Single beam processing and Sound Velocity Profiles Theory Introduction to data processing Accuracy, precision, uncertainty and standard deviation Single beam processing Sound velocity profiles Bar check Practical Exercise CARIS HIPS & SIPS: SBES Data Processing Excel: SVP Plots Excel: Bar Check Hydrographic Data Processing WiSe 24/25 4 Definition Hydrography One possible definition of Hydrography (Schiller, 2012): „Hydrography is a branch of the science of surveying and geoinformation. It investigates the surface waters of the earth and collects the related data and information. Its goal is to expand the knowledge of waters in order to use them responsibly and safely and to protect the habitat. The practical engineering and geoscientific work is divided into three main fields of activity: 1. Surveying of waters and recording of aquatic data; 2. Processing of the data, administering the data in information systems and analyzing the total set of data; 3. Visualizing the waters on charts and in information systems and informing about the waters“ Hydrographic Data Processing WiSe 24/25 5 Object of investigation of hydrography 1. Course of the water’s limit 2. Nature of the adjacent land strip (coastal zone resp. shoreline) 3. Traffic situation on waters 4. Characteristics of waves 5. Water level 6. Height profile of the water’s surface (orthometric height) 7. Individual parameters of the water column (temperature, salinity) 8. Water depths 9. Water quality (particle concentration, radioactivity) 10. Characteristics of currents 11. Nature of the bottom 12. Structure of the deeper soil layers 13. Natural and artificial objects in and on the waters [Schiller, 2012] Hydrographic Data Processing WiSe 24/25 6 Hydrographic Surveying Procedure 1. Survey Specifications From policy decisions, product user reports or requests, national defence needs, … Accuracy requirements from IHO S-44 2. Survey Planning Facts to consider: Exact area, type and scope of survey, platforms, systems, limiting factors (budget, political operational constraints, logistics,..), … Review of existing information: satellite data, topographic maps, nautical charts,… 3. Data Gathering Data requirements: data density, coverage, precision, redundancy … 4. Data Processing Generation of valid data: depth / attitude / sound velocity corrections, merging of positions and depths, … 5. Data Analysis Accuracy of results needs to be quoted to show how good/reliable the data is (error analysis) 6. Data Quality Fitness for the use: the extent to which a data set or result satisfies the requirements 7. Data Production Digitally or analogous: e.g. Nautical Charts [IHO, 2005] Hydrographic Data Processing WiSe 24/25 7 Data Processing Softwares Proprietary commercial software QPS Qinsy (Acquisition and processing) Qimera (Post processing) FMGT (Post processing) FM Midwater (Post processing) CARIS HIPS and SIPS HIPS: Hydrographic Information Processing System SIPS: Sidescan Image Processing System Hypack (Acquisition and processing) BeamworX (Acquisition and processing) …. and many more… Hydrographic Data Processing WiSe 24/25 8 Requirements for data processing software Import of proprietary raw data types Removal of any systematic and accidental errors Apply tide and sound velocity or other corrections so that depths are referenced to an established vertical datum Review of positioning and orientation information to locate depth soundings at their correct geographic coordinates Provide mathematical algorithms for „automatic“ data cleaning (filtering) Give the user a lot of useful views to distinguish between correct or erroneous data Visualization of soundings and processed digital terrain models Possibility to extract information such as XYZ data, contours, track line, information on data quality Product creation Hydrographic Data Processing WiSe 24/25 9 Data Types/ File Formats Position (GNSS antenna) NMEA String Position, Heading Heading NMEA String GNSS antenna 1 (2 GNSS antennas) GNSS antenna 2 Roll, Pitch, Yaw NMEA String (IMU) Manufacturers data Data production Sound Velocity Water depth Data analysis and format e.g. *.vdp , Data processing (SVP/CTD) (Merging) quality assessment (3D model, DTM, …) *.asvp IMU Different formats Tide e.g. *.txt, *csv Multibeam Manufacturers data Measurement format e.g. *.all USBL SVP/ SBP transducer hydrophone MBES Water Depth CTD transducer Time (GNSS anntena) NMEA / PPS time synchronisation Sound velocity Roll, Pitch, Yaw Hydrographic Data Processing WiSe 24/25 10 File Formats XTF XTF (eXtended Triton Format) is a common standard forr sonar data exchange In XTF sensor data is stored in different data packages in a chorological order Some of these packages are: Attitude Packet: motion sensor data Navigation Packet: GNSS data Bathymetry Packet: Bathymetry data Hydrographic Data Processing WiSe 24/25 11 Agenda Lecture 1 Single beam processing and Sound Velocity Profiles Theory Introduction to data processing Accuracy, precision, uncertainty and standard deviation Single beam processing Sound velocity profiles Bar check Practical Exercise CARIS HIPS & SIPS: SBES Data Processing Excel: SVP Plots Excel: Bar Check Hydrographic Data Processing WiSe 24/25 12 Accuracy, Uncertainty and Precision [Villarraga-Gómez, 2016] Hydrographic Data Processing WiSe 24/25 13 Standard deviation Standard deviation measures the dispersion of a dataset relative to its mean 100 1* standard deviation Standard deviation is calculated as the square root of the variance 80 Where: s = standard deviation 60 mean s² = variance 40 The variance measures how far each point in the set is from the mean. 20 1* standard deviation Theoretical (population) Empirical (sample) 0 ∑(𝑋𝑋 − 𝜇𝜇)² ∑(𝑋𝑋 − x̄)² 𝑠𝑠² = Why standard deviation? 𝜎𝜎² = 𝑛𝑛 − 1 𝑁𝑁 Standard deviation is expressed in the same unit of Where: Where: measurement as the data σ² = population variance s² = sample variance With standard deviation statisticians may determine X = each value in the data set X = each value in the data set if the data has a normal curve or other relationship 𝜇𝜇 = mean of all values in the data set x̄ = sample mean of all values in the data set In General: N = number of values in the data set n = number of values in the data set The more spread out the data, the higher is the standard deviation When we do not know the average of our data set, When we do know the average of our data set Outliers have a higher impact then we have to estimate it with x̄ Hydrographic Data Processing WiSe 24/25 14 Standard deviation Confidence Intervals Confidence interval is a range of values we are fairly sure our true value lies in: Normal distribution Standard deviation (σ) # of soundings ∑(𝑋𝑋 − 𝜇𝜇)² σ= 𝑁𝑁 Depth bias -3 -2 -1 0 1 2 3 68 % c.I. -1 σ 1σ Three-sigma-rule (almost all 95 % c.I. -2 σ 2σ values lie within three standard 99.7 % c.I. -3 σ 3σ deviations of the mean) Hydrographic Data Processing WiSe 24/25 15 Standard deviation Measurement Person A Person B 1 5 5 Distribution of the measurements 12 2 4 5 10 3 10 5 4 6 6 8 5 5 5 6 6 5 5 4 7 1 5 2 8 5 5 9 6 5 0 1 2 3 4 5 6 7 8 9 10 10 4 5 Person A Person B Hydrographic Data Processing WiSe 24/25 16 Standard deviation Measurement Person A Person B Measurement Person A Person B 1 5 5 Arithmetic mean 1 5 5 2 4 5 2 4 5 3 10 5 3 10 5 4 6 6 4 6 6 ∑(𝑋𝑋 − 𝜇𝜇)² 5 5 5 𝜎𝜎² = Theoretical Variance 5 5 5 𝑁𝑁 6 5 5 6 5 5 7 1 5 7 1 5 8 5 5 8 5 5 9 6 5 𝜎𝜎 = 𝜎𝜎𝜎 Standard deviation 9 6 5 10 4 5 10 4 5 Mean 5,1 5,1 Std. Deviation 2,12 0,3 Hydrographic Data Processing WiSe 24/25 17 Standard deviation Measurement Person A Person B 1 5 5 Distribution of the measurements 12 2 4 5 10 3 10 5 4 6 6 8 5 5 5 6 6 5 5 4 7 1 5 2 8 5 5 9 6 5 0 1 2 3 4 5 6 7 8 9 10 10 4 5 Person A Person B Mean 5,1 5,1 Std. Deviation 2,12 0,3 Hydrographic Data Processing WiSe 24/25 18 Agenda Lecture 1 Single beam processing and Sound Velocity Profiles Theory Introduction to data processing Accuracy, precision, uncertainty and standard deviation Single beam processing Sound velocity profiles Bar check Practical Exercise CARIS HIPS & SIPS: SBES Data Processing Excel: SVP Plots Excel: Bar Check Hydrographic Data Processing WiSe 24/25 19 Single beam echo sounder Basics Beam width (opening angle): approx. 10° Frequency: 10 - 500 kHz Vertical resolution δz 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 = 𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 𝑥𝑥 𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 c: sound velocity 𝑊𝑊𝑊𝑊𝑊𝑊𝑊𝑊𝑊𝑊 𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉 𝑥𝑥 𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 T: pulse duration 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷ℎ = 2 Horizontal resolution δx H: water depth ϴ: beamwidth Hydrographic Data Processing WiSe 24/25 20 Single beam echo sounder Dual Frequency SBES with two frequencies High and low frequency E.g. 32 kHz & 200 kHz or 15 kHz & 100 kHz Water depth Frequency Characteristics [IHO, 2010] < 100 m ≥ 200 kHz Good reflection of soft ground (e.g. silt, mud); low penetration; few disturbances in the water column (screw water) < 1500 m 50 kHz – 100 kHz: Mean depth of penetration 200 kHz > 1500 m 12 kHz – 12 kHz: Good reflection of solid ground, nautical depth, 50 kHz disturbances in the water column (air bubbles, screws water, fish,...) Sediment < 8 kHz A higher frequency means a lower range and a lower penetration but a higher resolution. Hydrographic Data Processing WiSe 24/25 21 Single beam echo sounder Wide beam vs. Narrow beam Narrow beam typically 2 ° - 5 ° opening angle used for high resolution mapping Beam stabilisation needed Wide Beam typically > 10 ° opening angle Used for hazard detection Usually not stabilised Hydrographic Data Processing WiSe 24/25 22 Single beam echo sounder Applications Depth measurements (shallow waters to full ocean depth) Fishing Seafloor Classification Navigation Subbottom Profiler Lowrance Fish Finder Kongsberg EA440 Different Kongsberg transducer Hydrographic Data Processing WiSe 24/25 23 Single beam echo sounder Data Processing with CARIS HIPS & SIPS CARIS HIPS&SIPS is designed to convert and to process bathymetric and side-scan sonar data from a lot of raw data standard formats HIPS provide different data editors to check all data sensors manually and automatically With vessel configuration files HIPS&SIPS is able to combine all sensor raw data into XYZ coordinate datasets These datasets have to be validated by area based editors which are provided by HIPS HIPS is able to export the final dataset into several data formats, which can be used in [Teledyne CARIS, 2022] visualization tools etc. Hydrographic Data Processing WiSe 24/25 24 Single beam echo sounder Data Processing with CARIS HIPS & SIPS HIPS data structure During the import process all raw data will be converted into a Project / Vessel / Day / Line structure A HIPS Vessel File, i.e. a configuration file for a survey vessel, contains all interesting information (shape of the vessel, type of sensor, accuracy, the position in reference to a vessel coordinate system) about the used equipment: Multibeam echo sounder Navigation sensor (GNSS) Attitude / Motion sensor Heading sensor (Gyro) The vessel file is divided into different sections, each section for one of the sensors Due to time-tagging of every entry it is possible to have different configurations for different time periods The HIPS vessel information is stored in a XML file Hydrographic Data Processing WiSe 24/25 25 Single beam echo sounder Data Processing with CARIS HIPS & SIPS Pre-Cleaning Beam to Beam Slope: > 75 ° Moving Average: Window 3 Points / Threshold 5 Sigma Every beam which creates slope angles greater than 75 ° to both adjacent beams will be filtered Every beam that falls outside a 5-sigma threshold within a certain window will be filtered Hydrographic Data Processing WiSe 24/25 26 Single beam echo sounder Data Processing with CARIS HIPS & SIPS Validation of Sensor Data  Navigation and Attitude  More Info during lecture 2 Validation of Single Beam Data The Singlebeam Editor is used to validate and to clean bathymetric data gathered by a single beam echo sounder Singlebeam echo sounders mostly creates a analog or a digital picture of all received echoes (echogram) Additionally all of these echo sounders export depths (calculated from the first received echo) In some cases the first echo is not the true bottom echo because of bubbles / fish in the water column The operators task is now to compare the echo picture with all digitized depth values  Digital/Analoque Comparison To start the Singlebeam Editor following command from the menu bar can be used: Tools / Single Beam Editor [Thies, 2014] Hydrographic Data Processing WiSe 24/25 27 Single beam echo sounder Data Processing with CARIS HIPS & SIPS Validation of Single Beam Data Orange: Primary Data (high frequency soundings) Blue: Secondary Data (low frequency soundings) Grey: Rejected soundings [Thies, 2014] Hydrographic Data Processing WiSe 24/25 28 Single beam echo sounder Data Processing with CARIS HIPS & SIPS Merging of RAW data into 3D coordinates At this stage the data of every sensor (GPS, Motion Sensor, Tide, Echo sounder) is isolated from the others In the next step all raw data will be merged together A key feature of this process is the HIPS Vessel File (HVF) The HVF contains all information about the sensors and their spatial relationship to each other During the Merge-process: for each depth record the position of the center of the beam is calculated and interpolated based on motion and gyro data for each depth record the tide is interpolated and applied in order to reach a vertical datum processed depth files will be created for every line Creating Surface A Gridded Surface with the Track Lines coloured by sounding depth will be created Can be exported as GeoTiff and used for further analysis Hydrographic Data Processing WiSe 24/25 29 Agenda Lecture 1 Single beam processing and Sound Velocity Profiles Theory Introduction to data processing Accuracy, precision, uncertainty and standard deviation Single beam processing Sound velocity profiles Bar check Practical Exercise CARIS HIPS & SIPS: SBES Data Processing Excel: SVP Plots Excel: Bar Check Hydrographic Data Processing WiSe 24/25 30 Sound Velocity Profile Basics The changes of the sound velocity are linearly dependent on the changes in temperature, salinity and depth. Temperature: 2,8 m/s per 1 °C Salinity: 1,14 m/s per 1 ppt Depth: 0,016 m/s per 1 m Default sound velocity? 1500 m/s Hydrographic Data Processing WiSe 24/25 31 Sound Velocity Profile Why is it necessary? To calculate the correct depth as the depth determination depends on the sound velocity To correct for refraction errors especially at multi beam echo sounders’ tilted beams Hydrographic Data Processing WiSe 24/25 32 Sound Velocity Profile Sensors Sound Velocity Profiler (SVP) Pressure sensor Transducer and reflector at a fixed distance Conductivity, Temperature, Depth (CTD) determination of conductivity => salinity temperature depth Settings control Known target depths (e.g. locks) Disadvantage: usually not in the measuring area Barcheck for Hydrography Hydrographic Data Processing WiSe 24/25 33 Agenda Lecture 1 Single beam processing and Sound Velocity Profiles Theory Introduction to data processing Accuracy, precision, uncertainty and standard deviation Single beam processing Sound velocity profiles Bar check Practical Exercise CARIS HIPS & SIPS: SBES Data Processing Excel: SVP Plots Excel: Bar Check Hydrographic Data Processing WiSe 24/25 34 Bar check Principle Lowering a bar or plate underneath the transducer at several depths (e.g. every two meters down to 20 m – 30 m) Marking of distances on holding rope Note the distance between water surface and bar check plate  Actual Depth Note the recording distance from the transducer face to bar check plate  Measured Depth Results Water sound velocity Transducer immersion depth Transducer depth Measured depth Actual depth [IHO, 2005] Hydrographic Data Processing WiSe 24/25 35 Bar check Calculations Determine the speed of sound and the transducer immersion depth 𝑚𝑚 Default setting: 𝑣𝑣 = 1500 and transducer immersion depth = 0 m 𝑠𝑠 S_actual [m] 1,0 1,5 2,0 2,5 3,0 3,5 S_measured [m] 0,35 0,86 1,37 1,84 2,3 2,78 Form the equation: y = ax+b Where a is the scale error, i.e. the difference from the true Measured depth water sound velocity and b corresponds to the zero error, i.e. Data points the correction for the transducer immersion depth Regression line Equation: y = 0,97x – 0,6 𝑚𝑚 1500 𝑠𝑠 𝑚𝑚 Therefore: 𝑣𝑣 = = 1543 0,97 𝑠𝑠 Transducer immersion depth: | 0 m – 0,6 | = 0,6 m Actual depth Hydrographic Data Processing WiSe 23/24 36 Agenda Lecture 1 Single beam processing and Sound Velocity Profiles Theory Introduction to data processing Accuracy, precision, uncertainty and standard deviation Single beam processing Sound velocity profiles Bar check Practical Exercise CARIS HIPS & SIPS: SBES Data Processing Excel: SVP Plots Excel: Bar Check Hydrographic Data Processing WiSe 24/25 37 Referenzes International Hydrographic Organization. (2005): Manual on Hydrography. Monaco: International Hydrographic Bureau. Retrieved from https://iho.int/uploads/user/pubs/cb/c-13/english/C-13_Chapter_1_and_contents.pdf, last accessed 09.10.2023. Schiller, L. & Böder, V. & Schenke, H. (2012): A comprehensive definition and systematic subdivision of hydrography. 10.3990/2.266. Teledyne CARIS. (2022): HIPS & SIPS. Retrieved from https://www.teledynecaris.com/en/products/hips-and-sips/, last accessed on 24.10.2022. Thies, Thmoas. (2014): Lecture on Hydrographic Data Processing. HafenCity University Hamburg. Villarraga-Gómez, Herminso. (2016): X-ray Computed Tomography for Dimensional Measurements. Hydrographic Data Processing WiSe 24/25 38  Thank you Ellen Heffner [email protected] HafenCity Universität Hamburg Henning-Voscherau-Platz 1 20457 Hamburg