3. Watching a solar eclipse using an OOI moored echosounder#

Jupyter notebook accompanying the manuscript:

Echopype: A Python library for interoperable and scalable processing of ocean sonar data for biological information
Authors: Wu-Jung Lee, Emilio Mayorga, Landung Setiawan, Kavin Nguyen, Imran Majeed, Valentina Staneva

3.1. Introduction#

3.1.1. Goals#

Illustrate a common workflow for echosounder data conversion, calibration and use. This workflow leverages the standardization applied by echopype. and the power, ease of use and familiarity of libraries in the scientific Python ecosystem.
Demonstrate the ease to interoperate echosounder data with those from a different instrument in a single computing environment. Without echopype, additional wrangling across more than one software systems is needed to achieve the same visualization and comparison.

3.1.2. Description#

This notebook uses EK60 echosounder data from the U.S. Ocean Observatories Initiative (OOI) to illustrate a common workflow for data conversion, combination, calibration and analysis using echopype, as well as the data interoperability it enables. Without echopype, additional wrangling across more than one software systems is needed to achieve the same visualization and comparison.

We will use data from the OOI Oregon Offshore Cabled Shallow Profiler Mooring collected on August 20-21, 2017. This was the day before and of a solar eclipse, during which the reduced sunlight affected the regular diel vertical migration (DVM) patterns of marine life. This change was directly observed using the upward-looking echosounder mounted on this mooring platform that happened to be within the totality zone. The effect of the solar eclipse was clearly seen by aligning and comparing the echosounder observations with solar radiation data collected by the Bulk Meteorology Instrument Package located on the nearby Coastal Endurance Oregon Offshore Surface Mooring, also maintained by the OOI.

The data used are 19 .raw files with a total volume of approximately 1 GB. With echopype functionality, the raw data files hosted on the OOI Raw Data Archive (an HTTP server) are directly parsed and organized into a standardized representation following in the SONAR-netCDF4 v1.0 convention, and stored to the cloud-optimized Zarr format. The individual converted files are later combined into a single entity that can be easily explored and manipulated.

3.1.3. Outline#

Establish connection with the OOI Raw Data Archive and generate list of target EK60 .raw files
Process the archived raw files with echopype: convert and combine into a single quantity (an EchoData object) in a standardized format.
Obtain solar radiation data from an OOI Thredds server.
Plot the echosounder and solar radiation data together to visualize the zooplankton response to a solar eclipse.

3.1.4. Running the notebook#

This notebook can be run with a conda environment created using the conda environment file https://github.com/OSOceanAcoustics/echopype-examples/blob/main/binder/environment.yml. The notebook creates a directory ./exports/ooifiles and save all generated Zarr and netCDF files there.

3.1.5. Warning#

The compute_MVBS step in this notebook is not efficient for lazy-loaded data with echopype version 0.6.3. We plan to address this issue soon.

3.1.6. Note#

We encourage importing echopype as ep for consistency.

from pathlib import Path
import itertools as it
import datetime as dt
from dateutil import parser as dtparser

import fsspec
import xarray as xr
import matplotlib.pyplot as plt
import hvplot.xarray

import echopype as ep

import warnings
warnings.simplefilter("ignore", category=DeprecationWarning)

3.2. Establish connection with the OOI Raw Data Archive and generate list of target EK60 `.raw` files#

Access and inspect the publicly accessible OOI Raw Data Archive (an HTTP server) as if it were a local file system. This will be done through the Python fsspec file system and bytes storage interface. We will use fsspec.filesystem.glob (fs.glob) to generate a list of all EK60 .raw data files in the archive then filter on file names for target dates of interest.

fs = fsspec.filesystem('https')

ooi_raw_url = (
    "https://rawdata.oceanobservatories.org/files/"
    "CE04OSPS/PC01B/ZPLSCB102_10.33.10.143/2017/08"
)

Now let’s specify the range of dates we will be pulling data from. Note that the data filenames contain the time information but were recorded at UTC time.

def in_range(raw_file: str, start: dt.datetime, end: dt.datetime) -> bool:
    """Check if file url is in datetime range"""
    file_name = Path(raw_file).name
    file_datetime = dtparser.parse(file_name, fuzzy=True)
    return file_datetime >= start and file_datetime <= end

start_datetime = dt.datetime(2017, 8, 21, 7, 0)
end_datetime = dt.datetime(2017, 8, 22, 7, 0)

On the OOI Raw Data Archive, the monthly folder is further split to daily folders, so we can simply grab data from the desired days.

desired_day_urls = [f"{ooi_raw_url}/{day}" for day in range(start_datetime.day, end_datetime.day + 1)]

desired_day_urls

['https://rawdata.oceanobservatories.org/files/CE04OSPS/PC01B/ZPLSCB102_10.33.10.143/2017/08/21',
 'https://rawdata.oceanobservatories.org/files/CE04OSPS/PC01B/ZPLSCB102_10.33.10.143/2017/08/22']

Grab all raw files within daily folders by using the filesytem glob, just like the Linux glob.

all_raw_file_urls = it.chain.from_iterable([fs.glob(f"{day_url}/*.raw") for day_url in desired_day_urls])

desired_raw_file_urls = list(filter(
    lambda raw_file: in_range(
        raw_file, 
        start_datetime-dt.timedelta(hours=3),  # 3 hour buffer to select files
        end_datetime+dt.timedelta(hours=3)
    ), 
    all_raw_file_urls
))


print(f"There are {len(desired_raw_file_urls)} raw files within the specified datetime range.")

There are 19 raw files within the specified datetime range.

3.4. Obtain solar radiation data from an OOI THREDDS server#

Now we have the sonar data ready, the next step is to pull solar radiation data collected by a nearby surface mooring also maintained by the OOI. The Bulk Meteorology Instrument Package is located on the Coastal Endurance Oregon Offshore Surface Mooring.

Note: an earlier version of this notebook used the same dataset but pulled from the National Data Buoy Center (NDBC). We thank the Rutgers OOI Data Lab for pointing out the direct data source in one of the data nuggets.

metbk_url = (
    "http://thredds.dataexplorer.oceanobservatories.org/thredds/dodsC/ooigoldcopy/public/"
    "CE04OSSM-SBD11-06-METBKA000-recovered_host-metbk_a_dcl_instrument_recovered/"
    "deployment0004_CE04OSSM-SBD11-06-METBKA000-recovered_host-metbk_a_dcl_instrument_recovered_20170421T022518.003000-20171013T154805.602000.nc#fillmismatch"
)

metbk_ds = (
    xr.open_dataset(metbk_url)
    .swap_dims({'obs': 'time'})
    .drop('obs')
    .sel(time=slice(start_datetime, end_datetime))[['shortwave_irradiance']]
)
metbk_ds.time.attrs.update({'long_name': 'Time', 'units': 'UTC'})

metbk_ds

<xarray.Dataset>
Dimensions:               (time: 1441)
Coordinates:
  * time                  (time) datetime64[ns] 2017-08-21T07:00:08.232999936...
Data variables:
    shortwave_irradiance  (time) float32 2.2 2.3 2.2 2.2 2.2 ... 2.4 2.3 2.3 2.4
Attributes: (12/73)
    node:                               SBD11
    comment:                            
    publisher_email:                    
    sourceUrl:                          http://oceanobservatories.org/
    collection_method:                  recovered_host
    stream:                             metbk_a_dcl_instrument_recovered
    ...                                 ...
    geospatial_vertical_positive:       down
    lat:                                44.36555
    lon:                                -124.9407
    DODS.strlen:                        14
    DODS.dimName:                       string14
    DODS_EXTRA.Unlimited_Dimension:     obs

xarray.Dataset

Dimensions:
- time: 1441

Coordinates: (1)

time

(time)

datetime64[ns]

2017-08-21T07:00:08.232999936 .....

standard_name :: time
long_name :: Time
axis :: T
_ChunkSizes :: 10000
units :: UTC

array(['2017-08-21T07:00:08.232999936', '2017-08-21T07:01:12.539999744',
       '2017-08-21T07:02:16.888000000', ..., '2017-08-22T06:58:15.237000192',
       '2017-08-22T06:59:19.614000128', '2017-08-22T06:59:53.616999936'],
      dtype='datetime64[ns]')

Data variables: (1)
- shortwave_irradiance
  (time)
  float32
  ...
  long_name :
  Downwelling Shortwave Irradiance
  precision :
  1
  coordinates :
  time lat lon
  data_product_identifier :
  SHRTIRR_L1
  standard_name :
  downwelling_shortwave_flux_in_air
  units :
  W m-2
  _ChunkSizes :
  10000
```
array([2.2, 2.3, 2.2, ..., 2.3, 2.3, 2.4], dtype=float32)
```
Attributes: (73)
node :
SBD11
comment :
publisher_email :
sourceUrl :
http://oceanobservatories.org/
collection_method :
recovered_host
stream :
metbk_a_dcl_instrument_recovered
featureType :
point
creator_email :
publisher_name :
Ocean Observatories Initiative
date_modified :
2022-09-14T02:11:23.614696
keywords :
cdm_data_type :
Point
references :
More information can be found at http://oceanobservatories.org/
Metadata_Conventions :
Unidata Dataset Discovery v1.0
date_created :
2022-09-14T02:11:23.614694
id :
CE04OSSM-SBD11-06-METBKA000-recovered_host-metbk_a_dcl_instrument_recovered
requestUUID :
d1af8c41-d009-4475-a862-29bd4a3d5e16
contributor_role :
summary :
Dataset Generated by Stream Engine from Ocean Observatories Initiative
keywords_vocabulary :
institution :
Ocean Observatories Initiative
naming_authority :
org.oceanobservatories
feature_Type :
point
infoUrl :
http://oceanobservatories.org/
license :
contributor_name :
uuid :
d1af8c41-d009-4475-a862-29bd4a3d5e16
creator_name :
Ocean Observatories Initiative
title :
Data produced by Stream Engine version 1.20.1 for CE04OSSM-SBD11-06-METBKA000-recovered_host-metbk_a_dcl_instrument_recovered
sensor :
06-METBKA000
standard_name_vocabulary :
NetCDF Climate and Forecast (CF) Metadata Convention Standard Name Table 29
acknowledgement :
Conventions :
CF-1.6
project :
Ocean Observatories Initiative
source :
CE04OSSM-SBD11-06-METBKA000-recovered_host-metbk_a_dcl_instrument_recovered
publisher_url :
http://oceanobservatories.org/
creator_url :
http://oceanobservatories.org/
nodc_template_version :
NODC_NetCDF_TimeSeries_Orthogonal_Template_v1.1
subsite :
CE04OSSM
processing_level :
L2
history :
2022-09-14T02:11:23.614658 generated from Stream Engine
Manufacturer :
Star Engineering
ModelNumber :
ASIMET
SerialNumber :
LGR032
Description :
Bulk Meteorology Instrument Package: METBK LGR Module
FirmwareVersion :
Not specified.
SoftwareVersion :
Not specified.
AssetUniqueID :
CGINS-METLGR-00032
Notes :
Not specified.
Owner :
Oregon State University
RemoteResources :
[]
ShelfLifeExpirationDate :
Not specified.
Mobile :
False
AssetManagementRecordLastModified :
2022-09-02T13:10:19.313000
time_coverage_start :
2017-04-21T02:25:18.003000
time_coverage_end :
2017-10-13T15:48:05.602000
time_coverage_resolution :
P60.00S
geospatial_lat_min :
44.36555
geospatial_lat_max :
44.36555
geospatial_lat_units :
degrees_north
geospatial_lat_resolution :
0.1
geospatial_lon_min :
-124.9407
geospatial_lon_max :
-124.9407
geospatial_lon_units :
degrees_east
geospatial_lon_resolution :
0.1
geospatial_vertical_units :
meters
geospatial_vertical_resolution :
0.1
geospatial_vertical_positive :
down
lat :
44.36555
lon :
-124.9407
DODS.strlen :
14
DODS.dimName :
string14
DODS_EXTRA.Unlimited_Dimension :
obs

3.5. Combine sonar observation with solar radiation measurements#

We can finally put everything together and figure out the impact of the eclipse-driven reduction in sunlight on marine zooplankton!

metbk_plot = metbk_ds.hvplot.line(
    x='time', y='shortwave_irradiance'
).options(width=800, height=120, logy=True, xlim=(start_datetime, end_datetime))

mvbs_plot = ds_MVBS["Sv"].sel(frequency_nominal=200000).hvplot.image(
    x='ping_time', y='depth', 
    color='Sv', rasterize=True, 
    cmap='jet', clim=(-80, -30),
    xlabel='Time (UTC)',
    ylabel='Depth (m)'
).options(width=800, invert_yaxis=True, xlim=(start_datetime, end_datetime))

(metbk_plot + mvbs_plot).cols(1)

Look how the dip at solar radiation reading matches exactly with the upwarding moving “blip” at UTC 17:21, August 22, 2017 (local time 10:22 AM). During the solar eclipse, the animals were fooled by the temporary mask of the sun and thought it’s getting dark as at dusk!

3.6. Package versions#

import datetime
print(f"echopype: {ep.__version__}, xarray: {xr.__version__}, fsspec: {fsspec.__version__}, "
      f"hvplot: {hvplot.__version__}")

print(f"\n{datetime.datetime.utcnow()} +00:00")

echopype: 0.6.3, xarray: 2022.3.0, fsspec: 2022.8.2, hvplot: 0.8.1

2022-10-19 02:00:27.942388 +00:00

Echopype Examples

Watching a solar eclipse using an OOI moored echosounder

Contents

3. Watching a solar eclipse using an OOI moored echosounder#

3.1. Introduction#

3.1.1. Goals#

3.1.2. Description#

3.1.3. Outline#

3.1.4. Running the notebook#

3.1.5. Warning#

3.1.6. Note#

3.2. Establish connection with the OOI Raw Data Archive and generate list of target EK60 `.raw` files#

3.3. Process the archived raw files with `echopype`#

3.3.1. Examine the workflow by processing just one file#

3.3.2. Converting from raw data files to a standardized data format#

3.3.3. Basic echo processing#

3.3.4. Quickly visualize the result#

3.3.5. Convert multiple files and combine into a single `EchoData` object#

3.3.6. Calibrate the combined `EchoData` and visualize the mean Sv#

3.3.7. Visualize MVBS interactively using `hvPlot`#

3.4. Obtain solar radiation data from an OOI THREDDS server#

3.5. Combine sonar observation with solar radiation measurements#

3.6. Package versions#

Echopype Examples

Watching a solar eclipse using an OOI moored echosounder

Contents

3. Watching a solar eclipse using an OOI moored echosounder#

3.1. Introduction#

3.1.1. Goals#

3.1.2. Description#

3.1.3. Outline#

3.1.4. Running the notebook#

3.1.5. Warning#

3.1.6. Note#

3.2. Establish connection with the OOI Raw Data Archive and generate list of target EK60 .raw files#

3.3. Process the archived raw files with echopype#

3.3.1. Examine the workflow by processing just one file#

3.3.2. Converting from raw data files to a standardized data format#

3.3.3. Basic echo processing#

3.3.4. Quickly visualize the result#

3.3.5. Convert multiple files and combine into a single EchoData object#

3.3.6. Calibrate the combined EchoData and visualize the mean Sv#

3.3.7. Visualize MVBS interactively using hvPlot#

3.4. Obtain solar radiation data from an OOI THREDDS server#

3.5. Combine sonar observation with solar radiation measurements#

3.6. Package versions#

3.2. Establish connection with the OOI Raw Data Archive and generate list of target EK60 `.raw` files#

3.3. Process the archived raw files with `echopype`#

3.3.5. Convert multiple files and combine into a single `EchoData` object#

3.3.6. Calibrate the combined `EchoData` and visualize the mean Sv#

3.3.7. Visualize MVBS interactively using `hvPlot`#