API#

Exchange Format#

class cchdo.hydro.exchange.FileType(value)[source]#

An enumeration.

BOTTLE = 'B'#
CTD = 'C'#
class cchdo.hydro.exchange._ExchangeData(single_profile, param_cols, flag_cols, error_cols, param_precisions, error_precisions, comments)[source]#

Dataclass containing exchange data which has been parsed into ndarrays

comments#
error_cols#
error_precisions#
flag_cols#
param_cols#
param_precisions#
set_expected(params, flags, errors)[source]#

Puts fill columns for expected params which are missing

This can occur when there are disjoint columns in CTD files

single_profile#
split_profiles()[source]#

Split into single profile containing _ExchangeData instances

Done by looking at the expocode+station+cast composate keys

property str_lens#

Figure out the length of all the string params

The char size can vary by platform.

class cchdo.hydro.exchange._ExchangeInfo(stamp_slice, comments_slice, ctd_headers_slice, params_idx, units_idx, data_slice, post_data_slice, _raw_lines)[source]#

Low level dataclass containing the parts of an exchange file

property _np_data_block#
_raw_lines#
property comments#

Returns the comments of the exchange file with leading # stripped

comments_slice#
property ctd_headers#

Returns a dict of the CTD headers and their value

ctd_headers_slice#
property data#

Returns the data block of an exchange file as a tuple of strs. One line per entry.

data_slice#
finalize(fill_values=('-999',))[source]#

Parse all the data into ndarrays of the correct dtype and shape

Returns an ExchangeData dataclass

classmethod from_lines(lines, ftype)[source]#

Figure out the line numbers/indicies of the parts of the exchange file

property params#

Returns a list of all parameters in the file (including CTD “headers”)

params_idx#
property post_data#

Returns any post data content as a tuple of strs

post_data_slice#
property stamp#

Returns the filestamp of the exchange file

e.g. “BOTTLE,20210301CCHSIOAMB”

stamp_slice#
property units#

Returns a list of all the units in the file (including CTD “headers”)

Will have the same shape as params

units_idx#
property whp_errors#

Parses the params and units for uncertanty values

returns a dict with a WHPName to column index of errors mapping

property whp_flags#

Parses the params and units for flag values

returns a dict with a WHPName to column index of flags mapping

property whp_params#

Parses the params and units for base parameters

Returns a dict with a WHPName to column index mapping

cchdo.hydro.exchange._bottle_get_errors(params_units, whp_params)[source]#

Given an ordered iterable of param unit pairs and WHPNames known to be in the file, return the index of the column indicies of the errors/uncertanties for the WHPNames.

Some parameters may have uncertanties associated with them, this function finds those columns and pairs them with the correct parameter.

Note

There is no programable way to find the error columns for a given unit (e.g. no common suffix like the flags). This must be done via lookup in the built in database of params.

Parameters
  • params_units (tuple in the form (str, str) or (str, None)) – Paired (e.g. zip()) parameter names and units

  • whp_params (Mapping of WHPName to int) – Mapping of parameters known to be in the file, this is the output of _bottle_get_params()

Returns

Mapping of WHPName to column indicies for the error column

Return type

dict with keys of WHPName and values of int

cchdo.hydro.exchange._bottle_get_flags(params_units, whp_params)[source]#

Given an ordered iterable of param unit pairs and WHPNames known to be in the file, return the index of the column indicies of the flags for the WHPNames.

Exchange files can have status flags for some of the parameters. Flag columns must have no units. Some parameters must not have status flags, these include the spatiotemporal parameters (e.g. lat, lon, but also pressure) and the sample identifying parameters (expocode, station, cast, sample, but not bottle id).

Parameters
  • params_units (tuple in the form (str, str) or (str, None)) – Paired (e.g. zip) parameter names and units

  • whp_params (Mapping of WHPName to int) – Mapping of parameters known to be in the file, this is the output of _bottle_get_params()

Returns

Mapping of WHPName to column indicies for the status flag column

Return type

dict with keys of WHPName and values of int

Raises
cchdo.hydro.exchange._bottle_get_params(params_units)[source]#

Given an ordered iterable of param, unit pairs, return the index of the column in the datafile for known WHP params.

Exchange files have comma separated parameter names on one line, and the corresponding units on the next. This function will search for this name+unit pair in the builtin database of known WHP parameter names and return a mapping of WHPName to column indicies.

It is currently an error for the parameter in a file to not be in the built in database.

This function will ignore uncertainty (error) columns and flag columns, those are parsed by other functions.

Warning

Convert semantically empty units (e.g. empty string, all whitespace) to None before passing into this function

Note

The parameter name database will convert unambiguous aliases to their canonical exchange parameter and unit pair.

Parameters

params_units (tuple in the form (str, str) or (str, None)) – Paired (e.g. zip) parameter names and units

Returns

Mapping of WHPName to column indicies

Return type

dict with keys of WHPName and values of int

Raises

ExchangeParameterUndefError – if the parameter unit pair cannot be found in the built in database

cchdo.hydro.exchange._combine_dt_ndarray(date_arr, time_arr=None, time_pad=False)[source]#
cchdo.hydro.exchange._ctd_get_header(line, dtype=<class 'str'>)[source]#
cchdo.hydro.exchange._extract_numeric_precisions(data)[source]#

Get the numeric precision of a printed decimal number

cchdo.hydro.exchange._get_fill_locs(arr, fill_values=('-999',))[source]#
cchdo.hydro.exchange._is_valid_exchange_numeric(data)[source]#
cchdo.hydro.exchange._load_raw_exchange(filename_or_obj)[source]#
cchdo.hydro.exchange.add_geometry_var(dataset)[source]#

Adds a CF-1.8 Geometry container variable to the dataset

This allows for compatabiltiy with tools like gdal

cchdo.hydro.exchange.add_profile_type(dataset, ftype)[source]#

Adds a profile_type string variable to the dataset.

This is for ODV compatability

Warning

Currently mixed profile types are not supported

cchdo.hydro.exchange.all_same(ndarr)[source]#

Test if all the values of an ndarray are the same value

cchdo.hydro.exchange.check_flags(dataset, raises=True)[source]#

Check WOCE flag values agaisnt their param and ensure that the param either has a value or is “nan” depedning on the flag definition.

Return a boolean array of invalid locations?

cchdo.hydro.exchange.check_is_subset_shape(a1, a2, strict='disallowed')[source]#

Ensure that the shape of the data in a2 is a subset (or strict subset) of the data shape of a1

For a given set of param, flag, and error arrays you would want to ensure that: * errors are a subset of params (strict is allowed) * params are a subset of flags (strict is allowed)

For string vars, the empty string is considered the “nothing” value. For woce flags, flag 9s should be converted to nans (depending on scheme flag 5 and 1 may not have param values)

Return a boolean array of invalid locations

cchdo.hydro.exchange.check_sorted(dataset)[source]#

Check that the dataset is sorted by the rules in :py:`sort_ds`

cchdo.hydro.exchange.combine_bottle_time(dataset)[source]#

Combine the bottle dates and times if present

Raises if only one is present

cchdo.hydro.exchange.combine_dt(dataset, is_coord=True, date_name=WHPName(whp_name='DATE', whp_unit=None, cf_name=None), time_name=WHPName(whp_name='TIME', whp_unit=None, cf_name=None), time_pad=False)[source]#

Combine the exchange style string variables of date and optinally time into a single variable containing real datetime objects

This will remove the time variable if present, and replace then rename the date variable. Date is replaced/renamed to maintain variable order in the xr.DataSet

cchdo.hydro.exchange.finalize_ancillary_variables(dataset)[source]#

Turn the ancillary variable attr into a space seperated string

It is nice to have the ancillary variable be a list while things are being read into it

cchdo.hydro.exchange.read_exchange(filename_or_obj, fill_values=('-999',))[source]#

Loads the data from filename_or_obj and returns a xr.Dataset with the CCHDO CF/netCDF structure

cchdo.hydro.exchange.set_axis_attrs(dataset)[source]#

Set the CF axis attribute on our axis variables (XYZT)

  • longitude = “X”

  • latitude = “Y”

  • pressure = “Z”, addtionally, positive is down

  • time = “T”

cchdo.hydro.exchange.set_coordinate_encoding_fill(dataset)[source]#

Sets the _FillValue encoidng to None for 1D coordinate vars

cchdo.hydro.exchange.sort_ds(dataset)[source]#

Sorts the data values in the dataset

Ensures that profiles are in the following order: * Earlier before later (time will increase) * Southerly before northerly (latitude will increase) * Westerly before easterly (longitude will increase)

The two xy sorts are esentially tie breakers for when we are missing “time”

Inside profiles: * Shallower before Deeper (pressure will increase)

exceptions#

exception cchdo.hydro.exchange.exceptions.ExchangeBOMError[source]#

Error raised when the exchange file has a byte order mark.

exception cchdo.hydro.exchange.exceptions.ExchangeDataColumnAlignmentError[source]#

Error raised when the number of columns in a data line does not match the expected number of columns based on the parameter/unit lines.

exception cchdo.hydro.exchange.exceptions.ExchangeDataError[source]#

Base exception for errors which occur when parsing the data porition of an exchange file.

exception cchdo.hydro.exchange.exceptions.ExchangeDataFlagPairError[source]#

There is a mismatch between what the flag value expects, and the fill/data value.

Examples:

  • something with a flag of 9 has a non fill value

  • something with a flag of 2 as a fill value instead of data

exception cchdo.hydro.exchange.exceptions.ExchangeDataInconsistentCoordinateError[source]#

Error raised if the reported latitude, longitude, and date (and time) vary for a single profile.

A “profile” in an exchange file is a grouping of data rows which all have the same EXPOCODE, STNNBR, and CASTNO. The SAMPNO/CTDPRS is allowed/requried to vary for a single profile and is what identifies samples within one profile.

exception cchdo.hydro.exchange.exceptions.ExchangeDataPartialCoordinateError[source]#

Error raised if values for latitude, longitude, or pressure are missing.

It is OK by the standard to omit the time of day.

exception cchdo.hydro.exchange.exceptions.ExchangeDataPartialKeyError[source]#

Error raised when there is no value for one (or more) of the following parameters.

  • EXPOCODE

  • STNNBR

  • CASTNO

  • SAMPNO (only for bottle files)

  • CTDPRS (only for CTD files)

These form the “composite key” which uniquely identify the “row” of exchange data.

exception cchdo.hydro.exchange.exceptions.ExchangeDuplicateKeyError[source]#

Error raised when there is a duplicate composite key in the exchange file.

This would occur if the exact values for the following parameters occur in more than one data row:

  • EXPOCODE

  • STNNBR

  • CASTNO

  • SAMPNO (only for bottle files)

  • CTDPRS (only for CTD files)

exception cchdo.hydro.exchange.exceptions.ExchangeDuplicateParameterError[source]#

Error raised when the same parameter/unit pair occurs more than once in the excahnge file.

exception cchdo.hydro.exchange.exceptions.ExchangeEncodingError[source]#

Error raised when the bytes for some exchange file cannot be decoded as UTF-8.

exception cchdo.hydro.exchange.exceptions.ExchangeEndDataError[source]#

Erorr raised when END_DATA cannot be found in the exchange file.

exception cchdo.hydro.exchange.exceptions.ExchangeError[source]#

This is the base exception which all the other exceptions derive from. It is a subclass of ValueError.

exception cchdo.hydro.exchange.exceptions.ExchangeFlagUnitError[source]#

Error raised if a flag column has a non empty units.

exception cchdo.hydro.exchange.exceptions.ExchangeFlaglessParameterError[source]#

Error raised when a parameter has a flag column when it is not supposed to.

exception cchdo.hydro.exchange.exceptions.ExchangeInconsistentMergeType[source]#

Error raised when the merge_ex method is called on mixed ctd and bottle exchange types

exception cchdo.hydro.exchange.exceptions.ExchangeLEError[source]#

Error raised when the exchange file does not have the correct line endings.

exception cchdo.hydro.exchange.exceptions.ExchangeMagicNumberError[source]#

Error raised when the exchange file does not start with BOTTLE or CTD.

exception cchdo.hydro.exchange.exceptions.ExchangeOrphanErrorError[source]#

Error raised when there exists an error column with no corresponding parameter column.

exception cchdo.hydro.exchange.exceptions.ExchangeOrphanFlagError[source]#

Error raised when there exists a flag column with no corresponding parameter column.

exception cchdo.hydro.exchange.exceptions.ExchangeParameterError[source]#

Base exception for errors related to parameters and units.

exception cchdo.hydro.exchange.exceptions.ExchangeParameterUndefError[source]#

Error raised when the library does not have a definition for a parameter/unit pair in the exchange file.

exception cchdo.hydro.exchange.exceptions.ExchangeParameterUnitAlignmentError[source]#

Error raised when there is a mismatch between the number of parameters and number of units in the exchange file.

exception cchdo.hydro.exchange.exceptions.ExchangeRecursiveZip[source]#

Error raised if there are zip files inside the zip file that read exchange is trying to read

flags#

A Collection of Flag Schemes

class cchdo.hydro.exchange.flags.ExchangeBottleFlag(value)[source]#

Enum representing a WHP Bottle flag

This flag represents information about the sampling device itself (i.e. the niskin bottle). It should only be used for “BTLNBR_FLAG_W” values and should never be used with CTD files.

BAD_TRIP = 4#
DISCREPANCY = 6#
GOOD = 2#
LEAKING = 3#
NOFLAG = 0#
NOT_REPORTED = 5#
NOT_SAMPLED = 9#
NO_INFO = 1#
PAIR = 8#
UNKNOWN = 7#
property cf_def#
property definition#

Prints the human readable flag definition

property has_value#

Should the data this is a flag for contain a value

class cchdo.hydro.exchange.flags.ExchangeCTDFlag(value)[source]#

An enumeration.

BAD = 4#
DESPIKED = 7#
GOOD = 2#
INTERPOLATED = 6#
NOFLAG = 0#
NOT_REPORTED = 5#
NOT_SAMPLED = 9#
QUESTIONABLE = 3#
UNCALIBRATED = 1#
property cf_def#
property definition#
property has_value#
class cchdo.hydro.exchange.flags.ExchangeSampleFlag(value)[source]#

An enumeration.

BAD = 4#
CHROMA_IRREGULAR = 8#
CHROMA_MANUAL = 7#
GOOD = 2#
MEAN = 6#
MISSING = 1#
NOFLAG = 0#
NOT_REPORTED = 5#
NOT_SAMPLED = 9#
QUESTIONABLE = 3#
property cf_def#
property definition#
property has_value#