API

Exchange Format

containers

class cchdo.hydro.exchange.containers.Exchange(file_type: 'FileType', comments: 'str', parameters: 'Tuple[WHPName, ...]', flags: 'Tuple[WHPName, ...]', errors: 'Tuple[WHPName, ...]', keys: 'Tuple[ExchangeCompositeKey, ...]', coordinates: 'Dict[ExchangeCompositeKey, ExchangeXYZT]', data: 'Dict[ExchangeCompositeKey, Dict[WHPName, ExchangeDataPoint]]')[source]
property _col_major_data
_ndarray_cache
property _param_source_c_formats
property _param_source_numeric_precision
comments
coord_to_dataarray(param)[source]
coordinates
data
error_to_dataarray(param, name=None)[source]
error_to_ndarray(param)[source]
errors
file_type
flag_to_dataarray(param, name=None)[source]
flag_to_ndarray(param)[source]
flags
gen_fname()[source]
iter_profile_coordinates()[source]
iter_profiles()[source]
key_to_dataarray(param)[source]
keys
property ndaray_indicies
parameter_to_dataarray(param, name=None)[source]
parameter_to_ndarray(param)[source]
parameters
sampletime_to_dataarray()[source]
sampletime_to_ndarray()[source]
property shape
time_precision()[source]
time_to_dataarray()[source]
time_to_ndarray()[source]

Time is a specal/funky case

Todo

Write why time is specal in exchange

to_exchange_csv(filename_or_obj=None, stamp='CCHDHYDRO', use_source_c_format=True)[source]

Export hydro.exchange.Exchange object to WHP-Exchange datafile(s)

to_xarray(source_c_format=True)[source]
class cchdo.hydro.exchange.containers.ExchangeCompositeKey(expocode: 'str', station: 'str', cast: 'int', sample: 'str')[source]
CASTNO = WHPName(whp_name='CASTNO', whp_unit=None, cf_name=None)
EXPOCODE = WHPName(whp_name='EXPOCODE', whp_unit=None, cf_name=None)
SAMPNO = WHPName(whp_name='SAMPNO', whp_unit=None, cf_name=None)
STNNBR = WHPName(whp_name='STNNBR', whp_unit=None, cf_name=None)
WHP_PARAMS = (WHPName(whp_name='EXPOCODE', whp_unit=None, cf_name=None), WHPName(whp_name='STNNBR', whp_unit=None, cf_name=None), WHPName(whp_name='CASTNO', whp_unit=None, cf_name=None), WHPName(whp_name='SAMPNO', whp_unit=None, cf_name=None))
_abc_impl = <_abc_data object>
_mapping
cast
expocode
classmethod from_data_line(data_line)[source]
property profile_id
sample
station
class cchdo.hydro.exchange.containers.ExchangeDataPoint(whpname: 'WHPName', value: 'Optional[Union[str, float, int]]', error: 'Optional[float]', flag: 'ExchangeFlags', source_c_format: 'Optional[int]')[source]
error
flag
classmethod from_ir(whpname, ir)[source]
source_c_format
value
whpname
class cchdo.hydro.exchange.containers.ExchangeXYZT(x: 'ExchangeDataPoint', y: 'ExchangeDataPoint', z: 'ExchangeDataPoint', t: 'np.datetime64', WHP_PARAMS: 'tuple' = (WHPName(whp_name='CTDPRS', whp_unit='DBAR', cf_name='sea_water_pressure'), WHPName(whp_name='DATE', whp_unit=None, cf_name=None), WHPName(whp_name='TIME', whp_unit=None, cf_name=None), WHPName(whp_name='LATITUDE', whp_unit=None, cf_name='latitude'), WHPName(whp_name='LONGITUDE', whp_unit=None, cf_name='longitude')))[source]
CF_AXIS = {WHPName(whp_name='DATE', whp_unit=None, cf_name=None): 'T', WHPName(whp_name='TIME', whp_unit=None, cf_name=None): 'T', WHPName(whp_name='LATITUDE', whp_unit=None, cf_name='latitude'): 'Y', WHPName(whp_name='LONGITUDE', whp_unit=None, cf_name='longitude'): 'X', WHPName(whp_name='CTDPRS', whp_unit='DBAR', cf_name='sea_water_pressure'): 'Z'}
CTDPRS = WHPName(whp_name='CTDPRS', whp_unit='DBAR', cf_name='sea_water_pressure')
DATE = WHPName(whp_name='DATE', whp_unit=None, cf_name=None)
LATITUDE = WHPName(whp_name='LATITUDE', whp_unit=None, cf_name='latitude')
LONGITUDE = WHPName(whp_name='LONGITUDE', whp_unit=None, cf_name='longitude')
TEMPORAL_PARAMS = (WHPName(whp_name='DATE', whp_unit=None, cf_name=None), WHPName(whp_name='TIME', whp_unit=None, cf_name=None))
TIME = WHPName(whp_name='TIME', whp_unit=None, cf_name=None)
WHP_PARAMS = (WHPName(whp_name='CTDPRS', whp_unit='DBAR', cf_name='sea_water_pressure'), WHPName(whp_name='DATE', whp_unit=None, cf_name=None), WHPName(whp_name='TIME', whp_unit=None, cf_name=None), WHPName(whp_name='LATITUDE', whp_unit=None, cf_name='latitude'), WHPName(whp_name='LONGITUDE', whp_unit=None, cf_name='longitude'))
_abc_impl = <_abc_data object>
property _date_part
_mapping
property _source_c_formats
property _time_part
classmethod from_data_line(data_line)[source]
t
x
y
z
class cchdo.hydro.exchange.containers.FileType(value)[source]

An enumeration.

BOTTLE = 'B'
CTD = 'C'
class cchdo.hydro.exchange.containers.IntermediateDataPoint(data, flag, error)[source]
_asdict()

Return a new dict which maps field names to their values.

_field_defaults = {}
_field_types = {'data': ForwardRef('str'), 'error': ForwardRef('Optional[str]'), 'flag': ForwardRef('Optional[str]')}
_fields = ('data', 'flag', 'error')
_fields_defaults = {}
classmethod _make(iterable)

Make a new IntermediateDataPoint object from a sequence or iterable

_replace(**kwds)

Return a new IntermediateDataPoint object replacing specified fields with new values

data

Alias for field number 0

error

Alias for field number 2

flag

Alias for field number 1

cchdo.hydro.exchange.containers._ndcache_helper(key)[source]

io

cchdo.hydro.exchange.io._bottle_get_errors(params_units, whp_params)[source]

Given an ordered iterable of param unit pairs and WHPNames known to be in the file, return the index of the column indicies of the errors/uncertanties for the WHPNames.

Some parameters may have uncertanties associated with them, this function finds those columns and pairs them with the correct parameter.

Note

There is no programable way to find the error columns for a given unit (e.g. no common suffix like the flags). This must be done via lookup in the built in database of params.

Parameters
  • params_units (tuple in the form (str, str) or (str, None)) – Paired (e.g. zip()) parameter names and units

  • whp_params (Mapping of WHPName to int) – Mapping of parameters known to be in the file, this is the output of _bottle_get_params()

Returns

Mapping of WHPName to column indicies for the error column

Return type

dict with keys of WHPName and values of int

cchdo.hydro.exchange.io._bottle_get_flags(params_units, whp_params)[source]

Given an ordered iterable of param unit pairs and WHPNames known to be in the file, return the index of the column indicies of the flags for the WHPNames.

Exchange files can have status flags for some of the parameters. Flag columns must have no units. Some parameters must not have status flags, these include the spatiotemporal parameters (e.g. lat, lon, but also pressure) and the sample identifying parameters (expocode, station, cast, sample, but not bottle id).

Parameters
  • params_units (tuple in the form (str, str) or (str, None)) – Paired (e.g. zip) parameter names and units

  • whp_params (Mapping of WHPName to int) – Mapping of parameters known to be in the file, this is the output of _bottle_get_params()

Returns

Mapping of WHPName to column indicies for the status flag column

Return type

dict with keys of WHPName and values of int

Raises
cchdo.hydro.exchange.io._bottle_get_params(params_units)[source]

Given an ordered iterable of param, unit pairs, return the index of the column in the datafile for known WHP params.

Exchange files have comma separated parameter names on one line, and the corresponding units on the next. This function will search for this name+unit pair in the builtin database of known WHP parameter names and return a mapping of WHPName to column indicies.

It is currently an error for the parameter in a file to not be in the built in database.

This function will ignore uncertainty (error) columns and flag columns, those are parsed by other functions.

Warning

Convert semantically empty units (e.g. empty string, all whitespace) to None before passing into this function

Note

The parameter name database will convert unambiguous aliases to their canonical exchange parameter and unit pair.

Parameters

params_units (tuple in the form (str, str) or (str, None)) – Paired (e.g. zip) parameter names and units

Returns

Mapping of WHPName to column indicies

Return type

dict with keys of WHPName and values of int

Raises

ExchangeParameterUndefError – if the parameter unit pair cannot be found in the built in database

cchdo.hydro.exchange.io._bottle_line_parser(names_index, flags_index, errors_index)[source]
cchdo.hydro.exchange.io._bottle_merge_sample_times(date, time)[source]
cchdo.hydro.exchange.io._ctd_get_header(line, dtype=<class 'str'>)[source]
cchdo.hydro.exchange.io._extract_comments(data, include_post_content=True)[source]

Destructively extract the comments from exchange data.

Exchange files may have zero or more lines of meaningless (to the format) comments. Between the CTD or BOTTLE stamp line and the start of the meaningful content of the file. These must be prefixed with a # character. Optionally, there might also be any amount of content after the END_DATA line of an exchange file. By default this function will extract that as well and append it to the retried comment string. This function will remove all the leading “#” from the comments.

Warning

This function expects the “stamp” line to have been popped from the deque already.

Parameters
  • data (collections.deque) – A deque containing the separated lines of an exchange file with the first line already popped.

  • include_post_content (bool) – If True, include any post END_DATA content as part of the comments, default: True.

Returns

The extracted comment lines from the deque

Return type

str

cchdo.hydro.exchange.io.read_exchange(filename_or_obj, parallelize='processpool', recursed=False)[source]

Open an exchange file and return an hydro.exchange.Exchange object

exceptions

exception cchdo.hydro.exchange.exceptions.ExchangeBOMError[source]

Error raised when the exchange file has a byte order mark.

exception cchdo.hydro.exchange.exceptions.ExchangeDataColumnAlignmentError[source]

Error raised when the number of columns in a data line does not match the expected number of columns based on the parameter/unit lines.

exception cchdo.hydro.exchange.exceptions.ExchangeDataError[source]

Base exception for errors which occur when parsing the data porition of an exchange file.

exception cchdo.hydro.exchange.exceptions.ExchangeDataFlagPairError[source]

There is a mismatch between what the flag value expects, and the fill/data value.

Examples:

  • something with a flag of 9 has a non fill value

  • something with a flag of 2 as a fill value instead of data

exception cchdo.hydro.exchange.exceptions.ExchangeDataInconsistentCoordinateError[source]

Error raised if the reported latitude, longitude, and date (and time) vary for a single profile.

A “profile” in an exchange file is a grouping of data rows which all have the same EXPOCODE, STNNBR, and CASTNO. The SAMPNO/CTDPRS is allowed/requried to vary for a single profile and is what identifies samples within one profile.

exception cchdo.hydro.exchange.exceptions.ExchangeDataPartialCoordinateError[source]

Error raised if values for latitude, longitude, or date are missing.

It is OK by the standard to omit the time of day.

exception cchdo.hydro.exchange.exceptions.ExchangeDataPartialKeyError[source]

Error raised when there is no value for one (or more) of the following parameters.

  • EXPOCODE

  • STNNBR

  • CASTNO

  • SAMPNO (only for bottle files)

  • CTDPRS (only for CTD files)

These form the “composite key” which uniquely identify the “row” of exchange data.

exception cchdo.hydro.exchange.exceptions.ExchangeDuplicateKeyError[source]

Error raised when there is a duplicate composite key in the exchange file.

This would occur if the exact values for the following parameters occur in more than one data row:

  • EXPOCODE

  • STNNBR

  • CASTNO

  • SAMPNO (only for bottle files)

  • CTDPRS (only for CTD files)

exception cchdo.hydro.exchange.exceptions.ExchangeDuplicateParameterError[source]

Error raised when the same parameter/unit pair occurs more than once in the excahnge file.

exception cchdo.hydro.exchange.exceptions.ExchangeEncodingError[source]

Error raised when the bytes for some exchange file cannot be decoded as UTF-8.

exception cchdo.hydro.exchange.exceptions.ExchangeEndDataError[source]

Erorr raised when END_DATA cannot be found in the exchange file.

exception cchdo.hydro.exchange.exceptions.ExchangeError[source]

This is the base exception which all the other exceptions derive from. It is a subclass of ValueError.

exception cchdo.hydro.exchange.exceptions.ExchangeFlagUnitError[source]

Error raised if a flag column has a non empty units.

exception cchdo.hydro.exchange.exceptions.ExchangeFlaglessParameterError[source]

Error raised when a parameter has a flag column when it is not supposed to.

exception cchdo.hydro.exchange.exceptions.ExchangeInconsistentMergeType[source]

Error raised when the merge_ex method is called on mixed ctd and bottle exchange types

exception cchdo.hydro.exchange.exceptions.ExchangeLEError[source]

Error raised when the exchange file does not have the correct line endings.

exception cchdo.hydro.exchange.exceptions.ExchangeMagicNumberError[source]

Error raised when the exchange file does not start with BOTTLE or CTD.

exception cchdo.hydro.exchange.exceptions.ExchangeOrphanFlagError[source]

Error raised when there exists a flag column with no corresponding parameter column.

exception cchdo.hydro.exchange.exceptions.ExchangeParameterError[source]

Base exception for errors related to parameters and units.

exception cchdo.hydro.exchange.exceptions.ExchangeParameterUndefError[source]

Error raised when the library does not have a definition for a parameter/unit pair in the exchange file.

exception cchdo.hydro.exchange.exceptions.ExchangeParameterUnitAlignmentError[source]

Error raised when there is a mismatch between the number of parameters and number of units in the exchange file.

exception cchdo.hydro.exchange.exceptions.ExchangeRecursiveZip[source]

Error raised if there are zip files inside the zip file that read exchange is trying to read

flags

A Collection of Flag Schemes

class cchdo.hydro.exchange.flags.ExchangeBottleFlag(value)[source]

Enum representing a WHP Bottle flag

This flag represents information about the sampling device itself (i.e. the niskin bottle). It should only be used for “BTLNBR_FLAG_W” values and should never be used with CTD files.

BAD_TRIP = 4
DISCREPANCY = 6
GOOD = 2
LEAKING = 3
NOFLAG = 0
NOT_REPORTED = 5
NOT_SAMPLED = 9
NO_INFO = 1
PAIR = 8
UNKNOWN = 7
property cf_def
property definition

Prints the human readable flag definition

property has_value

Should the data this is a flag for contain a value

class cchdo.hydro.exchange.flags.ExchangeCTDFlag(value)[source]

An enumeration.

BAD = 4
DESPIKED = 7
GOOD = 2
INTERPOLATED = 6
NOFLAG = 0
NOT_REPORTED = 5
NOT_SAMPLED = 9
QUESTIONABLE = 3
UNCALIBRATED = 1
property cf_def
property definition
property has_value
class cchdo.hydro.exchange.flags.ExchangeSampleFlag(value)[source]

Test Doc String

BAD = 4
CHROMA_IRREGULAR = 8
CHROMA_MANUAL = 7
GOOD = 2
MEAN = 6
MISSING = 1
NOFLAG = 0
NOT_REPORTED = 5
NOT_SAMPLED = 9
QUESTIONABLE = 3
property cf_def
property definition
property has_value

merge

cchdo.hydro.exchange.merge.merge_ex(*exchanges)[source]
cchdo.hydro.exchange.merge.ordered_merge(*tuples)[source]