pyiem.util module#

Utility functions for pyIEM package

This module contains utility functions used by various parts of the codebase.

class pyiem.util.CustomFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)[source]#

Bases: Formatter

A custom log formatter class.

format(record)[source]#

Return a string!

pyiem.util.ddhhmm2datetime(ddhhmm, utcnow)[source]#

Do the nasty WMO header timestamp conversion.

Parameters:
Return type:

datetime

pyiem.util.web2ldm(url, ldm_product_name, md5_from_name=False, pqinsert='pqinsert')[source]#

Download a URL and insert into LDM.

Implements a common IEM workflow whereby a web resource is downloaded, saved to a temporary file, and then inserted into LDM.

Parameters:

  • url (str) – Web resource to download.

  • ldm_product_name (str) – LDM product ID to use when inserting.

  • md5_from_name (bool) – Should pqinsert -i be used, which causes LDM to compute the MD5 value from the product name instead of data bytes.

  • pqinsert (str) – pqinsert command.

Returns:

bool - success of this workflow.

pyiem.util.load_geodf(dataname, epsg=4326)[source]#

Load a given bundled GeoDataFrame.

Parameters:

  • dataname (str) – The name of the dataset name to load.

  • epsg (int)

Returns:

GeoDataFrame

pyiem.util.convert_value(val, units_in, units_out)[source]#

DRY Helper to return magnitude of a metpy unit conversion.

Parameters:

  • val (mixed) – something with values.

  • units_in (str) – What units those values have.

  • units_out (str) – What values we want with given magnitude.

Returns:

magnitude of val with unit conversion applied

Return type:

mixed

pyiem.util.c2f(val)[source]#

Helper to return magnitude of Celcius to Fahrenheit conversion.

Parameters:

val (mixed) – something with values in C

Returns:

something with values in F

Return type:

val

pyiem.util.mm2inch(val)[source]#

Helper to return magnitude of milimeters to inch conversion.

Parameters:

val (mixed) – something with values in mm

Returns:

something with values in inch

Return type:

val

pyiem.util.html_escape(val)[source]#

Wrapper around cgi.escape deprecation.

pyiem.util.get_test_filepath(name)[source]#

Helper to get a testing filename, full path.

Parameters:

name (str)

Return type:

str

pyiem.util.get_test_file(name)[source]#

Helper to get data for test usage.

pyiem.util.logger(name='pyiem', level=None)[source]#

Get pyiem’s logger with a stream handler attached.

Parameters:

  • name (str) – The name of the logger to get, default pyiem

  • level (logging.LEVEL) – The log level for this pyiem logget, default is WARNING for non interactive sessions, INFO otherwise

Returns:

logger instance

pyiem.util.find_ij(lons, lats, lon, lat)[source]#

Compute the i,j closest cell.

pyiem.util.ssw(mixedobj)[source]#

python23 wrapper for sys.stdout.write

Parameters:

mixedobj (str or bytes) – what content we want to send

pyiem.util.ncopen(ncfn, mode='r', timeout=60, _sleep=5)[source]#

Safely open netcdf files

The issue here is that we can only have the following situation for a given NetCDF file. 1. Only 1 or more readers 2. Only 1 appender

The netcdf is being accessed over NFS and perhaps local disk, so writing lock files is problematic.

Parameters:

  • ncfn (str) – The netCDF filename

  • mode (str,optional) – The netCDF4.Dataset open mode, default ‘r’

  • timeout (int) – The total time in seconds to attempt a read, default 60

Returns:

netCDF4.Dataset or None

pyiem.util.utc(year=None, month=1, day=1, hour=0, minute=0, second=0, microsecond=0)[source]#

Create a datetime instance with tzinfo=timezone.utc

When no arguments are provided, returns datetime.now(timezone.utc).

Returns:

datetime with tzinfo set

pyiem.util.noaaport_text(text)[source]#

Make whatever text look like it is NOAAPort Pristine

Parameters:

text (string) – the inbound text

Returns:

text that looks noaaportish

pyiem.util.handle_date_err(exp, value, fmt)[source]#

Attempt to fix up a date string, when possible.

pyiem.util.get_autoplot_context(fdict, cfg, enforce_optional=False, **kwargs)[source]#

Get the variables out of a dict of strings

This helper for IEM autoplot gets values out of a dictionary of strings, as provided by CGI. It does some magic to get types right, defaults right and so on. The typical way this is called

ctx = iemutils.get_context(fdict, get_description())

Parameters:

  • fdict (dictionary) – what was likely provided by cgi.FieldStorage()

  • cfg (dictionary) – autoplot value of get_description

  • enforce_optional (bool,optional) – Should the optional flag be enforced

  • rectify_dates (bool,optional) – Attempt to fix common date errors like June 31. Default false.

Returns:

dictionary of variable names and values, with proper types!

pyiem.util.exponential_backoff(func, *args, **kwargs)[source]#

Exponentially backoff some function until it stops erroring

Parameters:

_ebfactor (int,optional) – Optional scale factor, allowing for faster test

pyiem.util.delete_property(name, cursor=None)[source]#

Delete a property from the database.

Parameters:

  • name (str) – The name of the property to delete

  • cursor (psycopg2.cursor) – Optional database cursor to use

pyiem.util.get_properties(cursor=None)[source]#

Fetch the properties set

Returns:

a dictionary of property names and values (both str)

Return type:

dict

pyiem.util.set_property(name, value, cursor=None)[source]#

Set a property value in the database.

Parameters:

  • name (str) – The name of the property to set

  • value (str,datetime) – The value to set

  • cursor (psycopg2.cursor) – Optional database cursor to use

pyiem.util.drct2text(drct)[source]#

Convert an degree value to text representation of direction.

Parameters:

drct (int or float) – Value in degrees to convert to text

Returns:

String representation of the direction, could be None

Return type:

str

pyiem.util.grid_bounds(lons, lats, bounds)[source]#

Figure out indices that we can truncate big grid

Parameters:

  • lons (np.array) – grid lons

  • lats (np.array) – grid lats

  • bounds (list) – [x0, y0, x1, y1]

Returns:

[x0, y0, x1, y1]

pyiem.util.archive_fetch(partialpath, localdir='/mesonet/ARCHIVE/data', method='get')[source]#

Helper to fetch a file from the archive, by first looking at the filesystem and then going to the website. This returns a filename. If a temporary file is created, it is deleted after the context manager exits.

Parameters:

  • partialpath (str) – Typically a path that starts with /YYYY/mm/dd

  • method (str) – HTTP method to use, default ‘get’, in the case of head, we only check the existence and return an empty string if found.

  • localdir (str)

Returns:

filename of the file found and available for use

Return type:

str

pyiem.util.__getattr__(name)[source]#

Shim to allow deprecation notices to start.

Parameters:

name (str)