pyiem package

Subpackages

Submodules

pyiem.database module

Database helpers.

pyiem.database.get_dbconn(database='mesosite', user=None, host=None, port=5432, **kwargs)

Helper function with business logic to get a database connection

Note that this helper could return a read-only database connection if the connection to the primary server fails.

Parameters:

  • database (str,optional) – the database name to connect to. default: mesosite

  • user (str,optional) – hard coded user to connect as, default: current user

  • host (str,optional) – hard coded hostname to connect as, default: iemdb.local

  • port (int,optional) – the TCP port that PostgreSQL is listening defaults to 5432

  • password (str,optional) – the password to use.

Returns:

psycopg database connection

pyiem.database.get_dbconnc(database: str = 'mesosite', user: str = None, host: str = None, cursor_name: str = None, **kwargs) → tuple[Connection[Dict[str, Any]], ServerCursor[Dict[str, Any]]]

Helper function to get a database connection + dict_row cursor.

Note that this helper could return a read-only database connection if the connection to the primary server fails.

Parameters:

  • database (str,optional) – the database name to connect to. default: mesosite

  • user (str,optional) – hard coded user to connect as, default: current user

  • host (str,optional) – hard coded hostname to connect as, default: iemdb.local

  • cursor_name (str,optional) – name of the cursor to create

  • port (int,optional) – the TCP port that PostgreSQL is listening defaults to 5432

  • password (str,optional) – the password to use.

pyiem.database.get_dbconnstr(name, **kwargs) → str

Create a database connection string/URI.

Parameters:

  • name (str) – the database name to connect to.

  • **kwargs – any additional arguments to pass to psycopg.connect user (str): the database user to connect as host (str): the database host to connect to port (int): the database port to connect to connect_timeout (int): Connection timeout in seconds, default 30.

Returns:

str

pyiem.database.get_sqlalchemy_conn(name: str, **kwargs) → Generator[Connection, None, None]

An auto-disposing sqlalchemy context-manager helper.

This is used for when we really do not want to manage having pools of database connections open. So this isn’t something that is fast!

Parameters:

  • name (str) – the database to connect to, passed to get_dbconnstr

  • **kwargs – any additional arguments to pass to get_dbconnstr

pyiem.datatypes module

Classes Representing various variables

class pyiem.datatypes.basetype(value, units)

Bases: object

Base Object for all our vars

get_units()

Returns the units for this current datatype

Returns:

the unit of this datatype

Return type:

(str)

class pyiem.datatypes.direction(value, units)

Bases: basetype

Direction from North

known_units = ['DEG', 'RAD']
value(units)

Convert to a value in the given units

class pyiem.datatypes.distance(value, units)

Bases: basetype

Distance

known_units = ['SM', 'MI', 'M', 'KM', 'FT', 'MM', 'IN', 'CM']
value(units)

Convert to a value in the given units

class pyiem.datatypes.humidity(value, units)

Bases: basetype

Humidity, this is not as straight forward as the others

known_units = ['%']
value(units)

Convert to a value in the given units

class pyiem.datatypes.mixingratio(value, units)

Bases: basetype

Mixing Ratio.

known_units = ['KG/KG']
value(units)

Convert the value into the provided units

class pyiem.datatypes.precipitation(value, units)

Bases: basetype

Precipitation

known_units = ['IN', 'CM', 'MM']
value(units)

Convert to a value in the given units

class pyiem.datatypes.pressure(value, units)

Bases: basetype

Pressure

known_units = ['MB', 'HPA', 'IN', 'PA']
value(units)

Convert to a value in the given units

class pyiem.datatypes.speed(value, units)

Bases: basetype

Speed

known_units = ['KT', 'MPH', 'MPS', 'KMH', 'KTS']
value(units=None)

Convert to a value in the given units

class pyiem.datatypes.temperature(value, units)

Bases: basetype

Temperature

known_units = ['F', 'K', 'C']
value(units)

Convert to a value in the given units

pyiem.dep module

Utilities for the Daily Erosion Project

pyiem.dep.get_cli_fname(lon, lat, scenario=0)

Get the climate file name for the given lon, lat, and scenario

pyiem.dep.load_scenarios()

Build a dataframe of DEP scenarios.

pyiem.dep.man2df(mandict: dict, year1: int = 1) → DataFrame

Convert nasty dictionary returned from read_man into pd.DataFrame.

The DataFrame is oriented with OFE, year.

Parameters:

  • mandict (dict) – Dictionary populated from read_man.

  • year1 (int,optional) – What does WEPP year index 1 equate to in the real world! The default of 1 just uses what WEPP does.

Returns:

pd.DataFrame

pyiem.dep.read_cli(filename, compute_rfactor=False, return_rfactor_metric=True)

Read WEPP CLI File, Return DataFrame

Parameters:

  • filename (str) – Filename to read

  • compute_rfactor (bool, optional) – Should the R-factor be computed as well, adds computational expense and default is False.

  • return_rfactor_metric (bool, optional) – should the R-factor be computed as the common metric value. Default is True.

Returns:

pandas.DataFrame

pyiem.dep.read_crop(filename)

Read WEPP’s plant and residue output file.

Parameters:

filename (str) – The file to read in.

Returns:

pandas.DataFrame

pyiem.dep.read_env(filename, year0=2006)

Read WEPP .env file, return a dataframe

Parameters:

  • filename (str) – Filename to read

  • year0 (int,optional) – The simulation start year minus 1

Returns:

pd.DataFrame

pyiem.dep.read_man(filename)

Implements WEPP’s INFILE.for for reading management file

Parameters:

filename (str) – Filename to read

Returns:

dict of management info

pyiem.dep.read_ofe(filename, year0=2006)

Read OFE .ofe file, return a dataframe

Parameters:

  • filename (str) – Filename to read

  • year0 (int,optional) – The simulation start year minus 1

Returns:

pd.DataFrame

pyiem.dep.read_slp(filename)

read WEPP slp file.

Parameters:

filename (str) – Filename to read

Returns:

list of slope profiles

pyiem.dep.read_wb(filename)

Read a custom WEPP .wb file into Pandas Data Table

pyiem.dep.read_yld(filename)

read WEPP yld file with some local mods to include a year

Parameters:

filename (str) – Filename to read

Returns:

pandas.DataFrame

pyiem.dep.rfactor(times, points, return_rfactor_metric=True)

Compute the R-factor.

https://www.hydrol-earth-syst-sci.net/19/4113/2015/hess-19-4113-2015.pdf It would appear that a strict implementation would need to have a six hour dry period around events and require more then 12mm of precipitation.

Parameters:

  • times (list) – List of decimal time values for a date.

  • points (list) – list of accumulated precip values (mm).

  • return_rfactor_metric (bool, optional) – Should this return a metric (default) or english unit R value.

Returns:

Units of MJ mm ha-1 h-1

Return type:

rfactor (float)

pyiem.exceptions module

Custom Exceptions.

exception pyiem.exceptions.BadWebRequest

Bases: Exception

Raised when a bad web request is made.

exception pyiem.exceptions.CLIException

Bases: Exception

Custom Exception for CLI Parsing Issues

exception pyiem.exceptions.HWOException

Bases: Exception

Exception for HWO Parsing.

exception pyiem.exceptions.IncompleteWebRequest

Bases: Exception

Raised for a HTTP GET request without required params (422).

exception pyiem.exceptions.InvalidArguments

Bases: Exception

Provided method arguments were not valid (invalid units).

exception pyiem.exceptions.InvalidPolygon

Bases: Exception

Parsing of polygon raised a known error condition.

exception pyiem.exceptions.InvalidSHEFEncoding

Bases: Exception

Product is not encoded to SHEF standard specification.

exception pyiem.exceptions.InvalidSHEFValue

Bases: Exception

SHEF element value fails to be processed to a float.

exception pyiem.exceptions.MCDException

Bases: Exception

exception pyiem.exceptions.NHCException

Bases: Exception

exception pyiem.exceptions.NewDatabaseConnectionFailure

Bases: Exception

Exception for when a new database connection fails.

exception pyiem.exceptions.NoDataFound

Bases: Exception

Exception for when no data was found for request.

exception pyiem.exceptions.SAWException

Bases: Exception

Custom local exception

exception pyiem.exceptions.SIGMETException

Bases: Exception

Custom SIGMET Parsing Exception.

exception pyiem.exceptions.TextProductException

Bases: Exception

Exception for Text Parsing.

exception pyiem.exceptions.UGCParseException

Bases: Exception

Custom Exception this parser can raise

exception pyiem.exceptions.UnitsError

Bases: Exception

Exception for bad Units.

exception pyiem.exceptions.UnknownStationException

Bases: Exception

Exception for unknown station.

pyiem.geom_util module

Geometry utility functions.

pyiem.geom_util.rhs_split(poly, splitter)

Provide the Right Hand Side Polygon associated with a split operation.

Parameters:

  • poly (shapely.geometry.Polygon) – polygon to split.

  • splitter (shapely.geometry.LineString) – linestring to spliy by.

Returns:

shapely.geometry.Polygon

pyiem.htmlgen module

Utility functions that generate HTML.

pyiem.htmlgen.make_select(name, selected, data, jscallback=None, cssclass=None, multiple=False, showvalue=True) → str

Generate a HTML select element.

The trick here is what data looks like. The basic form is a dict. You can get optgroup by having the dictionary keys be additional lists or dicts.

Parameters:

  • name (str) – The select name attribute to assign.

  • selected (mixed) – The option value that should be set to selected.

  • data (dict) – The structure to build our select from.

  • jscallback (str) – javascript to place in the onChange attribute.

  • cssclass (str) – CSS class to assign to the select element.

  • showvalue (bool) – Should option label be prepended by key label.

Return type:

html_string

pyiem.htmlgen.station_select(network, selected, name, select_all=False, only_online=False) → str

Select a station from a given network.

Parameters:

  • network (str) – IEM Network identifier.

  • selected (str) – The option value to flag as selected.

  • name (str) – The HTML select name attribute.

  • select_all (bool) – Add an option with key of _ALL.

  • only_online (bool) – Include stations in the network that are online only, default is False.

Return type:

html_string

pyiem.iemre module

Support library for the IEM Reanalysis code.

pyiem.iemre.d2l(val) → str

Convert a domain label to a string used within filenames.

pyiem.iemre.daily_offset(ts)

Compute the timestamp index in the netcdf file

pyiem.iemre.find_ij(lon: float, lat: float, domain: str = '') → Tuple[int | None, int | None]

Return the i, j grid indices (based 0) for given lat/lon.

pyiem.iemre.get_daily_mrms_ncname(year)

Get the daily netcdf MRMS filename for the given year

pyiem.iemre.get_daily_ncname(year, domain: str = '') → str

Get the daily netcdf filename for the given year

pyiem.iemre.get_dailyc_mrms_ncname()

Get the MRMS daily climatology filename

pyiem.iemre.get_dailyc_ncname(domain: str = '') → str

Return the filename of the daily climatology netcdf file

pyiem.iemre.get_domain(lon: float, lat: float) → str | None

Compute the domain that contains the given point.

pyiem.iemre.get_gid(lon, lat, domain: str = '') → int | None

Compute the grid id for the given location.

pyiem.iemre.get_grids(valid, varnames=None, cursor=None, table=None, domain: str = '')

Fetch grid(s) from the database, returning xarray.

Parameters:

  • valid (datetime or date) – If datetime, load hourly, if date, load daily

  • varnames (str or list,optional) – Which variables to fetch from database, defaults to all available

  • cursor (database cursor,optional) – cursor to use for query

  • table (str,optional) – Hard coded table to fetch data from, useful in the case of forecast data.

  • domain (str,optional) – IEMRE domain to fetch data from

Returns:

xarray.Dataset

pyiem.iemre.get_hourly_ncname(year, domain: str = '') → str

Get the daily netcdf filename for the given year

pyiem.iemre.get_table(valid)

Figure out which table should be used for given valid.

Parameters:

valid (datetime or date) – which time is of interest

Returns:

str tablename

pyiem.iemre.grb2iemre(grb, resampling=None, domain: str = '') → ndarray

Reproject a grib message onto the IEMRE grid.

A helper frontend to reproject2iemre.

Parameters:

  • grb (pygrib.gribmessage) – single message to reproject

  • resampling (rasterio.warp.Resampling,optional) – defaults to nearest

  • domain (str) – IEMRE domain to reproject onto

Returns:

numpy.ma.array of reprojected grid oriented S to N like IEMRE

pyiem.iemre.hourly_offset(dtobj)

Return time index for given timestamp

Parameters:

dtobj (datetime) – datetime, if no tzinfo, we assume it is UTC

Returns:

int time index in the netcdf file

pyiem.iemre.reproject2iemre(grid, affine_in, crs_in: str, resampling=None, domain: str = '')

Reproject the given grid to IEMRE grid, returning S to N oriented grid.

Note: If the affine_in is dy > 0 then the grid is assumed to be S to N.

Parameters:

  • grid (numpy.array) – 2D grid to reproject

  • affine_in (affine.Affine) – affine transform of input grid

  • crs_in (pyproj.Proj) – projection of input grid

  • resampling (rasterio.warp.Resampling,optional) – defaults to nearest

  • domain (str) – IEMRE domain to reproject onto

Returns:

numpy.ma.array of reprojected grid oriented S to N like IEMRE

pyiem.iemre.set_grids(valid, ds, table=None, domain: str = '')

Update the database with a given xarray.Dataset.

Parameters:

  • valid (datetime or date) – If datetime, save hourly, if date, save daily

  • ds (xarray.Dataset) – The xarray dataset to save

  • table (str,optional) – hard coded database table to use to set the data on. Usually dynamically computed.

  • domain (str,optional) – IEMRE domain to save data to

pyiem.meteorology module

We do meteorological things, when necessary

pyiem.meteorology.clearsky_shortwave_irradiance_year(lat, elevation)

Compute the Clear Sky Shortwave Irradiance for year in MJ m**-2

Parameters:

  • lat (float) – latitude

  • elevation (float) – location elevation in meters

Returns:

irradiance (list)

pyiem.meteorology.comprehensive_climate_index(airtemp, rh, windspeed, solarrad, shade_effect: bool = False)

Compute the Comprehensive Climate Index.

Note: shade_effect multiplies the solarrad value by 0.2 (80% reduction)

pyiem.meteorology.dewpoint(temperature, relhumid)

Compute Dew Point given a temperature and RH%

pyiem.meteorology.dewpoint_from_pq(pressure, mixingratio)

Compute the Dew Point given a Pressure and Mixing Ratio

pyiem.meteorology.drct(u, v)

Compute the wind direction given a u and v wind speed

Parameters:

  • u (dt.speed) – u component wind speed

  • v (dt.speed) – v component wind speed

Returns:

dt.direction value

pyiem.meteorology.gdd(high, low, base=50.0, ceiling=86.0)

Compute Growing Degree Days

Parameters:

  • high (temperature, or metpy.units) – High Temperature

  • low (temperature or metpy.units) – Low Temperature

  • base (int) – Base to use in GDD Computation (F)

  • ceiling (int) – Ceiling to use in GDD Computation (F)

Returns:

float value for GDDs

pyiem.meteorology.heatindex(temperature, polyarg)

Compute the heat index based on

Stull, Richard (2000). Meteorology for Scientists and Engineers, Second Edition. Brooks/Cole. p. 60. ISBN 9780534372149.

Another opinion on appropriate equation: http://www.wpc.ncep.noaa.gov/html/heatindex_equation.shtml

http://www.weather.gov/media/ffc/ta_htindx.PDF

pyiem.meteorology.mcalc_feelslike(tmpf, dwpf, smps, mask_undefined=False)

Compute a feels like temperature

Parameters:

  • temperature (temperature) – The dry bulb temperature

  • dewpoint (temperature) – The dew point temperature

  • speed (speed) – the wind speed

  • mask_undefined (bool) – If True, mask values where feels like is undef.

Returns:

The feels like temperature

Return type:

temperature (temperature)

pyiem.meteorology.mixing_ratio(_dewpoint)

Compute the mixing ratio

Parameters:

dewpoint (temperature) – Dew Point temperature

Returns:

mixing ratio

pyiem.meteorology.relh(temperature, _dewpoint)

Compute Relative Humidity based on a temperature and dew point

pyiem.meteorology.temperature_humidity_index(temperature, humidity)

Compute the Temperature Humidity Index

Parameters:

  • temperature (temperature) – The Air Temperature

  • humidity (humidity) – The Relative Humidity

Returns:

The THI

Return type:

temperature (temperature)

pyiem.meteorology.uv(speed, direction)

Compute the u and v components of the wind @param wind speed in whatever units @param dir wind direction with zero as north @return u and v components

pyiem.meteorology.windchill(temperature, speed)

Compute the wind chill temperature

http://www.ofcm.gov/publications/reports2.htm

Parameters:

  • temperature (temperature) – The Air Temperature

  • speed (speed) – The Wind Speed

Returns:

The Wind Chill Temperature

Return type:

temperature (temperature)

pyiem.mrms module

Multi-RADAR Multi-Sensor (MRMS) Helper Functions.

Hopefully useful functions to help with the processing of MRMS data

pyiem.mrms.fetch(product, valid: datetime, tmpdir='/mesonet/tmp')

Get a desired MRMS product.

Applies the following logic:

  • does the file exist in tmpdir?

  • can I fetch it from mtarchive?

  • if recent, can I fetch it from NOAA MRMS website

Parameters:

  • product (str) – MRMS product type

  • valid (datetime) – Datetime object for desired timestamp

  • tmpdir (str,optional) – location to check/place the downloaded file

pyiem.mrms.find_ij(lon: float, lat: float) → tuple[int | None, int | None]

CAUTION: This is for the netcdf files, not grib2 files.

pyiem.mrms.get_url(center, valid, product)

Return the URL given the provided options.

pyiem.mrms.is_gzipped(text)

Check that we have gzipped content.

pyiem.mrms.make_colorramp()

Make me a crude color ramp.

pyiem.mrms.reader(fn)

Return metadata and the data.

pyiem.mrms.write_worldfile(filename)

Write a worldfile to the given filename.

Parameters:

filename (str) – filename to write the world file information to

pyiem.network module

Network Table.

class pyiem.network.Table(network, cursor=None, only_online=True)

Bases: object

Our class

get_id_by_key(key, value) → str

Find a station id by a given attribute = value.

Parameters:

  • key (str) – attribute to lookup.

  • value (mixed) – value to compare against

Returns:

station_id

get_threading_id(sid, valid) → str

Return a station identifier (not iemid) based on threading.

Lookup what the threaded station identifier is based on this given timestamp/date.

Parameters:

  • sid (str) – station identifier to check threading for.

  • valid (datetime.date) – lookup for comparison.

pyiem.observation module

A class representing an observation stored in the IEM database

class pyiem.observation.Observation(station=None, network=None, valid=None, iemid=None, tzname=None)

Bases: object

my observation object

calc()

Compute things not usually computed

compute_iemid(txn)

Load in what our metadata is to save future queries

load(txn)

Load the current observation for this site and time

save(txn, force_current_log=False, skip_current=False)

Save this observation to the database via a psycopg transaction @param txn is a psycopg transaction @param force_current_log boolean - make sure this observation goes to the current_log table in the case that it is old, this allows reprocessing by the METAR ingestor et al @param skip_current boolean - optionally skip updating the current table. This is useful for partial obs @return: boolean if this updated one row each

pyiem.observation.bounded(val, floor, ceiling)

Make sure this is not NaN and between some value.

pyiem.observation.get_summary_table(valid)

Optimize the summary table we potentially use.

Parameters:

valid (datetime with time zone) – Datetime

Returns:

str table to query

pyiem.observation.summary_update(txn, data)

Updates the summary table and returns affected rows.

Parameters:

  • txn (psycopg.transaction)

  • data (dict)

Returns:

affected rows count

Return type:

int

pyiem.prism module

Helper for PRISM BIL data

pyiem.prism.daily_offset(ts)

Compute the timestamp index in the netcdf file

pyiem.prism.find_ij(lon, lat)

Compute which grid cell this lon, lat resides within

pyiem.reference module

Some Magic Here.

pyiem.tracker module

IEM Tracker Related Stuff.

class pyiem.tracker.TrackerEngine(icursor, pcursor, maxactions=0)

Bases: object

A processing engine of tracking offline/online events

offline_logic(sid, ob, pnetwork, nt)

offline logic

Parameters:

  • sid (str) – site identifier

  • ob (dict) – observation dictionary

  • pnetwork (str) – Portfolio name of this network

  • nt (dict) – provider of station metadata

online_logic(sid, offline, ob, pnetwork, nt)

online logic

Parameters:

  • sid (str) – site identifier

  • offline (dict) – dictionary of offline metadata

  • ob (dict) – observation dictionary

  • pnetwork (str) – Portfolio name of this network

  • nt (dict) – provider of station metadata

process_network(obs, pnetwork, nt, threshold)

Process a list of dicts representing the network’s observations

Parameters:

  • obs (list) – list of dicts representing the network obs

  • pnetwork (str) – the identifier of this network used in Portfolio DB

  • nt (NetworkTable) – dictionary provider of station metadata

  • threshold (datetime) – datetime instance with tzinfo set represeting the minimum time a site is considered to be ‘online’ within

send_emails()

Send out those SPAM emails!

pyiem.tracker.loadqc(cursor=None, date=None)

Load the current IEM Tracker QC’d variables

Parameters:

  • cursor (cursor,optional) – Optionally provided database cursor

  • date (date,optional) – Defaults to today, which tickets are valid for now

pyiem.util module

Utility functions for pyIEM package

This module contains utility functions used by various parts of the codebase.

class pyiem.util.CustomFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

Bases: Formatter

A custom log formatter class.

format(record)

Return a string!

pyiem.util.archive_fetch(partialpath: str, localdir: str = '/mesonet/ARCHIVE/data')

Helper to fetch a file from the archive, by first looking at the filesystem and then going to the website. This returns a filename. If a temporary file is created, it is deleted after the context manager exits.

Parameters:

partialpath (str) – Typically a path that starts with /YYYY/mm/dd

Returns:

filename of the file found and available for use

Return type:

str

pyiem.util.c2f(val)

Helper to return magnitude of Celcius to Fahrenheit conversion.

Parameters:

val (mixed) – something with values in C

Returns:

something with values in F

Return type:

val

pyiem.util.convert_value(val, units_in, units_out)

DRY Helper to return magnitude of a metpy unit conversion.

Parameters:

  • val (mixed) – something with values.

  • units_in (str) – What units those values have.

  • units_out (str) – What values we want with given magnitude.

Returns:

magnitude of val with unit conversion applied

Return type:

mixed

pyiem.util.delete_property(name, cursor=None)

Delete a property from the database.

Parameters:

  • name (str) – The name of the property to delete

  • cursor (psycopg2.cursor) – Optional database cursor to use

pyiem.util.drct2text(drct)

Convert an degree value to text representation of direction.

Parameters:

drct (int or float) – Value in degrees to convert to text

Returns:

String representation of the direction, could be None

Return type:

str

pyiem.util.exponential_backoff(func, *args, **kwargs)

Exponentially backoff some function until it stops erroring

Parameters:

_ebfactor (int,optional) – Optional scale factor, allowing for faster test

pyiem.util.find_ij(lons, lats, lon, lat)

Compute the i,j closest cell.

pyiem.util.get_autoplot_context(fdict, cfg, enforce_optional=False, **kwargs)

Get the variables out of a dict of strings

This helper for IEM autoplot gets values out of a dictionary of strings, as provided by CGI. It does some magic to get types right, defaults right and so on. The typical way this is called

ctx = iemutils.get_context(fdict, get_description())

Parameters:

  • fdict (dictionary) – what was likely provided by cgi.FieldStorage()

  • cfg (dictionary) – autoplot value of get_description

  • enforce_optional (bool,optional) – Should the optional flag be enforced

  • rectify_dates (bool,optional) – Attempt to fix common date errors like June 31. Default false.

Returns:

dictionary of variable names and values, with proper types!

pyiem.util.get_properties(cursor=None)

Fetch the properties set

Returns:

a dictionary of property names and values (both str)

Return type:

dict

pyiem.util.get_test_file(name)

Helper to get data for test usage.

pyiem.util.get_test_filepath(name: str) → str

Helper to get a testing filename, full path.

pyiem.util.grid_bounds(lons, lats, bounds)

Figure out indices that we can truncate big grid

Parameters:

  • lons (np.array) – grid lons

  • lats (np.array) – grid lats

  • bounds (list) – [x0, y0, x1, y1]

Returns:

[x0, y0, x1, y1]

pyiem.util.handle_date_err(exp, value, fmt)

Attempt to fix up a date string, when possible.

pyiem.util.html_escape(val)

Wrapper around cgi.escape deprecation.

pyiem.util.load_geodf(dataname: str, epsg: int = 4326)

Load a given bundled GeoDataFrame.

Parameters:

dataname (str) – The name of the dataset name to load.

Returns:

GeoDataFrame

pyiem.util.logger(name='pyiem', level=None)

Get pyiem’s logger with a stream handler attached.

Parameters:

  • name (str) – The name of the logger to get, default pyiem

  • level (logging.LEVEL) – The log level for this pyiem logget, default is WARNING for non interactive sessions, INFO otherwise

Returns:

logger instance

pyiem.util.mm2inch(val)

Helper to return magnitude of milimeters to inch conversion.

Parameters:

val (mixed) – something with values in mm

Returns:

something with values in inch

Return type:

val

pyiem.util.ncopen(ncfn, mode='r', timeout=60, _sleep=5)

Safely open netcdf files

The issue here is that we can only have the following situation for a given NetCDF file. 1. Only 1 or more readers 2. Only 1 appender

The netcdf is being accessed over NFS and perhaps local disk, so writing lock files is problematic.

Parameters:

  • ncfn (str) – The netCDF filename

  • mode (str,optional) – The netCDF4.Dataset open mode, default ‘r’

  • timeout (int) – The total time in seconds to attempt a read, default 60

Returns:

netCDF4.Dataset or None

pyiem.util.noaaport_text(text)

Make whatever text look like it is NOAAPort Pristine

Parameters:

text (string) – the inbound text

Returns:

text that looks noaaportish

pyiem.util.set_property(name, value, cursor=None)

Set a property value in the database.

Parameters:

  • name (str) – The name of the property to set

  • value (str,datetime) – The value to set

  • cursor (psycopg2.cursor) – Optional database cursor to use

pyiem.util.ssw(mixedobj)

python23 wrapper for sys.stdout.write

Parameters:

mixedobj (str or bytes) – what content we want to send

pyiem.util.utc(year=None, month=1, day=1, hour=0, minute=0, second=0, microsecond=0)

Create a datetime instance with tzinfo=timezone.utc

When no arguments are provided, returns datetime.now(timezone.utc).

Returns:

datetime with tzinfo set

pyiem.util.web2ldm(url, ldm_product_name, md5_from_name=False, pqinsert='pqinsert')

Download a URL and insert into LDM.

Implements a common IEM workflow whereby a web resource is downloaded, saved to a temporary file, and then inserted into LDM.

Parameters:

  • url (str) – Web resource to download.

  • ldm_product_name (str) – LDM product ID to use when inserting.

  • md5_from_name (bool) – Should pqinsert -i be used, which causes LDM to compute the MD5 value from the product name instead of data bytes.

  • pqinsert (str) – pqinsert command.

Returns:

bool - success of this workflow.

pyiem.webutil module

Utility functions for iemwebfarm applications.

class pyiem.webutil.CGIModel

Bases: BaseModel

A Pydantic model that parses CGI arguments.

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod xss_protect(v)

Protect against XSS attacks.

class pyiem.webutil.TELEMETRY(timing, status_code, client_addr, app, request_uri, vhost)

Bases: tuple

app

Alias for field number 3

client_addr

Alias for field number 2

request_uri

Alias for field number 4

status_code

Alias for field number 1

timing

Alias for field number 0

vhost

Alias for field number 5

pyiem.webutil.add_to_environ(environ, form, **kwargs)

Build out some things auto-parsed from the request.

pyiem.webutil.clean_form(form)

Opinionated cleaning of form data.

pyiem.webutil.compute_ts(form, suffix)

Figure out the timestamp.

pyiem.webutil.compute_ts_from_string(form, key)

Convert a string to a timestamp.

pyiem.webutil.ensure_list(environ, key, parse_commas=True) → list

Ensure that we get something that is at least an empty list.

Parameters:

  • environ – the WSGI environ

  • key – the key to look for

  • parse_commas – split each found value based on commas, default True.

pyiem.webutil.iemapp(**kwargs)

Attempt to do all kinds of nice things for the user and the developer.

kwargs:

  • default_tz: The default timezone to use for timestamps, the default is America/Chicago.

  • enable_telemetry: Enable telemetry logging, default True.

  • help: Default help text, default Help not available.

  • parse_times: Parse the form for timestamps, default True.

  • iemdb: (str or list) The database(s) to connect to, these will be bundled into the environ with keys of iemdb.<name>.conn and iemdb.<name>.cursor. No commit is performed. You can specify a single cursor name with iemdb_cursorname=<name>.

  • schema (BaseModel): A Pydantic model to parse the form with.

  • memcachekey (str or callable): A memcache key to use for caching the response. If the callable returns None, no caching is done.

  • memcacheexpire (int or callable): The number of seconds to cache the response, defaults to 3600.

  • content_type (str or callable): The content type to use for the response.

  • allowed_as_list (list): CGI parameters that are permitted to be lists.

What all this does:

  1. Attempts to catch database connection errors and handle nicely

  2. Updates environ with some auto-parsed values + form content.

  3. If the wrapped function returns a str or bytes, it will be encoded and made into a list for the WSGI response.

Notes

  • raise NoDataFound to have a nice error message generated

pyiem.webutil.log_request(environ)

Log the request to database for future processing.

pyiem.webutil.model_to_rst(model: BaseModel) → str

Convert a Pydantic model to a reStructuredText table.

Parameters:

model – The Pydantic model to convert

Returns: A reStructuredText table

pyiem.webutil.write_telemetry(data: TELEMETRY) → bool

Write telemetry to the database.

pyiem.wellknowntext module

Convert Well-Known Text format to python objects

Main entry point is the convert_well_known_text function which takes a geometry in Well-Known Text format and returns a python object with the geometry.

pyiem.wellknowntext.convert_well_known_text(wkt)

Return the geometry given in well-known text format as python objects

The function accepts only 2D data and supports the POINT, POLYGON, MULTIPOLYGON, LINESTRING and MULTILINESTRING geometries.

The structure of the return value depends on the geometry type. For MULTIPOLYGON and MULTILINESTRING return a list of lists of coordinate pairs. For POLYGON and LINESTRING return a list of coordinate pairs. For POINT return a coordinate pair. All coordinates are floats.

The string wkt may contain an SRID specification in addition to the actual geometry. This SRID is ignored.

pyiem.wellknowntext.parse_coordinate_lists(wkt)

Return the coordinates in wkt as a list of lists of coordinate pairs.

The wkt parameter is the coordinate part of a geometry in well-known text format.

pyiem.wellknowntext.parse_linestring(wkt)

Return the LINESTRING geometry in wkt as a list of float pairs

pyiem.wellknowntext.parse_multilinestring(wkt)

Return the MULTILINESTRING geometry wkt as a list of lists of float pairs

pyiem.wellknowntext.parse_multipolygon(wkt)

Return the MULTIPOLYGON geometry wkt as a list of lists of float pairs

pyiem.wellknowntext.parse_point(wkt)

Return the POINT geometry in wkt format as pair of floats

pyiem.wellknowntext.parse_polygon(wkt)

Return the POLYGON geometry in wkt as a list of float pairs

pyiem.windrose_utils module

util script to call windrose package

pyiem.windrose_utils.windrose(station, **kwargs)

Utility function that generates a windrose plot

Parameters:

  • station (str) – station identifier to search database for

  • database (str,optional) – database name to look for data within

  • months (list,optional) – optional list of months to limit plot to

  • hours (list,optional) – optional list of hours to limit plot to

  • sts (datetime,optional) – start datetime

  • ets (datetime,optional) – end datetime

  • units (str,optional) – units to plot values as, default to mph.

  • nsector (int,optional) – number of bins to devide the windrose into

  • justdata (boolean,optional) – if True, write out the data only

  • sname (str,optional) – The name of this station, if not specified it will default to the ((station)) identifier

  • sknt (list,optional) – A list of wind speeds in knots already generated

  • drct (list,optional) – A list of wind directions (deg N) already generated

  • valid (list,optional) – A list of valid datetimes (with tzinfo set), used in the case of providing sknt and drct.

  • level (int,optional) – In case of RAOB, which level interests us (hPa)

  • bins (list,optional) – bins to use for the wind speed

  • tzname (str,optional) – Time zone to use for the plot.

  • cmap (cmap,optional) – Matplotlib colormap to pass to barplot.

  • limit_by_doy (bool,optional) – Use the sts and ets to define a period of days each year to limit the data by. Default false.

  • plot_convention (str) – How to orient the bars. The default is the meteorological convention of from, the other option is to. This only impacts the plot, the provided data should still follow from.

Returns:

matplotlib.Figure instance or textdata

pyiem.wmo module

Very light weight WMO header parser.

class pyiem.wmo.WMOProduct(text, utcnow: datetime | None = None)

Bases: object

Base class for Products with a WMO Header.

parse_wmo()

Parse things related to the WMO header

Module contents

Python Utilities developed for/by Iowa Environmental Mesonet

Python is an important part of the Iowa Environmental Mesonet (IEM). This package is used by many parts of the IEM codebase and hopefully somewhat useful to others!?!?