sc_versioning

sc_versioning

Functions for checking and saving versioning information, such as Python package versions, git versions, etc.

Highlights

• sc.freeze(): programmatically store “pip freeze” output
• sc.require(): require a specific version of a package
• sc.gitinfo(): gets the git information (if available) of a given file
• sc.compareversions(): easy way to compare version numbers
• sc.metadata(): collects relevant metadata into a dictionary
• sc.savearchive(): saves data as a zip file including versioning metadata

Functions

Name	Description
compareversions	Function to compare versions, expecting both arguments to be a string of the
freeze	Alias for pip freeze.
getcaller	Try to get information on the calling function, but fail gracefully. See also
gitinfo	Retrieve git info
loadarchive	Load a zip file saved with sc.savearchive().
loadmetadata	Read metadata from a saved image; currently only PNG and SVG are supported.
metadata	Collect common metadata: useful for exactly recreating (or remembering) the environment
require	Check whether environment requirements are met. Alias to pkg_resources.require().
savearchive	Save any object as a pickled zip file, including metadata as a separate JSON file.

compareversions

sc_versioning.compareversions(version1, version2)

Function to compare versions, expecting both arguments to be a string of the format 1.2.3, but numeric works too. Returns 0 for equality, -1 for v1<v2, and 1 for v1>v2.

If version2 starts with >, >=, <, <=, or ==, the function returns True or False depending on the result of the comparison.

Examples:

sc.compareversions('1.2.3', '2.3.4') # returns -1
sc.compareversions(2, '2') # returns 0
sc.compareversions('3.1', '2.99') # returns 1
sc.compareversions('3.1', '>=2.99') # returns True
sc.compareversions(mymodule.__version__, '>=1.0') # common usage pattern
sc.compareversions(mymodule, '>=1.0') # alias to the above

New in version 1.2.1: relational operators

freeze

sc_versioning.freeze(lower=False)

Alias for pip freeze.

Parameters

Name	Type	Description	Default
lower	bool	convert all keys to lowercase	False

Example:

assert 'numpy' in sc.freeze() # One way to check for versions

• New in version 1.2.2.
• New in version 3.1.3: use importlib instead of pkg_resources

getcaller

sc_versioning.getcaller(
    frame=2,
    tostring=True,
    includelineno=False,
    includeline=False,
    relframe=0,
    die=False,
)

Try to get information on the calling function, but fail gracefully. See also sc.thisfile().

Frame 1 is the file calling this function, so not very useful. Frame 2 is the default assuming it is being called directly. Frame 3 is used if another function is calling this function internally.

Parameters

Name	Type	Description	Default
frame	int	how many frames to descend (e.g. the caller of the caller of the…), default 2	2
tostring	bool	whether to return a string instead of a dict with filename and line number	True
includelineno	bool	if tostring, whether to also include the line number	False
includeline	bool	if not tostring, also store the line contents	False
relframe	int	relative frame – another way of specifying the frame; added to “frame”	0
die	bool	whether to raise an exception if calling information cannot be retrieved	False

Returns

Name	Type	Description
output	str / dict	the filename (and line number) of the calling function, either as a string or dict

Examples:

sc.getcaller()
sc.getcaller(tostring=False)['filename'] # Equivalent to sc.getcaller()
sc.getcaller(frame=3) # Descend one level deeper than usual
sc.getcaller(frame=1, tostring=False, includeline=True) # See the line that called sc.getcaller()

• New in version 1.0.0.
• New in version 1.3.3: do not include line by default
• New in version 3.0.0: “relframe” argument; “die” argument

gitinfo

sc_versioning.gitinfo(path=None, hashlen=7, die=False, verbose=True)

Retrieve git info

This function reads git branch and commit information from a .git directory. Given a path, it will check for a .git directory. If the path doesn’t contain that directory, it will search parent directories for .git until it finds one. Then, the current information will be parsed.

Note: if direct directory reading fails, it will attempt to use the gitpython library.

Parameters

Name	Type	Description	Default
path	str	A folder either containing a .git directory, or with a parent that contains a .git directory	None
hashlen	int	Length of hash to return (default: 7)	7
die	bool	whether to raise an exception if git information can’t be retrieved (default: no)	False
verbose	bool	if not dying, whether to print information about the exception	True

Returns

Name	Type	Description
		Dictionary containing the branch, hash, and commit date

Examples:

info = sc.gitinfo() # Get git info for current script repository
info = sc.gitinfo(my_package.__file__) # Get git info for a particular Python package

loadarchive

sc_versioning.loadarchive(
    filename,
    folder=None,
    loadobj=True,
    loadmetadata=False,
    remapping=None,
    die=True,
    **kwargs,
)

Load a zip file saved with sc.savearchive().

Note: Since this function relies on pickle, it can potentially execute arbitrary code, so you should only use it with sources you trust. For more information, see: https://docs.python.org/3/library/pickle.html

Parameters

Name	Type	Description	Default
filename	str / path	the file load to (usually ends in .zip)	required
folder	str	optional additional folder to load from	None
loadobj	bool	whether to load the saved object	True
loadmetadata	bool	whether to load the metadata as well	False
remapping	dict	any known module remappings between the saved pickle version and the current libraries	None
die	bool	whether to fail if an exception is raised (else, just return the metadata)	True
kwargs	dict	passed to sc.load()	{}

Returns

Name	Type	Description
		If loadobj=True and loadmetadata=False, return the object;
		If loadobj=False and loadmetadata=True, return the metadata
		If loadobj=True and loadmetadata=True, return a dictionary of both

Example:

obj = MyClass() # Create an arbitrary object
sc.savearchive('my-class.zip', obj)

# Much later...
data = sc.loadarchive('my-class.zip', loadmetadata=True)
metadata, obj = data['metadata'], data['obj']

Note: This function expects the zip file to contain two files in it, one called “metadata.json” and one called “sciris_pickle.obj”. If you need to change these, you can manually modify sc.sc_versioning._metadata_filename and sc.sc_versioning._obj_filename, respectively. However, you almost certainly should not do so!

New in version 3.0.0.

loadmetadata

sc_versioning.loadmetadata(filename, load_all=False, die=True)

Read metadata from a saved image; currently only PNG and SVG are supported.

Only for use with images saved with sc.savefig(). Metadata retrieval for PDF is not currently supported. To load metadata saved with sc.metadata(), you can also use sc.loadjson() instead. To load metadata saved with sc.savearchive(), use sc.loadarchive() instead.

Parameters

Name	Type	Description	Default
filename	str	the name of the file to load the data from	required
load_all	bool	whether to load all metadata available in an image (else, just load what was saved by Sciris)	False
die	bool	whether to raise an exception if the metadata can’t be found	True

Example:

plt.plot([1,2,3], [4,2,6])
sc.savefig('example.png')
sc.loadmetadata('example.png')

metadata

sc_versioning.metadata(
    outfile=None,
    version=None,
    comments=None,
    require=None,
    pipfreeze=True,
    user=True,
    caller=True,
    git=True,
    asdict=False,
    tostring=False,
    relframe=0,
    **kwargs,
)

Collect common metadata: useful for exactly recreating (or remembering) the environment at a moment in time.

Parameters

Name	Type	Description	Default
outfile	str	if not None, then save as JSON to this filename	None
version	str	if supplied, the user-supplied version of the data being stored	None
comments	str / dict	additional comments on the data to store	None
require	str / dict	if provided, an additional manual set of requirements	None
pipfreeze	bool	store the current Python environment, equivalent to “pip freeze”	True
user	bool	store the username	True
caller	bool	store info on the calling file	True
git	bool	store git information on the calling file (branch, hash, etc.)	True
asdict	bool	construct as a dict instead of an objdict	False
tostring	bool	return a string rather than a dict	False
relframe	int	how far to descend into the calling stack (if used directly, use 0; if called by another function, use 1; etc)	0
kwargs	dict	any additional data to store (can be anything JSON-compatible)	{}

Returns

Name	Type	Description
		A dictionary with information on the date, plateform, executable, versions
		of key libraries (Sciris, Numpy, pandas, and Matplotlib), and the Python environment

Examples:

metadata = sc.metadata()
sc.compareversions(metadata.versions.pandas, '1.5.0')

sc.metadata('my-metadata.json') # Save to disk

New in version 3.0.0.

require

sc_versioning.require(
    reqs=None,
    *args,
    message=None,
    exact=False,
    detailed=False,
    die=True,
    warn=True,
    verbose=True,
    **kwargs,
)

Check whether environment requirements are met. Alias to pkg_resources.require().

Parameters

Name	Type	Description	Default
reqs	list / dict	a list of strings, or a dict of package names and versions	None
args	list	additional requirements	()
message	str	optionally provide a custom error message if requirements are not met; “” will be replaced with list of missing requirements	None
kwargs	dict	additional requirements	{}
exact	bool	use ‘==’ instead of ‘>=’ as the default comparison operator if not specified	False
detailed	bool	return a dict of which requirements are/aren’t met	False
die	bool	whether to raise an exception if requirements aren’t met	True
warn	bool	if not die, raise a warning if requirements aren’t met	True
verbose	bool	print out the exception if it’s not being raised or warned	True

Examples:

sc.require('numpy')
sc.require(numpy='')
sc.require(reqs={'numpy':'1.19.1', 'matplotlib':'3.2.2'})
sc.require('numpy>=1.19.1', 'matplotlib==3.2.2', die=False, message='Requirements <MISSING> not met, but continuing anyway')
sc.require(numpy='1.19.1', matplotlib='==4.2.2', die=False, detailed=True)

• New in version 1.2.2.
• New in version 3.0.0: “warn” argument
• New in version 3.1.3: “message” argument
• New in version 3.1.6: replace pkg_resources dependency with packaging

savearchive

sc_versioning.savearchive(
    filename,
    obj,
    files=None,
    folder=None,
    comments=None,
    require=None,
    user=True,
    caller=True,
    git=True,
    pipfreeze=True,
    method='dill',
    allow_nonzip=False,
    dumpargs=None,
    **kwargs,
)

Save any object as a pickled zip file, including metadata as a separate JSON file.

Pickles are usually not good for long-term data storage, since they rely on importing the libraries that were used to create the pickled object. This function partly addresses that by storing metadata along with the saved pickle. While there may still be issues opening the pickle, the metadata (which is stored separately) should give enough information to figure out how to reconstruct the original environment (allowing the pickle to be loaded, and then re-saved in a more persistent format if desired).

Note: Since this function relies on pickle, it can potentially execute arbitrary code, so you should only use it with sources you trust. For more information, see: https://docs.python.org/3/library/pickle.html

Parameters

Name	Type	Description	Default
filename	str / path	the file to save to (must end in .zip)	required
obj	any	the object to save	required
files	str / list	any additional files or folders to save	None
comments	str / dict	other comments/information to store in the metadata (must be JSON-compatible)	None
require	str / dict	if provided, an additional manual set of requirements	None
caller	bool	store information on the current user in the metadata (see sc.metadata())	True
caller	bool	store information on the calling file in the metadata (see sc.metadata())	True
git	bool	store the git version in the metadata (see sc.metadata())	True
pipfreeze	bool	store the output of “pip freeze” in the metadata (see sc.metadata())	True
method	str	the method to use saving the data; default “dill” for more robustness, but “pickle” is faster	'dill'
allow_nonzip	bool	whether to permit extensions other than .zip (note, may cause problems!)	False
dumpargs	dict	passed to sc.dumpstr()	None
kwargs	dict	passed to sc.savezip()	{}

Example:

obj = MyClass() # Create an arbitrary object
sc.savearchive('my-class.zip', obj)

# Much later...
obj = sc.loadarchive('my-class.zip')

New in version 3.0.0.