sc_utils

sc_utils

Miscellaneous utilities for type checking, printing, dates and times, etc.

Note: there are a lot! The design philosophy has been that it’s easier to ignore a function that you don’t need than write one from scratch that you do need.

Highlights

Classes

Name Description
KeyNotFoundError A tiny class to fix repr for KeyErrors. KeyError prints the repr of the error
LazyModule Create a “lazy” module that is loaded if and only if an attribute is called.
Link A class to differentiate between an object and a link to an object. The idea
LinkException An exception to raise when links are broken, for exclusive use with the Link
autolist A simple extension to a list that defines add methods to simplify appending
tryexcept Simple class to catch exceptions in a single line

KeyNotFoundError

sc_utils.KeyNotFoundError()

A tiny class to fix repr for KeyErrors. KeyError prints the repr of the error message, rather than the actual message, so e.g. newline characters print as the character rather than the actual newline.

Example:

raise sc.KeyNotFoundError('The key "foo" is not available, but these are: "bar", "cat"')

LazyModule

sc_utils.LazyModule(module, variable, namespace=None, overwrite=True)

Create a “lazy” module that is loaded if and only if an attribute is called.

Typically not for use by the user, but is used by sc.importbyname().

Parameters

Name Type Description Default
module str name of the module to (not) load required
variable str variable name to assign the module to required
namespace dict the namespace to use (if not supplied, globals()) None
overwrite bool whether to allow overwriting an existing variable (by default, yes) True

Example:

pd = sc.LazyModule('pandas', 'pd') # pd is a LazyModule, not actually pandas
df = pd.DataFrame() # Not only does this work, but pd is now actually pandas

New in version 2.0.0.

LinkException

sc_utils.LinkException()

An exception to raise when links are broken, for exclusive use with the Link class.

autolist

sc_utils.autolist(*args)

A simple extension to a list that defines add methods to simplify appending and extension.

Examples:

ls = sc.autolist(3) # Quickly convert a scalar to a list

ls = sc.autolist()
for i in range(5):
    ls += i # No need for ls += [i]

tryexcept

sc_utils.tryexcept(message=None, die=None, catch=None, verbose=1, history=None)

Simple class to catch exceptions in a single line

Effectively an alias to contextlib.suppress(), which itself is a programmatic equivalent to using try-except blocks.

By default, all errors are caught. If catch is not None, then by default raise all other exceptions; if die is an exception (list of exceptions), then by default suppress all other exceptions.

Due to Python’s fundamental architecture, exceptions can only be caught inside a with statement, and the with block will exit immediately as soon as the first exception is encountered.

Parameters

Name Type Description Default
message str a custom message to print; “” will be replaced in the message with the text of the exception None
die bool / exception default behavior of whether to raise caught exceptions; or, exceptions to die on (others to catch) None
catch exception one or more exceptions to catch regardless of “die” None
verbose bool whether to print caught exceptions (0 = silent, 1 = error type, 2 = full error information) 1
history list / tryexcept a tryexcept object, or a list of exceptions, to keep the history (see example below) None

Examples:

# Basic usage
values = [0,1]
with sc.tryexcept(): # Equivalent to contextlib.suppress(Exception)
    values[2]

# Raise only certain errors
with sc.tryexcept(die=IndexError): # Catch everything except IndexError
    values[2]

# Catch (do not raise) only certain errors, and print full error information
with sc.tryexcept(catch=IndexError, verbose=2): # Raise everything except IndexError
    values[2]

# Storing the history of multiple exceptions
tryexc = None
for i in range(5):
    with sc.tryexcept(history=tryexc) as tryexc:
        print(values[i])
tryexc.traceback()

# With a custom message
with sc.tryexcept('Caught exception: <EXCEPTION>'):
    values[2]
  • New in version 2.1.0.
  • New in version 3.0.0: renamed “print” to “traceback”; added “to_df” and “disp” options
  • New in version 3.1.0: renamed “exceptions” to “data”; added “exceptions” property
  • New in version 3.2.3: “message” argument

Attributes

Name Description
died Whether or not any exceptions were encountered
exception Retrieve the last exception, if any
exceptions Retrieve the last exception, if any

Methods

Name Description
to_df Convert the exceptions to a dataframe; most useful with multiple exceptions
traceback Print the exception (usually the last)
to_df
sc_utils.tryexcept.to_df()

Convert the exceptions to a dataframe; most useful with multiple exceptions

traceback
sc_utils.tryexcept.traceback(which=None, tostring=False)

Print the exception (usually the last)

Parameters
Name Type Description Default
which int / list which exception(s) to print; if None, print all None
tostring bool whether to return as a string (otherwise print) False

New in version 3.1.0: optionally print multiple tracebacks

Functions

Name Description
asciify Convert an arbitrary Unicode string to ASCII.
checktype A convenience function for checking instances. If objtype is a type,
cp Shortcut to perform a shallow copy operation
dcp Shortcut to perform a deep copy operation
download Download one or more URLs in parallel and return output or save them to disk.
fast_uuid Create a fast UID or set of UIDs. Note: for certain applications, sc.uuid()
flexstr Try converting any object to a “regular” string (i.e. str), but proceed
getplatform Return the name of the current “main” platform (e.g. ‘mac’)
getuser Get the current username
htmlify Convert a string to its HTML representation by converting unicode characters,
ifelse For a list of inputs, return the first one that meets the condition
importbyname Import modules by name.
importbypath Import a module by path.
isarray Check whether something is a Numpy array, and optionally check the dtype.
isfunc Quickly check if something is a function.
isiterable Determine whether or not the input is iterable, with optional types to exclude.
isjupyter Check if a command is running inside a Jupyter notebook.
islinux Alias to sc.getplatform('linux')
ismac Alias to sc.getplatform('mac')
ismodule Determine whether or not the input is a module.
isnumber Determine whether or not the input is a number.
isstring Determine whether or not the input is string-like (i.e., str or bytes).
iswindows Alias to sc.getplatform('windows')
mergedicts Small function to merge multiple dicts together.
mergelists Merge multiple lists together.
newlinejoin Alias to strjoin(*args, sep='\n').
pp Shortcut for pretty-printing the object.
robust_dcp Ultra-robust deepcopying
runcommand Make it easier to run shell commands.
sanitizestr Remove all non-“standard” characters from a string
sha Shortcut for the standard hashing (SHA) method
strjoin Like string join(), but handles more flexible inputs, converts items to
strsplit Convenience function to split common types of strings.
suggest Return suggested item
swapdict Swap the keys and values of a dictionary. Equivalent to {v:k for k,v in d.items()}
toarray Small function to ensure consistent format for things that should be arrays
tolist Make sure object is always a list (note: sc.tolist()/sc.promotetolist() are identical).
traceback Shortcut for accessing the traceback
transposelist Convert e.g. a list of key-value tuples into a list of keys and a list of values.
uniquename Given a name and a list of other names, add a counter to the name so that
urlopen Download a single URL.
uuid Shortcut for creating a UUID; default is to create a UUID4. Can also convert a UUID.

asciify

sc_utils.asciify(
    string,
    form='NFKD',
    encoding='ascii',
    errors='ignore',
    **kwargs,
)

Convert an arbitrary Unicode string to ASCII.

Parameters

Name Type Description Default
form str the type of Unicode normalization to use 'NFKD'
encoding str the output string to encode to 'ascii'
errors str how to handle errors 'ignore'
kwargs dict passed to string.decode() {}

Example:

sc.asciify('föö→λ ∈ ℝ') # Returns 'foo  R'

New in version 2.0.1.

checktype

sc_utils.checktype(obj=None, objtype=None, subtype=None, die=False)

A convenience function for checking instances. If objtype is a type, then this function works exactly like isinstance(). But, it can also be one of the following strings:

- 'str', 'string': string or bytes object
- 'num', 'number': any kind of number
- 'arr', 'array': a Numpy array (equivalent to np.ndarray)
- 'listlike': a list, tuple, or array
- 'arraylike': a list, tuple, or array with numeric entries
- 'none': a None object

If subtype is not None, then checktype will iterate over the object and check recursively that each element matches the subtype.

Parameters

Name Type Description Default
obj any the object to check the type of None
objtype str or type the type to confirm the object belongs to None
subtype str or type optionally check the subtype if the object is iterable None
die bool whether or not to raise an exception if the object is the wrong type False

Examples:

sc.checktype(rand(10), 'array', 'number') # Returns True
sc.checktype(['a','b','c'], 'listlike') # Returns True
sc.checktype(['a','b','c'], 'arraylike') # Returns False
sc.checktype([{'a':3}], list, dict) # Returns True
  • New in version 2.0.1: pd.Series considered ‘array-like’
  • New in version 3.0.0: allow list (in addition to tuple) of types; allow checking for NoneType
  • New in version 3.1.3: handle exceptions when casting to “arraylike”

cp

sc_utils.cp(obj, die=True)

Shortcut to perform a shallow copy operation

Almost identical to copy.copy(), but optionally allow failures

  • New in version 2.0.0: default die=True instead of False
  • New in version 3.1.4: “verbose” argument removed; warning raised

dcp

sc_utils.dcp(obj, memo=None, die=True, verbose=True)

Shortcut to perform a deep copy operation

Almost identical to copy.deepcopy(), but optionally fall back to copy.copy() if deepcopy fails.

Parameters

Name Type Description Default
die bool if False, fall back to copy.copy() True
  • New in version 2.0.0: default die=True instead of False
  • New in version 3.1.4: die=False passed to sc.cp(); “verbose” argument removed; warning raised
  • New in version 3.2.0: “memo” argument
  • New in verison 3.2.4: fall back to sc.robust_dcp() instead of sc.cp(); “verbose” argument added

download

sc_utils.download(
    url,
    *args,
    filename=None,
    save=True,
    parallel=True,
    die=True,
    verbose=True,
    **kwargs,
)

Download one or more URLs in parallel and return output or save them to disk.

A parallelized wrapper for sc.urlopen(), except with save=True by default.

Parameters

Name Type Description Default
url str / list / dict either a single URL, a list of URLs, or a dict of key:URL or filename:URL pairs required
*args list additional URLs to download ()
filename str / list either a string or a list of the same length as url (if not supplied, return output) None
save bool if supplied instead of filename, then use the default filename True
parallel bool whether to download multiple URLs in parallel True
die bool whether to raise an exception if a URL can’t be retrieved (default true) True
verbose bool whether to print progress (if verbose=2, print extra detail on each downloaded URL) True
**kwargs dict passed to sc.urlopen() {}

Examples:

html = sc.download('http://sciris.org') # Download a single URL
data = sc.download('http://sciris.org', 'http://covasim.org', save=False) # Download two in parallel
sc.download({'sciris.html':'http://sciris.org', 'covasim.html':'http://covasim.org'}) # Download two and save to disk
sc.download(['http://sciris.org', 'http://covasim.org'], filename=['sciris.html', 'covasim.html']) # Ditto
data = sc.download(dict(sciris='http://sciris.org', covasim='http://covasim.org'), save=False) # Download and store in memory
  • New in version 2.0.0.
  • New in version 3.0.0: “die” argument
  • New in version 3.1.1: default order switched from URL:filename to filename:URL pairs
  • New in version 3.1.3: output as objdict instead of odict

fast_uuid

sc_utils.fast_uuid(
    which=None,
    length=None,
    n=1,
    secure=False,
    forcelist=False,
    safety=1000,
    recursion=0,
    recursion_limit=10,
    verbose=True,
)

Create a fast UID or set of UIDs. Note: for certain applications, sc.uuid() is faster than sc.fast_uuid()!

Parameters

Name Type Description Default
which str the set of characters to choose from (default ascii) None
length int length of UID (default 6) None
n int number of UIDs to generate 1
secure bool whether to generate random numbers from sources provided by the operating system False
forcelist bool whether or not to return a list even for a single UID (used for recursive calls) False
safety float ensure that the space of possible UIDs is at least this much larger than the number requested 1000
recursion int the recursion level of the call (since the function calls itself if not all UIDs are unique) 0
recursion_limit int Maximum number of times to try regeneraring keys 10
verbose bool whether to show progress True

Returns

Name Type Description
uid str or list a string UID, or a list of string UIDs

Example:

uuids = sc.fast_uuid(n=100) # Generate 100 UUIDs

Inspired by https://stackoverflow.com/questions/2257441/random-string-generation-with-upper-case-letters-and-digits/30038250#30038250

flexstr

sc_utils.flexstr(arg, *args, force=True, join='')

Try converting any object to a “regular” string (i.e. str), but proceed if it fails. Note: this function calls repr() rather than str() to ensure a more robust representation of objects.

Parameters

Name Type Description Default
arg any the object to convert to a string required
args list additional arguments ()
force bool whether to force it to be a string True
join str if multiple arguments are provided, the character to use to join ''

Example:

sc.flexstr(b'foo', 'bar', [1,2]) # Returns 'foobar[1, 2]'

New in version 3.0.0: handle multiple inputs

getplatform

sc_utils.getplatform(expected=None, platform=None, die=False)

Return the name of the current “main” platform (e.g. ‘mac’)

Alias to sys.platform, except maps entries onto one of ‘linux’, ‘windows’, ‘mac’, or ‘other’.

Parameters

Name Type Description Default
expected str if not None, check if the current platform is this None
platform str if supplied, map this onto one of the “main” platforms, rather than determine it None
die bool if True and expected is defined, raise an exception False

Returns

Name Type Description
String, one of: ‘linux’, ‘windows’, ‘mac’, or ‘other’

Examples:

sc.getplatform() # Get current name of platform
sc.getplatform('windows', die=True) # Raise an exception if not on Windows
sc.getplatform(platform='darwin') # Normalize to 'mac'

getuser

sc_utils.getuser()

Get the current username

Alias to getpass.getuser() – see https://docs.python.org/3/library/getpass.html#getpass.getuser

Example:

sc.getuser()

New in version 3.0.0.

htmlify

sc_utils.htmlify(string, reverse=False, tostring=False)

Convert a string to its HTML representation by converting unicode characters, characters that need to be escaped, and newlines. If reverse=True, will convert HTML to string. If tostring=True, will convert the bytestring back to Unicode.

Examples:

output = sc.htmlify('foo&\nbar') # Returns b'foo&amp;<br>bar'
output = sc.htmlify('föö&\nbar', tostring=True) # Returns 'f&#246;&#246;&amp;&nbsp;&nbsp;&nbsp;&nbsp;bar'
output = sc.htmlify('foo&amp;<br>bar', reverse=True) # Returns 'foo&\nbar'

ifelse

sc_utils.ifelse(*args, default=None, check=None)

For a list of inputs, return the first one that meets the condition

By default, returns the first non-None item, but can also check truth value or an arbitrary function.

Parameters

Name Type Description Default
args any the arguments to check (note: cannot supply a single list, use * to unpack) ()
default any the default value to use if no arguments meet the condition None
check func must be None (check if arguments are not None), bool (check if arguments evaluate True), or a callable (which returns True/False) None

Equivalent to next((arg for arg in args if check(arg)), default)

Examples:

# 1. Standard usage
a = None
b = 3
out = sc.ifelse(a, b)

## Equivalent to:
out = a if a is not None else b

# 2. Boolean usage
args = ['', False, {}, 'ok']
out = sc.ifelse(*args, check=bool)

## Equivalent to:
out = next((arg for arg in args if arg), None)

# 3. Custom function
args = [1, 3, 5, 7]
out = sc.ifelse(*args, check=lambda x: x>5)

## Equivalent to:
out = None
for arg in args:
  if arg > 5:
    out = val
    break
  • New in version 3.1.5.

importbyname

sc_utils.importbyname(
    module=None,
    variable=None,
    path=None,
    namespace=None,
    lazy=False,
    overwrite=True,
    die=True,
    verbose=True,
    **kwargs,
)

Import modules by name.

sc.importbyname(x='y') is equivalent to “import y as x”, but allows module importing to be done programmatically.

See https://peps.python.org/pep-0690/ for a proposal for incorporating something similar into Python by default.

Parameters

Name Type Description Default
module str name of the module to import None
variable str the name of the variable to assign the module to (by default, the module’s name) None
path str / path optionally load from path instead of by name None
namespace dict the namespace to load the modules into (by default, globals) None
lazy bool whether to create a LazyModule object instead of load the actual module False
overwrite bool whether to allow overwriting an existing variable (by default, yes) True
die bool whether to raise an exception if encountered True
verbose bool whether to print a warning if an module can’t be imported True
**kwargs dict additional variable:modules pairs to import (see examples below) {}

Examples:

np = sc.importbyname('numpy') # Standard usage
sc.importbyname(pd='pandas', np='numpy') # Use dictionary syntax to assign to namespace
plt = sc.importbyname(plt='matplotlib.pyplot', lazy=True) # Won't actually import until e.g. plt.figure() is called
mymod = sc.importbyname(path='/path/to/mymod') # Import by path rather than name

See also sc.importbypath().

  • New in version 2.1.0: “verbose” argument
  • New in version 3.0.0: “path” argument

importbypath

sc_utils.importbypath(path, name=None, overwrite=False)

Import a module by path.

Useful for importing multiple versions of the same module for comparison purposes.

Parameters

Name Type Description Default
path str / path the path to load the module from (with or without init.py) required
name str the name of the loaded module (by default, the file or folder name from the path) None
overwrite bool if True, load the module by the original name even if it exists (otherwise, generate a unique name, e.g. module1) False

Examples:

# Load a module that isn't importable otherwise
mymod = sc.importbypath('my file with spaces.py')

# Load two versions of the same module
old = sc.importbypath('/path/to/old/mylib')
new = sc.importbypath('/path/to/new/mylib')
assert new.__version__ > old.__version__ # Example version comparison (see also sc.compareverisons())

See also sc.importbyname().

  • New in version 3.0.0.
  • New in version 3.2.0: Allow importing two modules that have self imports (e.g. “import mylib” from within mylib)
  • New in version 3.2.1: “overwrite” argument

isarray

sc_utils.isarray(obj, dtype=None)

Check whether something is a Numpy array, and optionally check the dtype.

Almost the same as isinstance(obj, np.ndarray).

Example:

sc.isarray(np.array([1,2,3]), dtype=float) # False, dtype is int
  • New in version 1.0.0.
  • New in version 3.1.6: explicit False return

isfunc

sc_utils.isfunc(obj)

Quickly check if something is a function.

This checks whether or not something is a method (types.MethodType) or a function (types.FunctionType), which is different to whether or not it is callable (e.g., classes are callable).

Note: Python doesn’t have a crystal-clear distinction between things that are and aren’t functions, so there may be some edge cases with this function. For example, dict.fromkeys is a builtin_function_or_method, which returns True. So is list().pop and list().remove. But list.pop and list.remove are different types, and return False. The full list of types it checks against is:

- types.FunctionType
- types.MethodType
- types.BuiltinFunctionType
- types.BuiltinMethodType
- types.LambdaType
- functools.partial
- staticmethod
- classmethod

Example:

sc.isfunc(list) # Returns False
callable(list) # Returns True
  • New in version 3.2.0.
  • New in version 3.2.3: also checks for partial and lambda functions

isiterable

sc_utils.isiterable(obj, *args, exclude=None, minlen=None)

Determine whether or not the input is iterable, with optional types to exclude.

Parameters

Name Type Description Default
obj any object to check for iterability required
args any additional objects to check for iterability ()
exclude list list of iterable objects to exclude (e.g., strings) None
minlen int if not None, check that an object has a defined length as well None

Examples:

obj1 = [1,2,3]
obj2 = 'abc'
obj3 = set()

sc.isiterable(obj1) # Returns True
sc.isiterable(obj1, obj2, obj3, exclude=str, minlen=1) # returns [True, False, False]

See also numpy.iterable() for a simpler version.

New in version 3.0.0: “exclude” and “minlen” args; support multiple arguments

isjupyter

sc_utils.isjupyter(detailed=False)

Check if a command is running inside a Jupyter notebook.

Returns true/false if detailed=False, or a string for the exact type of notebook (e.g., Google Colab) if detailed=True.

Parameters

Name Type Description Default
detailed bool return a string of IPython/Jupyter type instead of true/false False
verbose bool print out additional information if IPython can’t be imported required

Examples:

if sc.isjupyter():
    sc.options(jupyter=True)

if sc.isjupyter(detailed=True) == 'colab':
    print('You are running on Google Colab')

New in version 3.0.0.

islinux

sc_utils.islinux(die=False)

Alias to sc.getplatform('linux')

ismac

sc_utils.ismac(die=False)

Alias to sc.getplatform('mac')

ismodule

sc_utils.ismodule(obj)

Determine whether or not the input is a module.

Equivalent to isinstance(obj, (types.ModuleType))

isnumber

sc_utils.isnumber(obj, isnan=None)

Determine whether or not the input is a number.

Identical to isinstance(obj, numbers.Number) unless isnan is specified.

Parameters

Name Type Description Default
obj any the object to check if it’s a number required
isnan bool an optional additional check to determine whether the number is/isn’t NaN None

isstring

sc_utils.isstring(obj)

Determine whether or not the input is string-like (i.e., str or bytes).

Equivalent to isinstance(obj, (str, bytes))

iswindows

sc_utils.iswindows(die=False)

Alias to sc.getplatform('windows')

mergedicts

sc_utils.mergedicts(
    *args,
    _strict=False,
    _overwrite=True,
    _copy=False,
    _sameclass=True,
    _die=True,
    **kwargs,
)

Small function to merge multiple dicts together.

By default, skips any input arguments that are None, and allows keys to be set multiple times. This function is similar to dict.update(), except it returns a value. The first dictionary supplied will be used for the output type (e.g. if the first dictionary is an odict, an odict will be returned).

Note that arguments start with underscores to avoid possible collisions with keywords (e.g. sc.mergedicts(dict(loose=True, strict=True), strict=False, _strict=True)).

Note: This function is similar to the “|” operator introduced in Python 3.9. However, sc.mergedicts() is useful for cases such as function arguments, where the default option is None but you will need a dict later on.

Parameters

Name Type Description Default
_strict bool if True, raise an exception if an argument isn’t a dict False
_overwrite bool if False, raise an exception if multiple keys are found True
_copy bool whether or not to deepcopy the merged dictionary False
_sameclass bool whether to ensure the output has the same type as the first dictionary merged True
_die bool whether to raise an exception if something goes wrong True
*args list the sequence of dicts to be merged ()
**kwargs dict merge these into the dict as well {}

Examples:

d0 = sc.mergedicts(user_args) # Useful if user_args might be None, but d0 is always a dict
d1 = sc.mergedicts({'a':1}, {'b':2}) # Returns {'a':1, 'b':2}
d2 = sc.mergedicts({'a':1, 'b':2}, {'b':3, 'c':4}, None) # Returns {'a':1, 'b':3, 'c':4}
d3 = sc.mergedicts(sc.odict({'b':3, 'c':4}), {'a':1, 'b':2}) # Returns sc.odict({'b':2, 'c':4, 'a':1})
d4 = sc.mergedicts({'b':3, 'c':4}, {'a':1, 'b':2}, _overwrite=False) # Raises exception
  • New in version 1.1.0: “copy” argument
  • New in version 1.3.3: keywords allowed
  • New in version 2.0.0: keywords fully enabled; “_sameclass” argument
  • New in version 2.0.1: fixed bug with “_copy” argument

mergelists

sc_utils.mergelists(*args, coerce='default', copy=False, **kwargs)

Merge multiple lists together.

Often used to flexible handle the input arguments to functions; see example below.

Parameters

Name Type Description Default
args any the lists, or items, to be joined together into a list ()
coerce str what types of objects to treat as lists; see sc.tolist() for details 'default'
copy bool whether to deepcopy the resultant object False
kwargs dict passed to sc.tolist(), which is called on each argument {}

Examples:

# Simple usage
sc.mergelists(None)                                # Returns []
sc.mergelists([1,2,3], [4,5,6])                    # Returns [1, 2, 3, 4, 5, 6]
sc.mergelists([1,2,3], 4, 5, 6)                    # Returns [1, 2, 3, 4, 5, 6]
sc.mergelists([(1,2), (3,4)], (5,6))               # Returns [(1, 2), (3, 4), (5, 6)]
sc.mergelists((1,2), (3,4), (5,6))                 # Returns [(1, 2), (3, 4), (5, 6)]
sc.mergelists((1,2), (3,4), (5,6), coerce='tuple') # Returns [1, 2, 3, 4, 5, 6]

# Usage for handling flexible input arguments
def my_func(arg=None, *args):
    arglist = sc.mergelists(arg, list(args))
    return arglist

a = my_func()           # Returns []
b = my_func([1,2,3])    # Returns [1,2,3]
c = my_func(1,2,3)      # Returns [1,2,3]
d = my_func([1,2], 3)   # Returns [1,2,3]
f = my_func(1, *[2,3])  # Returns [1,2,3]
e = my_func(1, [2,3])   # Returns [1,[2,3]] since second argument is ambiguous
g = my_func([[1,2]], 3) # Returns [[1,2],3] since first argument is nested

New in version 1.1.0.

newlinejoin

sc_utils.newlinejoin(*args)

Alias to strjoin(*args, sep='\n').

Example:

sc.newlinejoin([1,2,3], 4, 'five')

New in version 1.1.0.

pp

sc_utils.pp(
    obj,
    jsonify=False,
    doprint=None,
    output=False,
    sort_dicts=False,
    **kwargs,
)

Shortcut for pretty-printing the object.

Almost identical to pprint.pprint(), but can also be used as an alias for pprint.pformat().

Parameters

Name Type Description Default
obj any object to print required
jsonify bool whether to first convert the object to JSON, to handle things like ordered dicts nicely False
doprint bool whether to show output (default true) None
output bool whether to return output as a string (default false) False
sort_dicts bool whether to sort dictionary keys (default false, unlike pprint.pprint()) False
kwargs dict passed to pprint.pprint() {}

Example:

d = {'my very': {'large': 'and', 'unwieldy': {'nested': 'dictionary', 'cannot': 'be', 'easily': 'printed'}}}
sc.pp(d)

New in version 1.3.1: output argument New in version 3.0.0: “jsonify” defaults to False; sort_dicts defaults to False; removed “verbose” argument

robust_dcp

sc_utils.robust_dcp(obj, _memo=None, verbose=False)

Ultra-robust deepcopying

Deep-copy anything that can be deep-copied, then try a shallow copy of that attribute, and otherwise return the original object.

Co-authored with ChatGPT.

  • New in version 3.2.4.

runcommand

sc_utils.runcommand(
    command,
    printinput=False,
    printoutput=None,
    wait=True,
    **kwargs,
)

Make it easier to run shell commands.

Alias to subprocess.Popen(). Returns captured output if wait=True, else returns the subprocess.

Parameters

Name Type Description Default
command str the command to run required
printinput bool whether to print the input string False
printoutput bool whether to print the output (default: False if wait=True, True if wait=False) None
wait bool whether to wait for the process to return (else, return immediately with the subprocess) True

Examples:

myfiles = sc.runcommand('ls').split('\n') # Get a list of files in the current folder
sc.runcommand('sshpass -f %s scp myfile.txt me@myserver:myfile.txt' % 'pa55w0rd', printinput=True, printoutput=True) # Copy a file remotely
sc.runcommand('sleep 600; mkdir foo', wait=False) # Waits 10 min, then creates the folder "foo", but the function returns immediately
sc.runcommand('find', wait=False) # Equivalent to executing 'find' in a terminal

New in version 3.1.1: print real-time output if wait=False

sanitizestr

sc_utils.sanitizestr(
    string=None,
    alphanumeric=False,
    nospaces=False,
    asciify=False,
    lower=False,
    validvariable=False,
    spacechar='_',
    symchar='?',
)

Remove all non-“standard” characters from a string

Can be used to e.g. generate a valid variable name from arbitrary input, remove non-ASCII characters (replacing with equivalent ASCII ones if possible), etc.

Parameters

Name Type Description Default
string str the string to sanitize None
alphanumeric bool allow only alphanumeric characters False
nospaces bool remove spaces False
asciify bool remove non-ASCII characters False
lower bool convert uppercase characters to lowercase False
validvariable bool convert to a valid Python variable name (similar to alphanumeric=True, nospaces=True; uses spacechar to substitute) False
spacechar str if nospaces is True, character to replace spaces with (can be blank) '_'
symchar str character to replace non-alphanumeric characters with (can be blank) '?'

Examples:

string1 = 'This Is a String'
sc.sanitizestr(string1, lower=True) # Returns 'this is a string'

string2 = 'Lukáš wanted €500‽'
sc.sanitizestr(string2, asciify=True, nospaces=True, symchar='*') # Returns 'Lukas_wanted_*500*'

string3 = '"Ψ scattering", María said, "at ≤5 μm?"'
sc.sanitizestr(string3, asciify=True, alphanumeric=True, nospaces=True, spacechar='') # Returns '??scattering??Mariasaid??at?5?m??'

string4 = '4 path/names/to variable!'
sc.sanitizestr(string4, validvariable=True, spacechar='') # Returns '_4pathnamestovariable'

New in version 3.0.0.

sha

sc_utils.sha(obj, digest=False, asint=False, encoding='utf-8')

Shortcut for the standard hashing (SHA) method

Equivalent to hashlib.sha224().

Parameters

Name Type Description Default
obj any the object to be hashed; if not a string, converted to one required
digest bool if True, return the hex digest instead of the hash object False
asint bool if True, return the (very large) integer corresponding to the hex digest False
encoding str the encoding to use 'utf-8'

Example:

sha1 = sc.sha(dict(foo=1, bar=2), True)
sha2 = sc.sha(dict(foo=1, bar=2), digest=True)
sha3 = sc.sha(dict(foo=1, bar=3), digest=True)
assert sha1 == sha2
assert sha2 != sha3
  • New in version 3.2.0: “asint” argument; changed argument order

strjoin

sc_utils.strjoin(*args, sep=', ')

Like string join(), but handles more flexible inputs, converts items to strings. By default, join with commas.

Parameters

Name Type Description Default
args list the list of items to join ()
sep str the separator string ', '

Example:

sc.strjoin([1,2,3], 4, 'five')

New in version 1.1.0.

strsplit

sc_utils.strsplit(string, sep=None, skipempty=True, lstrip=True, rstrip=True)

Convenience function to split common types of strings.

Note: to use regular expressions, use re.split() instead.

Parameters

Name Type Description Default
string str the string to split required
sep str / list the types of separator to accept (default space or comma, i.e. [’ ‘,’,’]) None
skipempty bool whether to skip empty entries (i.e. from consecutive delimiters) True
lstrip bool whether to strip any extra spaces on the left True
rstrip bool whether to strip any extra spaces on the right True

Examples:

sc.strsplit('a b c') # Returns ['a', 'b', 'c']
sc.strsplit('a,b,c') # Returns ['a', 'b', 'c']
sc.strsplit('a, b, c') # Returns ['a', 'b', 'c']
sc.strsplit('  foo_bar  ', sep='_') # Returns ['foo', 'bar']

New in version 2.0.0.

suggest

sc_utils.suggest(
    user_input,
    valid_inputs,
    n=1,
    threshold=None,
    fulloutput=False,
    die=False,
    which='damerau',
)

Return suggested item

Returns item with lowest Levenshtein distance, where case substitution and stripping whitespace are not included in the distance. If there are ties, then the additional operations will be included.

Parameters

Name Type Description Default
user_input str User’s input required
valid_inputs list List/collection of valid strings required
n int Maximum number of suggestions to return 1
threshold int Maximum number of edits required for an option to be suggested (by default, two-thirds the length of the input; for no threshold, set to -1) None
fulloutput bool Whether to return suggestions and distances. False
die bool If True, an informative error will be raised (to avoid having to implement this in the calling code) False
which str Distance calculation method used; options are “damerau” (default), “levenshtein”, or “jaro” 'damerau'

Returns

Name Type Description
suggestions str or list Suggested string. Returns None if no suggestions with edit distance less than threshold were found. This helps to make suggestions more relevant.

Examples:

>>> sc.suggest('foo', ['Foo','Bar'])
'Foo'
>>> sc.suggest('foo', ['FOO','Foo'])
'Foo'
>>> sc.suggest('foo', ['Foo ','boo'])
'Foo '

swapdict

sc_utils.swapdict(d)

Swap the keys and values of a dictionary. Equivalent to {v:k for k,v in d.items()}

Parameters

Name Type Description Default
d dict dictionary required

Example:

d1 = {'a':'foo', 'b':'bar'}
d2 = sc.swapdict(d1) # Returns {'foo':'a', 'bar':'b'}

New in version 1.3.0.

toarray

sc_utils.toarray(x, keepnone=False, asobject=True, dtype=None, **kwargs)

Small function to ensure consistent format for things that should be arrays (note: sc.toarray() and sc.promotetoarray() are identical).

Very similar to numpy.array, with the main difference being that sc.toarray(3) will return np.array([3]) (i.e. a 1-d array that can be iterated over), while np.array(3) will return a 0-d array that can’t be iterated over.

Parameters

Name Type Description Default
x any a number or list of numbers required
keepnone bool whether sc.toarray(None) should return np.array([]) or np.array([None], dtype=object) False
asobject bool whether to prefer to coerce arrays to object type rather than string True
kwargs dict passed to numpy.array() {}

Examples:

sc.toarray(5) # Returns np.array([5])
sc.toarray([3,5]) # Returns np.array([3,5])
sc.toarray(None, skipnone=True) # Returns np.array([])
sc.toarray([1, 'foo']) # Returns np.array([1, 'foo'], dtype=object)
  • New in version 1.1.0: replaced “skipnone” with “keepnone”; allowed passing kwargs to np.array().
  • New in version 2.0.1: added support for pandas Series and DataFrame
  • New in version 3.1.0: “asobject” argument; cast mixed-type arrays to object rather than string by default

tolist

sc_utils.tolist(obj=None, objtype=None, keepnone=False, coerce='default')

Make sure object is always a list (note: sc.tolist()/sc.promotetolist() are identical).

Used so functions can handle inputs like 'a' or ['a', 'b']. In other words, if an argument can either be a single thing (e.g., a single dict key) or a list (e.g., a list of dict keys), this function can be used to do the conversion, so it’s always safe to iterate over the output.

While this usually wraps objects in a list rather than converts them to a list, the “coerce” argument can be used to change this behavior. Options are:

  • ‘none’ or None: do not coerce
  • ‘default’: coerce objects that were lists in Python 2 (range, map, dict_keys, dict_values, dict_items)
  • ‘tuple’: all the types in default, plus tuples
  • ‘full’: all the types in default, plus tuples and arrays

Parameters

Name Type Description Default
obj anything object to ensure is a list None
objtype anything optional type to check for each element; see sc.checktype() for details None
keepnone bool if keepnone is false, then None is converted to []; else, it’s converted to [None] False
coerce str / tuple tuple of additional types to coerce to a list (as opposed to wrapping in a list) 'default'

See also sc.mergelists() to handle multiple input arguments.

Examples:

sc.tolist(5) # Returns [5]
sc.tolist(np.array([3,5])) # Returns [np.array([3,5])] -- not [3,5]!
sc.tolist(np.array([3,5]), coerce=np.ndarray) # Returns [3,5], since arrays are coerced to lists
sc.tolist(None) # Returns []
sc.tolist(range(3)) # Returns [0,1,2] since range is coerced by default
sc.tolist(['a', 'b', 'c'], objtype='number') # Raises exception

def myfunc(data, keys):
    keys = sc.tolist(keys)
    for key in keys:
        print(data[key])

data = {'a':[1,2,3], 'b':[4,5,6]}
myfunc(data, keys=['a', 'b']) # Works
myfunc(data, keys='a') # Still works, equivalent to needing to supply keys=['a'] without tolist()
  • New in version 1.1.0: “coerce” argument
  • New in version 1.2.2: default coerce values
  • New in version 2.0.2: tuple coersion

traceback

sc_utils.traceback(
    exc=None,
    value=None,
    tb=None,
    verbose=False,
    *args,
    **kwargs,
)
Shortcut for accessing the traceback

Alias for `traceback.format_exc()`.

If no argument is provided, then use the last exception encountered.

Args:
    exc (Exception, tuple/list, or type): the exception to get the traceback from
    value (Exception): the actual exception
    tb (Traceback): the traceback
    verbose (bool): whether to print the exception

**Examples**:

```python
# Use automatic exception info
mylist = [0,1]
try:
    mylist[2]
except:
    print(f'Error: {sc.traceback()}')

# Supply exception manually (also illustrating sc.tryexcept())
with sc.tryexcept() as te1:
    dict(a=3)['b']

with sc.tryexcept() as te2:
    [0,1][2]

tb1 = sc.traceback(te1.exception)
tb2 = sc.traceback(te2.exception)
print(f'Tracebacks were:

{tb1} {tb2}’) ```

transposelist

sc_utils.transposelist(obj, fix_uneven=True)

Convert e.g. a list of key-value tuples into a list of keys and a list of values.

Parameters

Name Type Description Default
obj list the list-of-lists to be transposed required
fix_uneven bool append None values where needed so all input lists have the same length True

Examples:

o = sc.odict(a=1, b=4, c=9, d=16)
itemlist = o.enumitems()
inds, keys, vals = sc.transposelist(itemlist)

listoflists = [
    ['a', 1, 3],
    ['b', 4, 5],
    ['c', 7, 8, 9, 10]
]
trans = sc.transposelist(listoflists, fix_uneven=True)

New in version 1.1.0.

uniquename

sc_utils.uniquename(
    name=None,
    namelist=None,
    style=None,
    human=False,
    suffix=None,
)

Given a name and a list of other names, add a counter to the name so that it doesn’t conflict with the other names.

Useful for auto-incrementing filenames, etc.

Parameters

Name Type Description Default
name str the string to ensure is unique None
namelist list the list of strings that are taken None
style str a custom style for appending the counter that takes a single argument for the integer repeat; default ‘%d’ None
human bool if True, use ’ (%d)’ as the style instead of ‘%d’ False
suffix str if provided, remove this suffix from each name and add it back to the unique name None

Examples:

sc.uniquename('out', ['out', 'out1']) # Returns 'out2'
sc.uniquename(name='file', namelist=['file', 'file (1)', 'file (2)', 'myfile'], human=True) # Returns 'file (3)'
sc.uniquename('results.csv', ['results.csv', 'results1.csv'], suffix='.csv') # Returns 'results2.csv'
  • New in version 3.2.0: “human” and “suffix” arguments, simpler default style

urlopen

sc_utils.urlopen(
    url,
    filename=None,
    save=None,
    headers=None,
    params=None,
    data=None,
    prefix='http',
    convert=True,
    die=False,
    response='text',
    verbose=False,
)

Download a single URL.

Alias to urllib.request.urlopen(url).read(). See also sc.download() for downloading multiple URLs. Note: sc.urlopen()/sc.wget() are aliases.

Parameters

Name Type Description Default
url str the URL to open, either as GET or POST required
filename str if supplied, save to file instead of returning output None
save bool if supplied instead of filename, then use the default filename None
headers dict a dictionary of headers to pass None
params dict a dictionary of parameters to pass to the GET request None
prefix str the string to ensure the URL starts with (else, add it) 'http'
convert bool whether to convert from bytes to string True
die bool whether to raise an exception if converting to text failed False
response str what to return: ‘text’ (default), ‘json’ (dictionary version of the data), ‘status’ (the HTTP status), or ‘full’ (the full response object) 'text'
verbose bool whether to print progress False

Examples:

html = sc.urlopen('wikipedia.org') # Retrieve into variable html
sc.urlopen('http://wikipedia.org', filename='wikipedia.html') # Save to file wikipedia.html
sc.urlopen('https://wikipedia.org', save=True, headers={'User-Agent':'Custom agent'}) # Save to the default filename (here, wikipedia.org), with headers
sc.urlopen('wikipedia.org', response='status') # Only return the HTTP status of the site
  • New in version 2.0.0: renamed from wget to urlopen; new arguments
  • New in version 2.0.1: creates folders by default if they do not exist
  • New in version 2.0.4: “prefix” argument, e.g. prepend “http://” if not present
  • New in version 3.1.4: renamed “return_response” to “response”; additional options

uuid

sc_utils.uuid(
    uid=None,
    which=None,
    die=False,
    tostring=False,
    length=None,
    n=1,
    **kwargs,
)

Shortcut for creating a UUID; default is to create a UUID4. Can also convert a UUID.

Parameters

Name Type Description Default
uid str or uuid if a string, convert to an actual UUID; otherwise, return unchanged None
which int or str if int, choose a Python UUID function; otherwise, generate a random alphanumeric string (default 4) None
die bool whether to fail for converting a supplied uuid (default False) False
tostring bool whether or not to return a string instead of a UUID object (default False) False
length int number of characters to trim to, if returning a string None
n int number of UUIDs to generate; if n>1, return a list 1

Returns

Name Type Description
uid UUID or str the UID object

Examples:

sc.uuid() # Alias to uuid.uuid4()
sc.uuid(which='hex') # Creates a length-6 hex string
sc.uuid(which='ascii', length=10, n=50) # Creates 50 UUIDs of length 10 each using the full ASCII character set