sc_nested

sc_nested

Functions for working on nested (multi-level) dictionaries and objects.

Highlights

• sc.getnested(): get a value from a highly nested dictionary
• sc.search(): find a value in a nested object
• sc.equal(): check complex objects for equality

Classes

Name	Description
IterObj	Object iteration manager

IterObj

sc_nested.IterObj(
    obj,
    func=None,
    inplace=False,
    copy=False,
    leaf=False,
    recursion=0,
    depthfirst=True,
    atomic='default',
    skip=None,
    rootkey='root',
    verbose=False,
    iterate=True,
    custom_type=None,
    custom_iter=None,
    custom_get=None,
    custom_set=None,
    *args,
    **kwargs,
)

Object iteration manager

For arguments and usage documentation, see sc.iterobj(). Use this class only if you want more control over how the object is iterated over.

Class-specific args

iterate (bool): whether to do iteration upon object creation custom_type (func): a custom function for returning a string for a specific object type (should return None by default) custom_iter (func): a custom function for iterating (returning a list of keys) over an object custom_get (func): a custom function for getting an item from an object custom_set (func): a custom function for setting an item in an object

Example:

import sciris as sc

# Create a simple class for storing data
class DataObj(sc.prettyobj):
    def __init__(self, **kwargs):
        self.keys   = tuple(kwargs.keys())
        self.values = tuple(kwargs.values())

# Create the data
obj1 = DataObj(a=[1,2,3], b=[4,5,6])
obj2 = DataObj(c=[7,8,9], d=[10])
obj = DataObj(obj1=obj1, obj2=obj2)

# Define custom methods for iterating over tuples and the DataObj
def custom_iter(obj):
    if isinstance(obj, tuple):
        return enumerate(obj)
    if isinstance(obj, DataObj):
        return [(k,v) for k,v in zip(obj.keys, obj.values)]

# Define custom method for getting data from each
def custom_get(obj, key):
    if isinstance(obj, tuple):
        return obj[key]
    elif isinstance(obj, DataObj):
        return obj.values[obj.keys.index(key)]

# Gather all data into one list
all_data = []
def gather_data(obj, all_data=all_data):
    if isinstance(obj, list):
        all_data += obj

# Run the iteration
io = sc.IterObj(obj, func=gather_data, custom_type=(tuple, DataObj), custom_iter=custom_iter, custom_get=custom_get)
print(all_data)

• New in version 3.1.2.
• New in version 3.1.5: “norecurse” argument; better handling of atomic classes
• New in version 3.1.6: “depthfirst” argument; replace recursion with a queue; “to_df()” method
• New in version 3.2.1: improved recursion handling; “disp()” method
• New in version 3.2.4: gracefully handle objects that do not have a __dict__ attribute; handle slots

Methods

Name	Description
check_iter_type	Shortcut to check_iter_type()
check_proceed	Check if we should continue or not
disp	Display the full object
flatten_traces	Flatten the traces
getitem	Get the value for the item
indent	Print, with output indented successively
iterate	Actually perform the iteration over the object
iteritems	Return an iterator over items in this object
process_obj	Process a single object
setitem	Set the value for the item
to_df	Convert the output dictionary to a dataframe.

check_iter_type

sc_nested.IterObj.check_iter_type(obj)

Shortcut to check_iter_type()

check_proceed

sc_nested.IterObj.check_proceed(key, subobj, newid)

Check if we should continue or not

disp

sc_nested.IterObj.disp()

Display the full object

flatten_traces

sc_nested.IterObj.flatten_traces(sep='_', inplace=True)

Flatten the traces

getitem

sc_nested.IterObj.getitem(key, parent)

Get the value for the item

indent

sc_nested.IterObj.indent(string='', space='  ')

Print, with output indented successively

iterate

sc_nested.IterObj.iterate()

Actually perform the iteration over the object

iteritems

sc_nested.IterObj.iteritems(parent, trace)

Return an iterator over items in this object

process_obj

sc_nested.IterObj.process_obj(parent, trace, key, subobj, newid)

Process a single object

setitem

sc_nested.IterObj.setitem(key, value, parent)

Set the value for the item

to_df

sc_nested.IterObj.to_df(skip_root=True)

Convert the output dictionary to a dataframe.

Parameters

Name	Type	Description	Default
skip_root	bool	if True (default), only include the object’s subcomponents	True

Functions

Name	Description
equal	Compare equality between two arbitrary objects
flattendict	Flatten nested dictionary
getnested	Get the value for the given list of keys
iternested	Return a list of all the twigs in the current dictionary
iterobj	Iterate over an object and apply a function to each node (item with or without children).
makenested	Make or set a nested object (such as a dictionary).
mergenested	Merge different nested dictionaries
nestedloop	Zip list of lists in order
search	Find a key/attribute or value within a list, dictionary or object.
setnested	Set the value for the given list of keys; alias for sc.makenested().

equal

sc_nested.equal(
    obj,
    obj2,
    *args,
    method=None,
    detailed=False,
    equal_nan=True,
    leaf=False,
    union=True,
    verbose=None,
    die=False,
    **kwargs,
)

Compare equality between two arbitrary objects

This method parses two (or more) objects of any type (lists, dictionaries, custom classes, etc.) and determines whether or not they are equal. By default it returns true/false for whether or not the objects match, but it can also return a detailed comparison of exactly which attributes (or keys, etc) match or don’t match between the two objects. It works by first parsing the entire object into “leaves” via sc.iterobj(), and then comparing each “leaf” via one of the methods described below.

There is no universal way to check equality between objects in Python. Some objects define their own equals method which may not evaluate to true/false (e.g., Numpy arrays and pandas dataframes). For others it may be undefined. For this reasons, different ways of checking equality may give different results in edge cases. The available methods are:

- `'eq'`: uses the objects' built-in `__eq__()` methods (most accurate, but most likely to fail)
- `'pickle'`: converts the object to a binary pickle (most robust)
- `'json'`: converts the object to a JSON via `jsonpickle` (gives most detailed object structure, but can be lossy)
- `'str'`: converts the object to its string representation (least amount of detail)
- In addition, any custom function can be provided

By default, ‘eq’ is tried first, and if that raises an exception, ‘pickle’ is tried (equivalent to method=['eq', 'pickle']).

Parameters

Name	Type	Description	Default
obj	any	the first object to compare	required
obj2	any	the second object to compare	required
args	list	additional objects to compare	()
method	str	see above	None
detailed	int	whether to compute a detailed comparison of the objects, and return a dataframe of the results (if detailed=2, return the value of each object as well)	False
equal_nan	bool	whether matching np.nan should compare as true (default True; NB, False not guaranteed to work with method='pickle' or 'str', which includes the default; True not guaranteed to work with method='json')	True
leaf	bool	if True, only compare the object’s leaf nodes (those with no children); otherwise, compare everything	False
union	bool	if True, construct the comparison tree as the union of the trees of each object (i.e., an extra attribute in one object will show up as an additional row in the comparison; otherwise rows correspond to the attributes of the first object)	True
verbose	bool	level of detail to print	None
die	bool	whether to raise an exception if an error is encountered (else return False)	False
kwargs	dict	passed to sc.iterobj()	{}

Examples:

o1 = dict(
    a = [1,2,3],
    b = np.array([4,5,6]),
    c = dict(
        df = sc.dataframe(q=[sc.date('2022-02-02'), sc.date('2023-02-02')])
    )
)

# Identical object
o2 = sc.dcp(o1)

# Non-identical object
o3 = sc.dcp(o1)
o3['b'][2] = 8

sc.equal(o1, o2) # Returns True
sc.equal(o1, o3) # Returns False
e = sc.Equal(o1, o2, o3, detailed=True) # Create an object
e.df.disp() # Show results as a dataframe

• New in version 3.1.0.
• New in version 3.1.3: “union” argument; more detailed output

flattendict

sc_nested.flattendict(nesteddict, sep=None, _prefix=None)

Flatten nested dictionary

Parameters

Name	Type	Description	Default
nesteddict	dict	the dictionary to flatten	required
sep	str	the separator used to separate keys	None

Example:

>>> sc.flattendict({'a':{'b':1,'c':{'d':2,'e':3}}})
{('a', 'b'): 1, ('a', 'c', 'd'): 2, ('a', 'c', 'e'): 3}
>>> sc.flattendict({'a':{'b':1,'c':{'d':2,'e':3}}}, sep='_')
{'a_b': 1, 'a_c_d': 2, 'a_c_e': 3}

Args: nesteddict (dict): Input dictionary potentially containing dicts as values sep (str): Concatenate keys using string separator. If None the returned dictionary will have tuples as keys _prefix: Internal argument for recursively accumulating the nested keys

Returns

Name	Type	Description
		A flat dictionary where no values are dicts

New in version 2.0.0: handle non-string keys.

getnested

sc_nested.getnested(nested, keylist, safe=False, default=None)

Get the value for the given list of keys

Parameters

Name	Type	Description	Default
nested	any	the nested object (dict, list, or object) to get from	required
keylist	str / list / tuple	the keys to get (typically a list)	required
safe	bool	whether to return the “default” value if the key is not found	False
default	any	the value to return if the key is not found (sets safe=True if provided)	None

Example:

sc.getnested(foo, ['a','b']) # Gets foo['a']['b']

See sc.makenested() for full documentation.

iternested

sc_nested.iternested(nesteddict, _previous=None)

Return a list of all the twigs in the current dictionary

Parameters

Name	Type	Description	Default
nesteddict	dict	the dictionary	required

Example:

twigs = sc.iternested(foo)

See sc.makenested() for full documentation.

iterobj

sc_nested.iterobj(
    obj,
    func=None,
    inplace=False,
    copy=False,
    leaf=False,
    recursion=0,
    depthfirst=True,
    atomic='default',
    skip=None,
    rootkey='root',
    verbose=False,
    flatten=False,
    to_df=False,
    *args,
    **kwargs,
)

Iterate over an object and apply a function to each node (item with or without children).

Can modify an object in-place, or return a value. See also sc.search() for a function to search through complex objects.

By default, lists, dictionaries, and objects are iterated over. For custom iteration options, see sc.IterObj().

Note: there are three different output possibilities, depending on the keywords:

- `inplace=False`, `copy=False` (default): collate the output of the function into a flat dictionary, with keys corresponding to each node of the project
- `inplace=True`, `copy=False`: modify the actual object in-place, such that the original object is modified
- `inplace=True`, `copy=True`: make a deep copy of the object, modify that object, and return it (the original is unchanged)

Parameters

Name	Type	Description	Default
obj	any	the object to iterate over	required
func	function	the function to apply; if None, return a flat dictionary of all nodes in the object	None
inplace	bool	whether to modify the object in place (else, collate the output of the functions)	False
copy	bool	if modifying an object in place, whether to make a copy first	False
leaf	bool	whether to apply the function only to leaf nodes of the object	False
recursion	int	number of recursive steps to allow, i.e. parsing the same objects multiple times (default 0)	0
depthfirst	bool	whether to parse the object depth-first (default) or breadth-first	True
atomic	list	a list of known classes to treat as atomic (do not descend into); if ‘default’, use defaults (e.g. tuple, np.array, pd.DataFrame); if ‘default-tuple’, use defaults except for tuples	'default'
skip	list / dict	a list of objects to skip over entirely; can also be a dict with “keys”, “ids”, “subclasses”, and/or “instances”, which skip each of those	None
rootkey	str	the key to list as the root of the object (default 'root')	'root'
verbose	bool	whether to print progress	False
flatten	bool	whether to use flattened traces (single strings) rather than tuples	False
to_df	bool	whether to return a dataframe of the output instead of a dictionary (not valid with inplace=True)	False
*args	list	passed to func()	()
**kwargs	dict	passed to func()	{}

Examples:

data = dict(a=dict(x=[1,2,3], y=[4,5,6]), b=dict(foo='string', bar='other_string'))

# Search through an object
def check_int(obj):
    return isinstance(obj, int)

out = sc.iterobj(data, check_int)
print(out)

# Modify in place -- collapse mutliple short lines into one
def collapse(obj, maxlen):
    string = str(obj)
    if len(string) < maxlen:
        return string
    else:
        return obj

sc.printjson(data)
sc.iterobj(data, collapse, inplace=True, maxlen=10) # Note passing of keyword argument to function
sc.printjson(data)

• New in version 3.0.0.
• New in version 3.1.0: default func, renamed “twigs_only” to “leaf”, “atomic” argument
• New in version 3.1.2: copy defaults to False; refactored into class
• New in version 3.1.3: “rootkey” argument
• New in version 3.1.5: “recursion” argument; better handling of atomic classes
• New in version 3.1.6: “skip”, “depthfirst”, “to_df”, and “flatten” arguments

makenested

sc_nested.makenested(
    obj=None,
    keylist=None,
    value=None,
    overwrite=True,
    generator=None,
    copy=False,
)

Make or set a nested object (such as a dictionary).

Parameters

Name	Type	Description	Default
obj	any	the object to make the nested list in	None
keylist	list	a list of keys (strings) of the path to make	None
value	any	the value to set at the final key	None
overwrite	bool	if True, overwrite a value even if it exists	True
generator	class/func	the function used to create new levels of nesting (default: same as original object)	None
copy	bool	if True, copy the object before modifying it	False

Functions to get and set data from nested dictionaries (including objects).

sc.getnested() will get the value for the given list of keys:

sc.getnested(foo, [‘a’,‘b’])

sc.setnested will set the value for the given list of keys:

sc.setnested(foo, [‘a’,‘b’], 3)

sc.makenested will recursively update a dictionary with the given list of keys:

sc.makenested(foo, [‘a’,‘b’])

sc.iternested will return a list of all the twigs in the current dictionary:

twigs = sc.iternested(foo)

Example 1:

foo = {}
sc.makenested(foo, ['a','b'])
foo['a']['b'] = 3
print(sc.getnested(foo, ['a','b'])) # 3
sc.setnested(foo, ['a','b'], 7)
print(sc.getnested(foo, ['a','b'])) # 7
sc.makenested(foo, ['bar','cat'], value='in the hat')
print(foo['bar'])  # {'cat': 'in the hat'}

Example 2:

foo = {}
sc.makenested(foo, ['a','x'])
sc.makenested(foo, ['a','y'])
sc.makenested(foo, ['a','z'])
sc.makenested(foo, ['b','a','x'])
sc.makenested(foo, ['b','a','y'])
count = 0
for twig in sc.iternested(foo):
    count += 1
    sc.setnested(foo, twig, count)   # {'a': {'y': 1, 'x': 2, 'z': 3}, 'b': {'a': {'y': 4, 'x': 5}}}

Example 3:

foo = sc.makenested(sc.prettyobj(), ['level1', 'level2', 'level3'], 'done')
assert foo.level1.level2.level3 == 'done'

• New in version 2014nov29.
• New in version 3.2.0: operate on arbitrary objects; “overwrite” defaults to True; returns object
• New in version 3.2.3: explicitly replace setnested functionality

mergenested

sc_nested.mergenested(dict1, dict2, die=False, verbose=False, _path=None)

Merge different nested dictionaries

See sc.makenested() for full documentation.

Adapted from https://stackoverflow.com/questions/7204805/dictionaries-of-dictionaries-merge

nestedloop

sc_nested.nestedloop(inputs, loop_order)

Zip list of lists in order

This function takes in a list of lists to iterate over, and their nesting order. It then yields tuples of items in the given order. Only tested for two levels but in theory supports an arbitrary number of items.

Parameters

Name	Type	Description	Default
inputs	list	List of lists. All lists should have the same length	required
loop_order	list	Nesting order for the lists	required

Returns

Name	Type	Description
		Generator yielding tuples of items, one for each list

Example usage:

list(sc.nestedloop([[‘a’,‘b’],[1,2]],[0,1])) [[‘a’, 1], [‘a’, 2], [‘b’, 1], [‘b’, 2]]

Notice how the first two items have the same value for the first list while the items from the second list vary. If the loop_order is reversed, then:

list(sc.nestedloop([[‘a’,‘b’],[1,2]],[1,0])) [[‘a’, 1], [‘b’, 1], [‘a’, 2], [‘b’, 2]]

Notice now how now the first two items have different values from the first list but the same items from the second list.

From Atomica by Romesh Abeysuriya.

New in version 1.0.0.

search

sc_nested.search(
    obj,
    query=_None,
    key=_None,
    value=_None,
    type=_None,
    method='exact',
    **kwargs,
)

Find a key/attribute or value within a list, dictionary or object.

This function facilitates finding nested key(s) or attributes within an object, by searching recursively through keys or attributes. See sc.iterobj() for more detail.

Parameters

Name	Type	Description	Default
obj	any	A dict, list, or object	required
query	any	The key or value to search for (or a function or a type); equivalent to setting both key and value	_None
key	any	The key to search for	_None
value	any	The value to search for	_None
type	type	The type (or list of types) to match against (for values only)	_None
method	str	if the query is a string, choose how to check for matches: ‘exact’ (test equality), ‘partial’ (partial/lowercase string match), or ‘regex’ (treat as a regex expression)	'exact'
kwargs	dict	passed to sc.iterobj()	{}

Returns

Name	Type	Description
		A dictionary of matching attributes; like sc.iterobj(),
		but filtered to only include matches.

Examples:

# Create a nested dictionary
nested = {'a':{'foo':1, 'bar':['moat', 'goat']}, 'b':{'car':3, 'cat':[1,2,4,8]}}

# Find keys
keymatches = sc.search(nested, 'bar', flatten=True)

# Find values
val = 4
valmatches = sc.search(nested, value=val).keys()[0] # Returns  ('b', 'cat', 2)
assert sc.getnested(nested, valmatches) == val # Get from the original nested object

# Find values with a function
def find(v):
    return True if isinstance(v, int) and v >= 3 else False

found = sc.search(nested, value=find)

# Find partial or regex matches
found = sc.search(nested, value='oat', method='partial', leaf=True) # Search keys only
keys,vals = sc.search(nested, '^.ar', method='regex', verbose=True)

• New in version 3.0.0: ability to search for values as well as keys/attributes; “aslist” argument
• New in version 3.1.0: “query”, “method”, and “verbose” keywords; improved searching for lists
• New in version 3.2.0: allow type matching; removed “return_values”; renamed “aslist” to “flatten” (reversed)

setnested

sc_nested.setnested(obj=None, keylist=None, value=None, **kwargs)

Set the value for the given list of keys; alias for sc.makenested().

Parameters

Name	Type	Description	Default
obj	any	the nested object (dict, list, or object) to modify	None
keylist	list	the list of keys to use	None
value	any	the value to set	None
**kwargs	dict	passed to sc.makenested()	{}

Example:

sc.setnested(foo, ['a','b'], 3) # Sets foo['a']['b'] = 3

See sc.makenested() for full documentation.