sc_fileio

sc_fileio

Functions for reading/writing to files, including pickles, JSONs, and Excel.

Highlights

• sc.save() / sc.load(): efficiently save/load any Python object (via pickling)
• sc.savetext() / sc.loadtext(): likewise, for text
• sc.savejson() / sc.loadjson(): likewise, for JSONs
• sc.saveyaml() / sc.saveyaml(): likewise, for YAML
• sc.thisdir(): get current folder
• sc.getfilelist(): easy way to access glob
• sc.rmpath(): remove files and folders

Classes

Name	Description
Blobject	A wrapper for a binary file – rarely used directly.
Failed	An empty class to represent a failed object loading. Not for use by the user.
Spreadsheet	A class for reading and writing Excel files in binary format. No disk IO needs
UnpicklingError	An error raised when unpickling an object fails
UnpicklingWarning	A warning raised when unpickling an object fails

Blobject

sc_fileio.Blobject(source=None, name=None, filename=None, blob=None)

A wrapper for a binary file – rarely used directly.

So named because it’s an object representing a blob.

“source” is a specification of where to get the data from. It can be anything supported by Blobject.load() which are (a) a filename, which will get loaded, or (b) a io.BytesIO which will get dumped into this instance

Alternatively, can specify blob which is a binary string that gets stored directly in the blob attribute

Methods

Name	Description
freshbytes	Refresh the bytes object to accept new data
load	This function loads the spreadsheet from a file or object. If no input argument is supplied,
save	This function writes the spreadsheet to a file on disk.
tofile	Return a file-like object with the contents of the file.

freshbytes

sc_fileio.Blobject.freshbytes()

Refresh the bytes object to accept new data

load

sc_fileio.Blobject.load(source=None)

This function loads the spreadsheet from a file or object. If no input argument is supplied, then it will read self.bytes, assuming it exists.

save

sc_fileio.Blobject.save(filename=None)

This function writes the spreadsheet to a file on disk.

tofile

sc_fileio.Blobject.tofile(output=True)

Return a file-like object with the contents of the file.

This can then be used to open the workbook from memory without writing anything to disk e.g.

- book = openpyxl.load_workbook(self.tofile())
- book = xlrd.open_workbook(file_contents=self.tofile().read())

Failed

sc_fileio.Failed(*args, **kwargs)

An empty class to represent a failed object loading. Not for use by the user.

• New in version 3.1.0: combined Failed and UniversalFailed classes
• New in version 3.2.0: added isempty() check

Methods

Name	Description
isempty	Check wether anything at all loaded

isempty

sc_fileio.Failed.isempty()

Check wether anything at all loaded

Spreadsheet

sc_fileio.Spreadsheet(*args, **kwargs)

A class for reading and writing Excel files in binary format. No disk IO needs to happen to manipulate the spreadsheets with openpyxl (or xlrd or pandas).

New version 1.3.0: Changed default from xlrd to openpyxl and added self.wb attribute to avoid the need to reload workbooks.

Examples::

Methods

Name	Description
new	Shortcut method to create a new openpyxl workbook
openpyexcel	Legacy name for openpyxl()
openpyxl	Return a book as opened by openpyxl
pandas	Return a book as opened by pandas
readcells	Alias to loadspreadsheet()
update	Updated the stored spreadsheet with book instead
writecells	Specify cells to write. Can supply either a list of cells of the same length
xlrd	Legacy method to load from xlrd

new

sc_fileio.Spreadsheet.new(**kwargs)

Shortcut method to create a new openpyxl workbook

openpyexcel

sc_fileio.Spreadsheet.openpyexcel(*args, **kwargs)

Legacy name for openpyxl()

openpyxl

sc_fileio.Spreadsheet.openpyxl(reload=False, store=True, **kwargs)

Return a book as opened by openpyxl

pandas

sc_fileio.Spreadsheet.pandas(reload=False, store=True, **kwargs)

Return a book as opened by pandas

readcells

sc_fileio.Spreadsheet.readcells(wbargs=None, *args, **kwargs)

Alias to loadspreadsheet()

update

sc_fileio.Spreadsheet.update(book)

Updated the stored spreadsheet with book instead

writecells

sc_fileio.Spreadsheet.writecells(
    cells=None,
    startrow=None,
    startcol=None,
    vals=None,
    sheetname=None,
    sheetnum=None,
    verbose=False,
    wbargs=None,
)

Specify cells to write. Can supply either a list of cells of the same length as the values, or else specify a starting row and column and write the values from there.

Examples:

S = sc.Spreadsheet()
S.writecells(cells=['A6','B7'], vals=['Cat','Dog']) # Method 1
S.writecells(cells=[np.array([2,3])+i for i in range(2)], vals=['Foo', 'Bar']) # Method 2
S.writecells(startrow=14, startcol=1, vals=np.random.rand(3,3)) # Method 3
S.save('myfile.xlsx')

xlrd

sc_fileio.Spreadsheet.xlrd(reload=False, store=True, **kwargs)

Legacy method to load from xlrd

UnpicklingError

sc_fileio.UnpicklingError()

An error raised when unpickling an object fails

New in version 3.1.0.

UnpicklingWarning

sc_fileio.UnpicklingWarning()

A warning raised when unpickling an object fails

New in version 3.1.0.

Functions

Name	Description
dumpstr	Dump an object to a bytes-like string (rarely used by the user); see sc.save()
getfilelist	A shortcut for using glob.glob().
getfilepaths	Alias for sc.getfilelist() that returns paths by default instead of strings.
ispath	Alias to isinstance(obj, Path).
jsonify	This is the main conversion function for Python data-structures into JSON-compatible
jsonpickle	Save any Python object to a JSON using jsonpickle.
jsonunpickle	Open a saved JSON pickle
load	Load a file that has been saved as a gzipped pickle file, e.g. by sc.save().
loadany	Load data from a file using all known load functions until one works.
loadjson	Convenience function for reading a JSON file (or string).
loadspreadsheet	Load a spreadsheet as a dataframe or a list of lists.
loadstr	Like sc.load(), but for a bytes-like string (rarely used).
loadtext	Convenience function for reading a text file
loadyaml	Convenience function for reading a YAML file (or string).
loadzip	Load the contents of a zip file into a variable.
makefilepath	Utility for taking a filename and folder – or not – and generating a
makepath	Alias for sc.makefilepath() that returns a path by default instead of a string
path	Alias to pathlib.Path() with some additional input sanitization:
printjson	Print an object as a JSON
readjson	Read JSON from a string
readyaml	Read YAML from a string
rmpath	Remove file(s) and folder(s). Alias to os.remove() (for files) and shutil.rmtree()
sanitizefilename	Takes a potentially Linux- and Windows-unfriendly candidate file name, and
sanitizepath	Alias for sc.sanitizefilename() that returns a path by default instead of a string.
save	Save any object to disk
savejson	Convenience function for saving to a JSON file.
savespreadsheet	Semi-simple function to save data nicely to Excel.
savetext	Convenience function for saving a text file – accepts a string or list of strings;
saveyaml	Convenience function for saving to a YAML file.
savezip	Create a zip file from the supplied list of files (or less commonly, supplied data)
thisdir	Tiny helper function to get the folder for a file, usually the current file.
thisfile	Return the full path of the current file.
thispath	Alias for sc.thisdir() that returns a path by default instead of a string.
unzip	Convenience function for reading a zip file
zsave	Save a file using zstandard (instead of gzip) compression. This is an alias

dumpstr

sc_fileio.dumpstr(obj=None, **kwargs)

Dump an object to a bytes-like string (rarely used by the user); see sc.save() instead.

Parameters

Name	Type	Description	Default
obj	any	the object to convert	None
kwargs	dict	passed to sc.save()	{}

New in version 3.0.0: uses sc.save() for more robustness

getfilelist

sc_fileio.getfilelist(
    folder=None,
    pattern=None,
    fnmatch=None,
    abspath=False,
    nopath=False,
    filesonly=False,
    foldersonly=False,
    recursive=True,
    aspath=None,
)

A shortcut for using glob.glob().

Note that sc.getfilelist() and sc.glob() are aliases of each other.

Parameters

Name	Type	Description	Default
folder	str	the folder to find files in (default, current)	None
pattern	str	the pattern to match (default, wildcard); can be excluded if part of the folder	None
fnmatch	str	optional additional string to filter results by	None
abspath	bool	whether to return the full path	False
nopath	bool	whether to return no path	False
filesonly	bool	whether to only return files (not folders)	False
foldersonly	bool	whether to only return folders (not files)	False
recursive	bool	passed to glob.glob() (note: ** is required as the pattern to match all subfolders)	True
aspath	bool	whether to return Path objects (if None, use sc.options.aspath)	None

Returns

Name	Type	Description
		List of files/folders

Examples:

sc.getfilelist() # return all files and folders in current folder
sc.getfilelist('~/temp', '*.py', abspath=True) # return absolute paths of all Python files in ~/temp folder
sc.getfilelist('~/temp/*.py') # Like above
sc.getfilelist(fnmatch='*.py') # Recursively find all files ending in .py

• New in version 1.1.0: “aspath” argument
• New in version 2.1.0: default pattern of “**“;”fnmatch” argument; default recursive=True
• New in version 3.2.1: avoid blank entries

getfilepaths

sc_fileio.getfilepaths(*args, aspath=True, **kwargs)

Alias for sc.getfilelist() that returns paths by default instead of strings.

New version 2.1.0.

ispath

sc_fileio.ispath(obj)

Alias to isinstance(obj, Path).

New in version 2.0.0.

jsonify

sc_fileio.jsonify(
    obj,
    verbose=True,
    die=False,
    tostring=False,
    custom=None,
    strkeys=True,
    **kwargs,
)

This is the main conversion function for Python data-structures into JSON-compatible data structures (note: sc.sanitizejson()/sc.jsonify() are identical).

Parameters

Name	Type	Description	Default
obj	any	almost any kind of data structure that is a combination of list, numpy.ndarray, odicts, etc.	required
verbose	bool	level of detail to print	True
die	bool	whether or not to raise an exception if conversion failed (otherwise, return a string)	False
tostring	bool	whether to return a string representation of the sanitized object instead of the object itself	False
custom	dict	custom functions for dealing with particular object types	None
strkeys	bool	whether to coerce all dictionary keys to strings (otherwise, leave numbers, bools, and other valid YAML types)	True
kwargs	dict	passed to json.dumps() if tostring=True	{}

Returns

Name	Type	Description
object	any or str	the converted object that should be JSON compatible, or its representation as a string if tostring=True

Examples:

data = dict(a=np.random.rand(3), b=dict(foo='cat', bar='dog'))
json = sc.jsonify(data)
jsonstr = sc.jsonify(data, tostring=True, indent=2)

# Use a custom function for parsing the data
custom = {np.ndarray: lambda x: f'It was an array: {x}'}
j2 = sc.jsonify(data, custom=custom)

• New in version 3.2.4: “strkeys” argument; don’t coerce dict keys to strings by default

jsonpickle

sc_fileio.jsonpickle(obj, filename=None, tostring=False, **kwargs)

Save any Python object to a JSON using jsonpickle.

Wrapper for the jsonpickle library: https://jsonpickle.github.io/

Note: unlike regular pickle, this is not guaranteed to exactly restore the original object. For example, at the time of writing it does not support pandas dataframes with mixed-dtype columns. If this sort of thing does not sound like it would be an issue for you, please proceed!

Parameters

Name	Type	Description	Default
obj	any	the object to pickle as a JSON	required
tostring	bool	whether to return a string (rather than the JSONified Python object)	False
kwargs	dict	passed to jsonpickle.pickler.Pickler()	{}

Returns

Name	Type	Description
		Either a Python object for the JSON, a string, or save to file

Examples:

# Create data
df1  = sc.dataframe(a=[1,2,3], b=['a','b','c'])

# Convert to JSON and read back
json = sc.jsonpickle(df1)
df2  = sc.jsonunpickle(json)

# Save to JSON and load back
sc.jsonpickle(df1, 'my-data.json')
df3  = sc.jsonunpickle('my-data.json')

New in version 3.1.0: “filename” argument

jsonunpickle

sc_fileio.jsonunpickle(json=None, filename=None)

Open a saved JSON pickle

See sc.jsonpickle() for full documentation.

Parameters

Name	Type	Description	Default
json	str or object	if supplied, restore the data from a string or object	None
filename	str / path	if supplied, restore data from file	None

New in version 3.1.0: “filename” argument

load

sc_fileio.load(
    filename=None,
    folder=None,
    verbose=None,
    die=False,
    remapping=None,
    method=None,
    auto_remap=True,
    **kwargs,
)

Load a file that has been saved as a gzipped pickle file, e.g. by sc.save(). Accepts either a filename (standard usage) or a file object as the first argument. Note that sc.load()/sc.loadobj() are aliases of each other.

Note 1: Since this function relies on pickle, it can potentially execute arbitrary code, so you should only use it with sources you trust. For more information, see: https://docs.python.org/3/library/pickle.html

Note 2: When a pickle file is loaded, Python imports any modules that are referenced in it. This is a problem if module has been renamed. In this case, you can use the remapping argument to point to the new modules or classes. For more robustness, use the sc.savearchive()/ sc.loadarchive() functions.

Parameters

Name	Type	Description	Default
filename	str / Path	the filename (or full path) to load	None
folder	str / Path	the folder (not needed if the filename includes it)	None
verbose	bool	print nothing (False), critical warnings (None), or full detail (True)	None
die	bool	whether to raise an exception if errors are encountered (otherwise, load as much as possible via the ‘robust’ method)	False
remapping	dict	way of mapping old/unavailable module names to new (see below for example)	None
method	str	method for loading (‘pickle’, ‘dill’, ‘pandas’, or ‘robust’; if None, try all)	None
auto_remap	bool	whether to use known deprecations to load failed pickles	True
kwargs	dict	passed to pickle.loads()/dill.loads()	{}

Examples:

obj = sc.load('myfile.obj') # Standard usage
old = sc.load('my-old-file.obj', method='dill', ignore=True) # Load classes from saved files
old = sc.load('my-old-file.obj', remapping={'foo.Bar': cat.Mat}) # If loading a saved object containing a reference to foo.Bar that is now cat.Mat
old = sc.load('my-old-file.obj', remapping={('foo', 'Bar'): ('cat', 'Mat')}, method='robust') # Equivalent to the above but force remapping and don't fail
old = sc.load('my-old-file.obj', remapping={'foo.Bar': None}) # Skip mapping foo.Bar and don't fail

• New in version 1.1.0: “remapping” argument
• New in version 1.2.2: ability to load non-gzipped pickles; support for dill; arguments passed to loader
• New in version 3.1.0: improved handling of pickling failures
• New in version 3.1.1: allow remapping to None

loadany

sc_fileio.loadany(filename, folder=None, verbose=False, **kwargs)

Load data from a file using all known load functions until one works.

Known formats are: pickle, JSON, YAML, Excel, CSV, zip, or plain text.

Parameters

Name	Type	Description	Default
filename	str / path	the name of the file to load	required
folder	str	optional additional folder for the filename	None
verbose	bool	print out the details of the process (verbose=2 to show errors)	False
kwargs	dict	passed to the load function	{}

Example:

data = sc.odict()
datafiles = ['headers.json', 'some-data.csv', 'more-data.xlsx', 'final-data.obj']
for datafile in datafiles:
    data[datafile] = sc.loadany(datafile)

• New in version 3.2.0.

loadjson

sc_fileio.loadjson(
    filename=None,
    folder=None,
    string=None,
    fromfile=True,
    encoding='utf-8',
    **kwargs,
)

Convenience function for reading a JSON file (or string).

Parameters

Name	Type	Description	Default
filename	str	the file to load, or the JSON object if using positional arguments	None
folder	str	folder if not part of the filename	None
string	str	if not loading from a file, a string representation of the JSON	None
fromfile	bool	whether or not to load from file	True
encoding	str	the file encoding (default UTF-8, as required by the JSON standard)	'utf-8'
kwargs	dict	passed to json.load()	{}

Returns

Name	Type	Description
output	dict	the JSON object

Examples:

json = sc.loadjson('my-file.json')
json = sc.loadjson(string='{"a":null, "b":[1,2,3]}')

See also sc.readjson() for loading a JSON from a string.

New in version 3.3.0: default to UTF-8 encoding

loadspreadsheet

sc_fileio.loadspreadsheet(
    filename=None,
    folder=None,
    fileobj=None,
    sheet=0,
    header=1,
    asdataframe=None,
    method='pandas',
    **kwargs,
)

Load a spreadsheet as a dataframe or a list of lists.

By default, an alias to pandas.read_excel() with a header, but also supports loading via openpyxl or xlrd. Read from either a filename or a file object.

Parameters

Name	Type	Description	Default
filename	str	filename or path to read	None
folder	str	optional folder to use with the filename	None
fileobj	obj	load from file object rather than path	None
sheet	str / int / list	name or number of sheet(s) to use (default 0)	0
asdataframe	bool	whether to return as a pandas/Sciris dataframe (default True)	None
header	bool	whether the 0-th row is to be read as the header	1
method	str	how to read (default ‘pandas’, other choices ‘openpyxl’ and ‘xlrd’)	'pandas'
kwargs	dict	passed to pd.read_excel(), openpyxl(), etc.	{}

Examples:

df = sc.loadspreadsheet('myfile.xlsx') # Alias to pd.read_excel(header=1)
wb = sc.loadspreadsheet('myfile.xlsx', method='openpyxl') # Returns workbook
data = sc.loadspreadsheet('myfile.xlsx', method='xlrd', asdataframe=False) # Returns raw data; requires xlrd

New version 1.3.0: change default from xlrd to pandas; renamed sheetname and sheetnum arguments to sheet.

loadstr

sc_fileio.loadstr(string, **kwargs)

Like sc.load(), but for a bytes-like string (rarely used).

Parameters

Name	Type	Description	Default
string	str	the bytes-like string to load	required
kwargs	dict	passed to sc.load()	{}

Example:

obj = sc.objdict(a=1, b=2)
bytestring = sc.dumpstr(obj)
obj2 = sc.loadstr(bytestring)
assert obj == obj2

• New in version 3.0.0: uses sc.load() for more robustness

loadtext

sc_fileio.loadtext(
    filename=None,
    folder=None,
    splitlines=False,
    encoding='utf-8',
)

Convenience function for reading a text file

Parameters

Name	Type	Description	Default
filename	str / path	the file to load	None
folder	str	folder if not part of the filename	None
splitlines	bool	whether to return a list of lines rather than a single string	False
encoding	str	the file encoding (default UTF-8, rather than the platform-dependent default)	'utf-8'

Example:

mytext = sc.loadtext('my-document.txt')

New in version 3.3.0: default to UTF-8 encoding

loadyaml

sc_fileio.loadyaml(
    filename=None,
    folder=None,
    string=None,
    fromfile=True,
    safe=False,
    loader=None,
    encoding='utf-8',
)

Convenience function for reading a YAML file (or string).

Parameters

Name	Type	Description	Default
filename	str	the file to load, or the YAML object if using positional arguments	None
folder	str	folder if not part of the filename	None
string	str	if not loading from a file, a string representation of the YAML	None
fromfile	bool	whether or not to load from file	True
safe	bool	whether to use the safe loader	False
loader	Loader	custom YAML loader (takes precedence over safe)	None
encoding	str	the file encoding (default UTF-8, as required by the YAML standard)	'utf-8'

Returns

Name	Type	Description
output	dict	the YAML object

Examples:

yaml = sc.loadyaml('my-file.yaml')
yaml = sc.loadyaml(string='{"a":null, "b":[1,2,3]}')

New in version 3.3.0: default to UTF-8 encoding

loadzip

sc_fileio.loadzip(filename=None, folder=None, load=True, convert=True, **kwargs)

Load the contents of a zip file into a variable.

See also sc.load() for loading a gzipped file.

Parameters

Name	Type	Description	Default
filename	str / path	the name of the zip file to load from	None
folder	str	optional additional folder for the filename	None
load	bool	whether to load the contents of the zip file; else just return the ZipFile itself	True
convert	bool	whether to convert bytes objects to strings	True
kwargs	dict	passed to sc.load()	{}

Returns

Name	Type	Description
		dict with each file loaded as a key

Example:

data = sc.loadzip('my-files.zip')

• New in version 2.0.0.
• New in version 3.0.0: load into memory instead of extracting to disk; see sc.unzip() for extracting
• New in version 3.1.4: optionally return just the zipfile object; convert bytes to string
• New in version 3.2.1: load gzip as well as zip files

makefilepath

sc_fileio.makefilepath(
    filename=None,
    folder=None,
    ext=None,
    default=None,
    split=False,
    aspath=None,
    abspath=True,
    makedirs=False,
    checkexists=None,
    sanitize=False,
    die=True,
    verbose=False,
)

Utility for taking a filename and folder – or not – and generating a valid path from them. By default, this function will combine a filename and folder using os.path.join, create the folder(s) if needed with os.makedirs, and return the absolute path.

Note: in most cases sc.makepath() should be used instead.

Parameters

Name	Type	Description	Default
filename (str or Path)		the filename, or full file path, to save to – in which case this utility does nothing	required
folder (str/Path/list)		the name of the folder to be prepended to the filename; if a list, fed to os.path.join()	required
ext (str)		the extension to ensure the file has	required
default (str or list)		a name or list of names to use if filename is None	required
split (bool)		whether to return the path and filename separately	required
aspath (bool)		whether to return a Path object (default: set by sc.options.aspath)	required
abspath (bool)		whether to conver to absolute path	required
makedirs (bool)		whether or not to make the folders to save into if they don’t exist	required
checkexists (bool)		if False/True, raises an exception if the path does/doesn’t exist	required
sanitize (bool)		whether or not to remove special characters from the path; see sc.sanitizepath() for details	required
die (bool)		whether or not to raise an exception if cannot create directory failed (otherwise, return a string)	required
verbose (bool)		how much detail to print	required

Returns

Name	Type	Description
filepath	str or Path	the validated path (or the folder and filename if split=True)

Simple example:

filepath = sc.makefilepath('myfile.obj') # Equivalent to os.path.abspath(os.path.expanduser('myfile.obj'))

Complex example:

filepath = makefilepath(filename=None, folder='./congee', ext='prj', default=[project.filename, project.name], split=True, abspath=True, makedirs=True)

Assuming project.filename is None and project.name is “recipe” and ./congee doesn’t exist, this will make folder ./congee and returns e.g. (‘/home/myname/congee’, ‘recipe.prj’)

• New in version 1.1.0: “aspath” argument
• New in version 3.0.0: “makedirs” defaults to False

makepath

sc_fileio.makepath(*args, aspath=True, **kwargs)

Alias for sc.makefilepath() that returns a path by default instead of a string (with apologies for the confusing terminology, kept for backwards compatibility).

New version 2.1.0.

path

sc_fileio.path(*args, **kwargs)

Alias to pathlib.Path() with some additional input sanitization:

- `None` entries are removed
- a list of arguments is converted to separate arguments

Examples:

sc.path('thisfile.py') # Returns PosixPath('thisfile.py')

sc.path('/a/folder', None, 'a_file.txt') # Returns PosixPath('/a/folder/a_file.txt')

• New in version 1.2.2.
• New in version 2.0.0: handle None or list arguments

printjson

sc_fileio.printjson(obj, indent=2, **kwargs)

Print an object as a JSON

Acts as an alias to print(sc.jsonify(..., tostring=True)).

Parameters

Name	Type	Description	Default
obj	any	the object to print	required
indent	int	the level of indent to use	2
kwargs	dict	passed to sc.jsonify()	{}

Example:

data = dict(a=dict(x=[1,2,3], y=[4,5,6]), b=dict(foo='string', bar='other_string'))
sc.printjson(data)

New in version 3.0.0.

readjson

sc_fileio.readjson(string, **kwargs)

Read JSON from a string

Alias to json.loads().

Parameters

Name	Type	Description	Default
string	str	a string representation of the JSON	required
kwargs	dict	passed to json.loads()	{}

See also sc.loadjson() for loading a JSON from a file.

Example:

string = '{"this":1, "is":2, "a":3, "JSON":4}'
json = sc.readjson(string)

New in version 3.0.0.

readyaml

sc_fileio.readyaml(string, **kwargs)

Read YAML from a string

Alias to sc.loadyaml(string=...).

Parameters

Name	Type	Description	Default
string	str	a string representation of the YAML	required
kwargs	dict	passed to sc.loadyaml()	{}

See also sc.loadyaml() for loading a YAML from a file.

Example:

string = '{"this":1, "is":2, "a":3, "YAML":4} # YAML allows comments!'
yaml = sc.readyaml(string)

New in version 3.0.0.

rmpath

sc_fileio.rmpath(
    path=None,
    *args,
    die=True,
    verbose=True,
    interactive=False,
    **kwargs,
)

Remove file(s) and folder(s). Alias to os.remove() (for files) and shutil.rmtree() (for folders).

Parameters

Name	Type	Description	Default
path	str / Path / list	file, folder, or list to remove	None
args	list	additional paths to remove	()
die	bool	whether or not to raise an exception if cannot remove	True
verbose	bool	how much detail to print	True
interactive	bool	whether to confirm prior to each deletion	False
kwargs	dict	passed to os.remove()/shutil.rmtree()	{}

Examples:

sc.rmpath('myobj.obj') # Remove a single file
sc.rmpath('myobj1.obj', 'myobj2.obj', 'myobj3.obj') # Remove multiple files
sc.rmpath(['myobj.obj', 'tests']) # Remove a file and a folder interactively
sc.rmpath(sc.getfilelist('tests/*.obj')) # Example of removing multiple files

New version 2.0.0.

sanitizefilename

sc_fileio.sanitizefilename(
    filename,
    sub='_',
    allowspaces=False,
    asciify=True,
    strict=False,
    disallowed=None,
    aspath=False,
)

Takes a potentially Linux- and Windows-unfriendly candidate file name, and returns a “sanitized” version that is more usable.

Parameters

Name	Type	Description	Default
filename	str	the filename to sanitize	required
sub	str	the character to substitute unsafe input characters with	'_'
allowspaces	bool	whether to allow spaces in the filename	False
asciify	bool	whether to convert the string from Unicode to ASCII	True
strict	bool	whether to remove (almost) all non-alphanumeric characters	False
disallowed	str	optionally supply a custom list of disallowed characters	None
aspath	bool	whether to return a Path object	False

Example:

bad = 'Nöt*a   file&name?!.doc'
good = sc.sanitizefilename(bad)

• New version 2.0.1: arguments “sub”, “allowspaces”, “asciify”, “strict”, and “disallowed”
• New version 3.1.1: disallow tabs and newlines even when strict=False

sanitizepath

sc_fileio.sanitizepath(*args, aspath=True, **kwargs)

Alias for sc.sanitizefilename() that returns a path by default instead of a string.

New version 2.1.0.

save

sc_fileio.save(
    filename='default.obj',
    obj=None,
    folder=None,
    method='pickle',
    compression='gzip',
    compresslevel=5,
    verbose=0,
    sanitizepath=True,
    die=False,
    allow_empty=False,
    **kwargs,
)

Save any object to disk

This function is similar to pickle.dump() in that it serializes the object to a file. Key differences include:

- It takes care of opening/closing the file for writing
- It compresses the data by default
- It supports different serialization methods (e.g. pickle or dill)

Once an object is saved, it can be loaded with sc.load(). Note that sc.save()/sc.saveobj() are identical.

Note 1: Since this function relies on pickle, it can potentially execute arbitrary code, so you should only use it with sources you trust. For more information, see: https://docs.python.org/3/library/pickle.html

Note 2: When a pickle file is loaded, Python imports any modules that are referenced in it. This is a problem if module has been renamed (in which case the pickle usually can’t be opened). For more robustness (e.g. to pickle custom classes), use method='dill' and/or the sc.savearchive()/sc.loadarchive() functions.

If you do not need to save arbitrary Python and just need to save data, consider saving the data in a standard format, e.g. JSON (sc.savejson()). For large amounts of tabular data, also consider formats like HDF5 or PyArrow.

Parameters

Name	Type	Description	Default
filename (str/path)		the filename or path to save to; if None, return an io.BytesIO filestream instead of saving to disk	required
obj (anything)		the object to save	required
folder (str)		optional additional folder, passed to sc.makepath()	required
method (str)		whether to use ‘pickle’ (default) or ‘dill’	required
compression (str)		type of compression to use: ‘gzip’ (default), ‘zstd’ (zstandard), or ‘none’ (no compression)	required
compresslevel (int)		the level of gzip/zstd compression (1 to 9 for gzip, -7 to 22 for zstandard, default 5)	required
verbose (int)		level of detail to print	required
sanitizepath (bool)		whether to sanitize the path prior to saving	required
die (bool)		whether to fail if the object can’t be pickled (else, try dill); if die is ‘never’	required
allow_empty (bool)		whether or not to allow “None” to be saved (usually treated as an error)	required
kwargs (dict)		passed to pickle.dumps() (or dill.dumps())	required

Examples:

# Standard usage
my_obj = ['this', 'is', 'my', 'custom', {'object':44}]
sc.save('myfile.obj', my_obj)
loaded = sc.load('myfile.obj')
assert loaded == my_obj

# Use dill instead, to save custom classes as well
class MyClass:
    def __init__(self, x):
        self.data = np.random.rand(100) + x
    def sum(self):
        return self.data.sum()
my_class = MyClass(10)

sc.save('my_class.obj', my_class, method='dill', compression='zstd')
loaded = sc.load('my_class.obj') # With dill, can be loaded anywhere, not just in the same script
assert loaded.sum() == my_class.sum()

See also sc.zsave() (identical except defaults to zstandard compression).

• New in version 1.1.1: removed Python 2 support.
• New in version 1.2.2: automatic swapping of arguments if order is incorrect; correct passing of arguments
• New in version 2.0.4: “die” argument for saving as dill
• New in version 2.1.0: “zstandard” compression method
• New in version 3.0.0: “allow_empty” argument; removed “args”

savejson

sc_fileio.savejson(
    filename=None,
    obj=None,
    folder=None,
    die=True,
    indent=2,
    keepnone=False,
    sanitizepath=True,
    encoding='utf-8',
    **kwargs,
)

Convenience function for saving to a JSON file.

Parameters

Name	Type	Description	Default
filename	str	the file to save	None
obj	anything	the object to save; if not already in JSON format, conversion will be attempted	None
folder	str	folder if not part of the filename	None
die	bool	whether or not to raise an exception if saving an empty object	True
indent	int	indentation to use for saved JSON	2
keepnone	bool	allow sc.savejson(None) to return ‘null’ rather than raising an exception	False
sanitizepath	bool	whether to sanitize the path prior to saving	True
encoding	str	the file encoding (default UTF-8, as required by the JSON standard)	'utf-8'
kwargs	dict	passed to json.dump()	{}

Returns

Name	Type	Description
		The filename saved to

Example:

json = {'foo':'bar', 'data':[1,2,3]}
sc.savejson('my-file.json', json)

See also sc.jsonify().

New in version 3.3.0: default to UTF-8 encoding

savespreadsheet

sc_fileio.savespreadsheet(
    filename=None,
    data=None,
    folder=None,
    sheetnames=None,
    close=True,
    workbook_args=None,
    formats=None,
    formatdata=None,
    verbose=False,
)

Semi-simple function to save data nicely to Excel.

Parameters

Name	Type	Description	Default
filename	str	Excel file to save to	None
data	list / array	data to write to the spreadsheet	None
folder	str	if supplied, merge with the filename to make a path	None
sheetnames	list	if data is supplied as a list of arrays, save each entry to a different sheet	None
close	bool	whether to close the workbook after saving	True
workbook_args	dict	arguments passed to xlxwriter.Workbook()	None
formats	dict	a definition of different types of formatting (see examples below)	None
formatdata	array	an array of which formats go where	None
verbose	bool	whether to print progress	False

Examples:

import numpy as np
import sciris as sc
import matplotlib.pyplot as plt

# Simple example
testdata1 = np.random.rand(8,3)
sc.savespreadsheet(filename='test1.xlsx', data=testdata1)

# Include column headers
test2headers = [['A','B','C']] # Need double brackets to get right shape
test2values = np.random.rand(8,3).tolist()
testdata2 = test2headers + test2values
sc.savespreadsheet(filename='test2.xlsx', data=testdata2)

# Multiple sheets
testdata3 = [np.random.rand(10,10), np.random.rand(20,5)]
sheetnames = ['Ten by ten', 'Twenty by five']
sc.savespreadsheet(filename='test3.xlsx', data=testdata3, sheetnames=sheetnames)

# Supply data as an odict
testdata4 = sc.odict([('First sheet', np.random.rand(6,2)), ('Second sheet', np.random.rand(3,3))])
sc.savespreadsheet(filename='test4.xlsx', data=testdata4)

# Include formatting
nrows = 15
ncols = 3
formats = {
    'header':{'bold':True, 'bg_color':'#3c7d3e', 'color':'#ffffff'},
    'plain': {},
    'big':   {'bg_color':'#ffcccc'}
}
testdata5  = np.zeros((nrows+1, ncols), dtype=object) # Includes header row
formatdata = np.zeros((nrows+1, ncols), dtype=object) # Format data needs to be the same size
testdata5[0,:] = ['A', 'B', 'C'] # Create header
testdata5[1:,:] = np.random.rand(nrows,ncols) # Create data
formatdata[1:,:] = 'plain' # Format data
formatdata[testdata5>0.7] = 'big' # Find "big" numbers and format them differently
formatdata[0,:] = 'header' # Format header
sc.savespreadsheet(filename='test5.xlsx', data=testdata5, formats=formats, formatdata=formatdata)

New version 2.0.0: allow arguments to be passed to the Workbook.

savetext

sc_fileio.savetext(filename=None, string=None, encoding='utf-8', **kwargs)

Convenience function for saving a text file – accepts a string or list of strings; can also save an arbitrary object, in which case it will first convert to a string.

Parameters

Name	Type	Description	Default
filename	str	the filename to save to	None
string	str	the string (or object) to save	None
encoding	str	the file encoding (default UTF-8, rather than the platform-dependent default)	'utf-8'
kwargs	dict	passed to np.savetxt() if saving an array	{}

Example:

text = ['Here', 'is', 'a', 'poem']
sc.savetext('my-poem.txt', text)

New in version 3.1.0: fixed bug with saving a list of strings

New in version 3.3.0: default to UTF-8 encoding

saveyaml

sc_fileio.saveyaml(
    filename=None,
    obj=None,
    folder=None,
    jsonify=True,
    sort_keys=True,
    die=True,
    keepnone=False,
    dumpall=False,
    sanitizepath=True,
    encoding='utf-8',
    **kwargs,
)

Convenience function for saving to a YAML file.

Parameters

Name	Type	Description	Default
filename	str	the file to save (if empty, return string representation of the YAML instead)	None
obj	anything	the object to save	None
folder	str	folder if not part of the filename	None
jsonify	bool	whether to convert the object to a JSON prior to saving to YAML (typically preserves more of the object structure)	True
sort_keys	bool	whether to sort the keys	True
die	bool	whether or not to raise an exception if saving an empty object	True
indent	int	indentation to use for saved YAML	required
keepnone	bool	allow sc.saveyaml(None) to return ‘null’ rather than raising an exception	False
dumpall	bool	if True, treat a list input as separate YAML pages	False
sanitizepath	bool	whether to sanitize the path prior to saving	True
encoding	str	the file encoding (default UTF-8, as required by the YAML standard)	'utf-8'
kwargs	dict	passed to yaml.dump()	{}

Returns

Name	Type	Description
		The filename saved to

Examples:

yaml = {'foo':'bar', 'data':[1,2,3]}
sc.saveyaml('my-file.yaml', yaml, sort_keys=False) # Save to file and do not sort the keys

string = sc.saveyaml(obj=yaml) # Export to string

New in version 3.3.0: default to UTF-8 encoding

savezip

sc_fileio.savezip(
    filename=None,
    files=None,
    data=None,
    folder=None,
    sanitizepath=True,
    basename=False,
    tobytes=True,
    verbose=True,
    **kwargs,
)

Create a zip file from the supplied list of files (or less commonly, supplied data)

Parameters

Name	Type	Description	Default
filename	str / path	the name of the zip file to write to	None
files	list	file(s) and/or folder(s) to compress	None
data	dict	if supplied, write this data as well or instead (must be a dictionary of filename keys and data values)	None
folder	str	optional additional folder for the filename	None
sanitizepath	bool	whether to sanitize the path prior to saving	True
basename	bool	whether to use only the file’s basename as the name inside the zip file (otherwise, store folder info)	False
tobytes	bool	if data is provided, convert it automatically to bytes (otherwise, up to the user)	True
verbose	bool	whether to print progress	True
kwargs	dict	passed to sc.save()	{}

Examples:

scripts = sc.getfilelist('./code/*.py')
sc.savezip('scripts.zip', scripts)

sc.savezip('mydata.zip', data=dict(var1='test', var2=np.random.rand(3)))

• New in version 2.0.0: saving data
• New in version 3.0.0: “tobytes” argument and kwargs; “filelist” renamed “files”

thisdir

sc_fileio.thisdir(file=None, path=None, *args, frame=1, aspath=None, **kwargs)

Tiny helper function to get the folder for a file, usually the current file. If not supplied, then use the current file.

Parameters

Name	Type	Description	Default
file	str	the file to get the directory from; usually file	None
path	str / list	additional path to append; passed to os.path.join()	None
args	list	also passed to os.path.join()	()
frame	int	if file is None, which frame to pull the folder from (default 1, the file that calls this function)	1
aspath	bool	whether to return a Path object instead of a string	None
kwargs	dict	passed to Path()	{}

Returns

Name	Type	Description
filepath	str	the full path to the folder (or filename if additional arguments are given)

Examples:

thisdir = sc.thisdir() # Get folder of calling file
thisdir = sc.thisdir('.') # Ditto (usually)
thisdir = sc.thisdir(__file__) # Ditto (usually)
file_in_same_dir = sc.thisdir(path='new_file.txt')
file_in_sub_dir = sc.thisdir('..', 'tests', 'mytests.py') # Merge parent folder with sufolders and a file
np_dir = sc.thisdir(np) # Get the folder that Numpy is loaded from (assuming "import numpy as np")

• New in version 1.1.0: “as_path” argument renamed “aspath”
• New in version 1.2.2: “path” argument
• New in version 1.3.0: allow modules
• New in version 2.1.0: frame argument

thisfile

sc_fileio.thisfile(frame=1, aspath=None)

Return the full path of the current file.

Parameters

Name	Type	Description	Default
frame	int	which frame to pull the filename from (default 1, the file that calls this function)	1
aspath	bool	whether to return a Path object	None

Examples:

my_script_name = sc.thisfile() # Get the name of the current file
calling_script = sc.thisfile(frame=2) # Get the name of the script that called this script

New in verison 2.1.0.

thispath

sc_fileio.thispath(*args, frame=1, aspath=True, **kwargs)

Alias for sc.thisdir() that returns a path by default instead of a string.

New in version 2.1.0.

unzip

sc_fileio.unzip(filename=None, outfolder='.', folder=None, members=None)

Convenience function for reading a zip file

Parameters

Name	Type	Description	Default
filename	str / path	the name of the zip file to write to	None
outfolder	str / path	the path location to extract the files to (default: current folder)	'.'
folder	str	optional additional folder for the filename	None
members	list	optional list of members	None

Returns

Name	Type	Description
		list of the names of the unzipped files

Example:

sc.unzip('my-files.zip', outfolder='my_data') # extracts all files

• New in version 3.0.0 (equivalent to sc.loadzip(…, extract=True) previously)

zsave

sc_fileio.zsave(*args, compression='zstd', **kwargs)

Save a file using zstandard (instead of gzip) compression. This is an alias for sc.save(..., compression='zstd'); see sc.save() for details.

Note: there is no matching function “zload()” since sc.load() will automatically try loading zstandard.

New in version 2.1.0.