sc_dataframe

sc_dataframe

Extension of the pandas dataframe to be more flexible, especially with filtering rows/columns and concatenating data.

Classes

Name	Description
dataframe	An extension of the pandas DataFrame with additional convenience methods for

dataframe

sc_dataframe.dataframe(
    data=None,
    index=None,
    columns=None,
    dtype=None,
    copy=None,
    dtypes=None,
    nrows=None,
    **kwargs,
)

An extension of the pandas DataFrame with additional convenience methods for accessing rows and columns and performing other operations, such as adding rows.

Parameters

Name	Type	Description	Default
data	dict / array / dataframe	the data to use; passed to pd.DataFrame()	None
index	array	the index to use; passed to pd.DataFrame()	None
columns	list	column labels (if a dict is supplied, the value sets the dtype)	None
dtype	type	a dtype for the whole dataframe; passed to pd.DataFrame()	None
copy	bool	whether to copy the data (ignored in pandas ≥ 3.0.0 due to Copy-on-Write behavior)	None
dtypes	list / dict	alternatively, list of data types to set each column to	None
nrows	int	the number of arrows to preallocate (default 0)	None
kwargs	dict	if provided, treat these as data columns	{}

Hint: Run the example below line by line to get a sense of how the dataframe changes.

Examples:

df = sc.dataframe(cols=['x','y'], data=[[1238,2],[384,5],[666,7]]) # Create data frame
df['x'] # Print out a column
df[0] # Print out a row
df['x',0] # Print out an element
df[0,:] = [123,6]; print(df) # Set values for a whole row
df['y'] = [8,5,0]; print(df) # Set values for a whole column
df['z'] = [14,14,14]; print(df) # Add new column
df.rmcol('z'); print(df) # Remove a column
df.addcol('z', [14,14,14]); print(df) # Alternate way to add new column
df.poprow(1); print(df) # Remove a row
df.append([555,2,14]); print(df) # Append a new row
df.insertrow(1,[556,2,14]); print(df) # Insert a new row
df.sort(); print(df) # Sort by the first column
df.sort('y'); print(df) # Sort by the second column
df.findrow(123) # Return the row starting with value 123
df.rmrow(); print(df) # Remove last row
df.rmrow(555); print(df) # Remove the row starting with element '555'

# Direct setting of data
df = sc.dataframe(a=[1,2,3], b=[4,5,6])

The dataframe can be used for both numeric and non-numeric data.

• New in version 2.0.0: subclass pandas DataFrame
• New in version 3.0.0: “dtypes” argument; handling of item setting
• New in version 3.1.0: use panda’s equality operator by default (unless an exception is raised); new “equal” method; “cat” can be an instance method now
• New in version 3.2.5: pandas 3.0.0 compatibility

Attributes

Name	Description
cols	Get columns as a list
ncols	Get the number of columns in the dataframe
nrows	Get the number of rows in the dataframe

Methods

Name	Description
addcol	Add new column(s) to the data frame
append	Alias to appendrow().
appendrow	Add row(s) to the end of the dataframe.
cat	Convenience class method for concatenating multiple dataframes. See df.concat()
col_index	Get the index of the column named col.
col_name	Get the name of the column(s) with index col.
concat	Concatenate additional data onto the current dataframe.
disp	Flexible display of a dataframe, showing all rows/columns by default.
enumrows	Efficiently enumerate the rows of the dataframe
equal	Class method returning boolean true/false equals that allows for more robust equality checks:
equals	Try the default equals(), but fall back
filtercols	Filter columns keeping only those specified – note, by default, do not perform in place
filterin	Keep only rows matching a criterion; see also df.filterout()
filterout	Remove rows matching a criterion (in place); see also df.filterin()
findind	Find the row index for a given value and column.
findinds	Return the indices of all rows matching the given key in a given column.
findrow	Return a row by searching for a matching value.
flexget	More complicated way of getting data from a dataframe. While getting directly
get	Alias to pandas getitem method; rarely used
insertrow	Insert row(s) at the specified location. See also df.concat()
merge	Alias to pd.merge, except merge in place.
popcols	Remove a column or columns from the data frame.
poprow	Remove a row from the data frame.
poprows	Remove multiple rows by index or value
read_csv	Alias to pd.read_csv <pandas.read_csv, returning a Sciris dataframe
read_csv_string	Read a CSV from a string rather than a file
read_excel	Alias to pd.read_excel <pandas.read_excel, returning a Sciris dataframe
replacecol	Replace all of one value in a column with a new value
replacedata	Replace data in the dataframe with other data; usually not used directly
set	Alias to pandas setitem method; rarely used
set_dtypes	Set dtypes in-place (see df.astype() for the user-facing version)
sort	Alias to sortrows().
sortcols	Like sortrows(), but change column order (usually in place) instead.
sortrows	Sort the dataframe rows in place by the specified column(s).
to_odict	Convert dataframe to a dict of columns, optionally specifying certain rows.
to_pandas	Convert to a plain pandas dataframe

addcol

sc_dataframe.dataframe.addcol(
    key=None,
    value=None,
    data=None,
    inplace=True,
    **kwargs,
)

Add new column(s) to the data frame

See also assign(), which is similar, but returns a new dataframe by default.

Parameters

Name	Type	Description	Default
key	str	the name of the column	None
value	array	the values for the column	None
data	dict	alternatively, specify a dictionary of columns to add	None
inplace	bool	whether to return a new dataframe	True
kwargs	dict	additional columns to add	{}

NB: a single argument is interpreted as “data”

Example:

df = sc.dataframe(dict(x=[1,2,3], y=[4,5,6]))
new_cols = dict(z=[1,2,3], a=[9,8,7])
df.addcol(new_cols)

append

sc_dataframe.dataframe.append(row, reset_index=True, inplace=True)

Alias to appendrow().

Note: pd.DataFrame.append was deprecated in pandas version 2.0; see https://github.com/pandas-dev/pandas/issues/35407 for details. Since this method is implemented using pd.concat(), it does not suffer from the performance problems that append did.

New in version 3.0.0.

appendrow

sc_dataframe.dataframe.appendrow(row, reset_index=True, inplace=True)

Add row(s) to the end of the dataframe.

See also df.concat() and df.insertrow(). Similar to the pandas operation df.iloc[-1] = ..., but faster and provides additional type checking.

Parameters

Name	Type	Description	Default
value	array	the row(s) to append	required
reset_index	bool	update the index	True
inplace	bool	whether to modify in-place	True

Note: “appendrow” and “concat” are equivalent, except appendrow() defaults to modifying in-place and “concat” defaults to returning a new dataframe.

Warning: modifying dataframes in-place is quite inefficient. For highest performance, construct the data in large chunks and then add to the dataframe all at once, rather than adding row by row.

Example:

import sciris as sc
import numpy as np

df = sc.dataframe(dict(
    a = ['foo','bar'],
    b = [1,2],
    c = np.random.rand(2)
))
df.appendrow(['cat', 3, 0.3])           # Append a list
df.appendrow(dict(a='dog', b=4, c=0.7)) # Append a dict

New in version 3.0.0: renamed “value” to “row”; improved performance

cat

sc_dataframe.dataframe.cat(data, *args, dfargs=None, **kwargs)

Convenience class method for concatenating multiple dataframes. See df.concat() for the equivalent instance method.

Parameters

Name	Type	Description	Default
data	dataframe / array	the dataframe/data to use as the basis of the new dataframe	required
args	list	additional dataframes (or object that can be converted to dataframes) to concatenate	()
dfargs	dict	arguments passed to construct each dataframe	None
kwargs	dict	passed to df.concat()	{}

Example:

arr1 = np.random.rand(6,3)
df2 = pd.DataFrame(np.random.rand(4,3))
df3 = sc.dataframe.cat(arr1, df2)

New in version 2.0.2.

col_index

sc_dataframe.dataframe.col_index(col=None, *args, die=True)

Get the index of the column named col.

Similar to df.columns.get_loc(col), and opposite of df.col_name.

Parameters

Name	Type	Description	Default
col	str / list	the column(s) to get the index of (return 0 if None)	None
args	list	additional column(s) to get the index of	()
die	bool	whether to raise an exception if the column could not be found (else, return None)	True

Examples:

df = sc.dataframe(dict(a=[1,2,3], b=[4,5,6], c=[7,8,9]))
df.col_index('b') # Returns 1
df.col_index(1) # Returns 1
df.col_index('a', 'c') # Returns [0, 2]

New in version 3.0.0: renamed from “_sanitizecols”; multiple arguments

col_name

sc_dataframe.dataframe.col_name(col=None, *args, die=True)

Get the name of the column(s) with index col.

Similar to df.columns[col], and opposite of df.col_index.

Note: This method always looks for named columns first. If col is name of a column, it will return col rather than columns[col]. See example below for more information.

Parameters

Name	Type	Description	Default
col	int / list	the column(s) to get the index of (return 0 if None)	None
args	list	additional column(s) to get the index of	()
die	bool	whether to raise an exception if the column could not be found (else, return None)	True

Examples:

df = sc.dataframe(dict(a=[1,2,3], b=[4,5,6], c=[7,8,9]))
df.col_name(1) # Returns 'b'
df.col_name('b') # Returns 'b'
df.col_name(0, 2) # Returns ['a', 'c']

New in version 3.0.0.

concat

sc_dataframe.dataframe.concat(
    data,
    *args,
    columns=None,
    reset_index=True,
    inplace=False,
    dfargs=None,
    **kwargs,
)

Concatenate additional data onto the current dataframe.

Similar to df.appendrow() and df.insertrow(); see also sc.dataframe.cat() for the equivalent class method.

Parameters

Name	Type	Description	Default
data	dataframe / array	the data to concatenate	required
*args	dataframe / array	additional data to concatenate	()
columns	list	if supplied, columns to go with the data	None
reset_index	bool	update the index	True
inplace	bool	whether to append in place	False
dfargs	dict	arguments passed to construct each dataframe	None
**kwargs	dict	passed to pd.concat()	{}

Example:

arr1 = np.random.rand(6,3)
df2 = sc.dataframe(np.random.rand(4,3))
df3 = df2.concat(arr1)

• New in version 2.0.2: “inplace” defaults to False
• New in version 3.0.0: improved type handling

disp

sc_dataframe.dataframe.disp(
    nrows=None,
    ncols=None,
    width=999,
    precision=4,
    options=None,
    **kwargs,
)

Flexible display of a dataframe, showing all rows/columns by default.

Parameters

Name	Type	Description	Default
nrows	int	maximum number of rows to show (default: all)	None
ncols	int	maximum number of columns to show (default: all)	None
width	int	maximum screen width (default: 999)	999
precision	int	number of decimal places to show (default: 4)	4
options	dict	an optional dictionary of additional options, passed to pd.option_context()	None
kwargs	dict	also passed to pd.option_context(), with ‘display.’ preprended if needed	{}

Examples:

df = sc.dataframe(data=np.random.rand(100,10))
df.disp()
df.disp(precision=1, ncols=5, colheader_justify='left')

New in version 2.0.1.

enumrows

sc_dataframe.dataframe.enumrows(cols=None, type='objdict')

Efficiently enumerate the rows of the dataframe

Similar to df.iterrows(), but up to 30x faster since uses tuples instead of pd.Series.

Parameters

Name	Type	Description	Default
cols	list	the list of columns to include in the enumeration (by default, all)	None
type	str / type	the output type for each row: options are ‘objdict’ (default), tuple (fastest), list (very fast), dict (pretty fast)	'objdict'

Examples:

df = sc.dataframe(dict(x=[0,1,2,3,4], y=[2,3,2,7,8], z=[5,5,4,3,2]))
for i,row in df.enumrows(): print(i, row.x+row.y) # Typical use case
for i,row in df.enumrows(type=tuple): print(i, row[0]+row[1]) # Fastest
for i,row in df.enumrows(type=dict): print(i, row['x']+row['y']) # Still fast
for i,(x,y) in df.enumrows(cols=['x', 'y'], type=tuple): print(i, x+y) # Even faster

equal

sc_dataframe.dataframe.equal(*args, equal_nan=True)

Class method returning boolean true/false equals that allows for more robust equality checks: same type, size, columns, and values. See df.equals() for equivalent instance method.

Examples:

df1 = sc.dataframe(a=[1, 2, np.nan])
df2 = sc.dataframe(a=[1, 2, 4])

sc.dataframe.equal(df1, df1) # Returns True
sc.dataframe.equal(df1, df1, equal_nan=False) # Returns False
sc.dataframe.equal(df1, df2) # Returns False
sc.dataframe.equal(df1, df1, df2) # Also returns False

New in version 3.1.0.

equals

sc_dataframe.dataframe.equals(other, *args, equal_nan=True)

Try the default equals(), but fall back on the more robust sc.dataframe.equal() if that fails.

New in version 3.1.0.

filtercols

sc_dataframe.dataframe.filtercols(
    cols=None,
    *args,
    keep=True,
    die=True,
    reset_index=True,
    inplace=False,
)

Filter columns keeping only those specified – note, by default, do not perform in place

Parameters

Name	Type	Description	Default
cols	str / list	the columns to keep (or remove if keep=False)	None
args	list	additional columns	()
keep	bool	whether to keep the named columns (else, remove them)	True
die	bool	whether to raise an exception if a column is not found	True
reset_index	bool	update the index	True
inplace	bool	whether to modify in-place	False

Examples:

df = sc.dataframe(cols=['a','b','c','d'], data=np.random.rand(3,4))
df2 = df.filtercols('a','b') # Keeps columns 'a' and 'b'
df3 = df.filtercols('a','c', keep=False) # Keeps columns 'b' and 'd'

filterin

sc_dataframe.dataframe.filterin(
    inds=None,
    value=None,
    col=None,
    verbose=False,
    reset_index=True,
    inplace=False,
)

Keep only rows matching a criterion; see also df.filterout()

filterout

sc_dataframe.dataframe.filterout(
    inds=None,
    value=None,
    col=None,
    verbose=False,
    reset_index=True,
    inplace=False,
)

Remove rows matching a criterion (in place); see also df.filterin()

findind

sc_dataframe.dataframe.findind(value=None, col=None, closest=False, die=True)

Find the row index for a given value and column.

See df.findrow() for the equivalent to return the row itself rather than the index of the row. See df.col_index() for the column equivalent.

Parameters

Name	Type	Description	Default
value	any	the value to look for (default: return last row index)	None
col	str	the column to look in (default: first)	None
closest	bool	if true, return the closest match if an exact match is not found	False
die	bool	whether to raise an exception if the value is not found (otherwise, return None)	True

Example:

df = sc.dataframe(data=[[2016,0.3],[2017,0.5]], columns=['year','val'])
df.findind(2016) # returns 0
df.findind(0.5, 'val') # returns 1
df.findind(2013) # returns None, or exception if die is True
df.findind(2013, closest=True) # returns 0

New in version 3.0.0: renamed from “_rowindex”

findinds

sc_dataframe.dataframe.findinds(value=None, col=None, **kwargs)

Return the indices of all rows matching the given key in a given column.

Parameters

Name	Type	Description	Default
value	any	the value to look for	None
col	str	the column to look in	None
kwargs	dict	passed to sc.findinds()	{}

Example:

df = sc.dataframe(cols=['year','val'],data=[[2016,0.3],[2017,0.5], [2018, 0.3]])
df.findinds(0.3, 'val') # Returns array([0,2])

findrow

sc_dataframe.dataframe.findrow(
    value=None,
    col=None,
    default=None,
    closest=False,
    asdict=False,
    die=False,
)

Return a row by searching for a matching value.

See df.findind() for the equivalent to return the index of the row rather than the row itself, and df.findinds() to find multiple row indices.

Parameters

Name	Type	Description	Default
value	any	the value to look for	None
col	str	the column to look for this value in	None
default	any	the value to return if key is not found (overrides die)	None
closest	bool	whether or not to return the closest row (overrides default and die)	False
asdict	bool	whether to return results as dict rather than list	False
die	bool	whether to raise an exception if the value is not found	False

Examples:

df = sc.dataframe(cols=['year','val'],data=[[2016,0.3],[2017,0.5], [2018, 0.3]])
df.findrow(2016) # returns array([2016, 0.3], dtype=object)
df.findrow(2013) # returns None, or exception if die is True
df.findrow(2013, closest=True) # returns array([2016, 0.3], dtype=object)
df.findrow(2016, asdict=True) # returns {'year':2016, 'val':0.3}

flexget

sc_dataframe.dataframe.flexget(
    cols=None,
    rows=None,
    asarray=False,
    cast=True,
    default=None,
)

More complicated way of getting data from a dataframe. While getting directly by key usually returns the array data directly, this usually returns another dataframe.

Parameters

Name	Type	Description	Default
cols	str / list	the column(s) to get	None
rows	int / list	the row(s) to get	None
asarray	bool	whether to return an array (otherwise, return a dataframe)	False
cast	bool	attempt to cast to an all-numeric array	True
default	any	the value to return if the column(s)/row(s) can’t be found	None

Example:

df = sc.dataframe(cols=['x','y','z'],data=[[1238,2,-1],[384,5,-2],[666,7,-3]]) # Create data frame
df.flexget(cols=['x','z'], rows=[0,2])

get

sc_dataframe.dataframe.get(key)

Alias to pandas getitem method; rarely used

insertrow

sc_dataframe.dataframe.insertrow(
    index=0,
    value=None,
    reset_index=True,
    inplace=True,
    die=True,
    **kwargs,
)

Insert row(s) at the specified location. See also df.concat() and df.appendrow().

Parameters

Name	Type	Description	Default
index	int	index at which to insert new row(s)	0
value	array	the row(s) to insert; can be an array, list, or dict	None
reset_index	bool	update the index	True
inplace	bool	whether to modify in-place	True
die	bool	raise an exception if the length/columns of the inserted row do not match the existing dataframe	True
kwargs	dict	passed to `df.concat()	{}

Warning: modifying dataframes in-place is quite inefficient. For highest performance, construct the data in large chunks and then add to the dataframe all at once, rather than adding row by row.

Example:

import sciris as sc
import numpy as np

df = sc.dataframe(dict(
    a = ['foo','cat'],
    b = [1,3],
    c = np.random.rand(2)
))
df.insertrow(1, ['bar', 2, 0.2])           # Insert a list
df.insertrow(0, dict(a='rat', b=0, c=0.7)) # Insert a dict

• New in version 3.0.0: renamed “row” to “index”
• New in version 3.2.3: “die” argument

merge

sc_dataframe.dataframe.merge(*args, reset_index=True, inplace=False, **kwargs)

Alias to pd.merge, except merge in place.

Parameters

Name	Type	Description	Default
reset_index	bool	update the index	True
inplace	bool	whether to append in place	False
**kwargs	dict	passed to pd.concat()	{}

New in version 3.0.0.

Example:

df = sc.dataframe(dict(x=[1,2,3], y=[4,5,6]))
df2 = sc.dataframe(dict(x=[1,2,3], z=[9,8,7]))
df.merge(df2, on='x', inplace=True)

popcols

sc_dataframe.dataframe.popcols(col=None, *args, die=True)

Remove a column or columns from the data frame.

Alias to pop(), except allowing multiple columns to be popped.

Parameters

Name	Type	Description	Default
col	str / list	the column(s) to be popped	None
args	list	additional columns to pop	()
die	bool	whether to raise an exception if a column is not found	True

Example:

df = sc.dataframe(cols=['a','b','c','d'], data=np.random.rand(3,4))
df.popcols('a','c')

poprow

sc_dataframe.dataframe.poprow(row=-1, returnval=True)

Remove a row from the data frame.

Alias to drop, except drop by position rather than label, and modify in-place. To pop multiple rows, see meth:df.poprows() <dataframe.poprows>.

Parameters

Name	Type	Description	Default
row	int	index of the row to pop	-1
returnval	bool	whether to return the row that was popped	True

To pop a column, see df.pop().

New in version 3.0.0: “key” argument renamed “row”

poprows

sc_dataframe.dataframe.poprows(
    inds=-1,
    value=None,
    col=None,
    reset_index=True,
    inplace=True,
    **kwargs,
)

Remove multiple rows by index or value

To pop a single row, see meth:df.poprow() <dataframe.poprow>.

Parameters

Name	Type	Description	Default
inds	list	the rows to remove	-1
values	list	alternatively, search for these values to remove; see df.findinds for details	required
col	str	if removing by value, use this column to find the values	None
reset_index	bool	update the index	True
inplace	bool	whether to modify in-place	True
kwargs	dict	passed to df.findinds	{}

Examples:

df = sc.dataframe(np.random.rand(10,3))
df.poprows([3,4,5])

df = sc.dataframe(dict(x=[0,1,2,3,4], y=[2,3,2,7,8]))
df.poprows(value=2, col='y')

read_csv

sc_dataframe.dataframe.read_csv(*args, **kwargs)

Alias to pd.read_csv <pandas.read_csv, returning a Sciris dataframe

read_csv_string

sc_dataframe.dataframe.read_csv_string(string, strip=True, **kwargs)

Read a CSV from a string rather than a file

Shortcut to sc.dataframe.read_csv(io.StringIO(string)).

Parameters

Name	Type	Description	Default
string	str	the string to parse as CSV data	required
strip	bool	whether to strip leading/trailing whitespace from the string first	True
kwargs	dict	passed to pd.read_csv	{}

Example:

df = sc.dataframe.read_csv_string('''
a,b
1,2
3,4
''')

New in version 3.3.0.

read_excel

sc_dataframe.dataframe.read_excel(*args, **kwargs)

Alias to pd.read_excel <pandas.read_excel, returning a Sciris dataframe

replacecol

sc_dataframe.dataframe.replacecol(col=None, old=None, new=None)

Replace all of one value in a column with a new value

replacedata

sc_dataframe.dataframe.replacedata(
    newdata=None,
    newdf=None,
    reset_index=True,
    inplace=True,
)

Replace data in the dataframe with other data; usually not used directly by the user, but used as part of e.g. df.concat().

Parameters

Name	Type	Description	Default
newdata	array	replace the dataframe’s data with these data	None
newdf	dataframe	substitute the current dataframe with this one	None
reset_index	bool	update the index	True
inplace	bool	whether to modify in-place	True

New in version 3.0.0: improved dtype handling New in version 3.2.5: support deprecation of the verify_is_copy argument in Pandas 3.0

set

sc_dataframe.dataframe.set(key, value=None)

Alias to pandas setitem method; rarely used

set_dtypes

sc_dataframe.dataframe.set_dtypes(dtypes)

Set dtypes in-place (see df.astype() for the user-facing version)

New in version 3.0.0.

sort

sc_dataframe.dataframe.sort(
    by=None,
    reverse=False,
    returninds=False,
    inplace=True,
    **kwargs,
)

Alias to sortrows().

New in version 3.0.0.

sortcols

sc_dataframe.dataframe.sortcols(sortorder=None, reverse=False, inplace=True)

Like sortrows(), but change column order (usually in place) instead.

Parameters

Name	Type	Description	Default
sortorder	list	the list of indices to resort the columns by (if none, then alphabetical)	None
reverse	bool	whether to reverse the order	False
inplace	bool	whether to modify the dataframe in-place	True

New in version 3.0.0: Ensure dtypes are preserved; “inplace” argument; “returninds” argument removed

sortrows

sc_dataframe.dataframe.sortrows(
    by=None,
    reverse=False,
    returninds=False,
    reset_index=True,
    inplace=True,
    **kwargs,
)

Sort the dataframe rows in place by the specified column(s).

Similar to df.sort_values(), except defaults to sorting in place, and optionally returns the indices used for sorting (like np.argsort()).

Parameters

Name	Type	Description	Default
col	str or int	column to sort by (default, first column)	required
reverse	bool	whether to reverse the sort order (i.e., ascending=False)	False
returninds	bool	whether to return the indices used to sort instead of the dataframe	False
reset_index	bool	update the index	True
inplace	bool	whether to modify the dataframe in-place	True
kwargs	dict	passed to df.sort_values()	{}

New in version 3.0.0: “inplace” argument; “col” argument renamed “by”

to_odict

sc_dataframe.dataframe.to_odict(row=None)

Convert dataframe to a dict of columns, optionally specifying certain rows.

Parameters

Name	Type	Description	Default
row	int / list	the rows to include	None

to_pandas

sc_dataframe.dataframe.to_pandas(**kwargs)

Convert to a plain pandas dataframe