sc_math

sc_math

Extensions to Numpy, including finding array elements and smoothing data.

Highlights

Functions

Name Description
approx Determine whether two scalars (or an array and a scalar) approximately match.
cat Like numpy.concatenate, but takes anything and returns an array. Useful for
convolve Like numpy.convolve, but always returns an array the size of the first array
count Count the number of matching elements.
dataindex Take an array of data and return either the first or last (or some other) non-NaN entry.
fillnans Alias for sc.sanitize(..., replacenans=True) with nearest interpolation
findfirst Alias for sc.findinds(..., first=True). New in version 1.0.0.
findinds Find matches even if two things aren’t eactly equal (e.g. floats vs. ints).
findlast Alias for sc.findinds(..., last=True). New in version 1.0.0.
findnans Alias for sc.findinds(np.isnan(data)).
findnearest Return the index of the nearest match in series to value – like sc.findinds(), but
gauss1d Gaussian 1D smoothing kernel.
gauss2d Gaussian 2D smoothing kernel.
getvaliddata Return the data value indices that are valid based on the validity of the input data.
getvalidinds Return the indices that are valid based on the validity of the input data from an arbitrary number
inclusiverange Like numpy.arange/numpy.linspace, but includes the start and stop points.
isprime Determine if a number is prime.
linregress Simple linear regression returning the line of best fit and R value. Similar
nanequal Compare two or more arrays for equality element-wise, treating NaN values as equal.
normalize Rescale an array between a minimum value and a maximum value.
normsum Multiply a list or array by some normalizing factor so that its sum is equal
numdigits Count the number of digits in a number (or list of numbers).
perturb Define an array of numbers uniformly perturbed with a mean of 1.
randround Round a float, list, or array probabilistically to the nearest integer. Works
rolling Alias to pandas.Series.rolling() (window) method to smooth a series.
safedivide Handle divide-by-zero and divide-by-nan elegantly.
sanitize Sanitize input to remove NaNs. (NB: sc.sanitize() and sc.rmnans() are aliases.)
sem Calculate the standard error of the mean (SEM).
similarity Compute pair-wise similarity for two or more sets
smooth Very simple function to smooth a 1D or 2D array.
smoothinterp Smoothly interpolate over values

approx

sc_math.approx(val1=None, val2=None, eps=None, **kwargs)

Determine whether two scalars (or an array and a scalar) approximately match. Alias for np.isclose() and may be removed in future versions.

Parameters

Name Type Description Default
val1 number or array the first value None
val2 number the second value None
eps float absolute tolerance None
kwargs dict passed to np.isclose() {}

Examples:

sc.approx(2*6, 11.9999999, eps=1e-6) # Returns True
sc.approx([3,12,11.9], 12) # Returns array([False, True, False], dtype=bool)

cat

sc_math.cat(*args, copy=False, **kwargs)

Like numpy.concatenate, but takes anything and returns an array. Useful for e.g. appending a single number onto the beginning or end of an array.

Parameters

Name Type Description Default
args any items to concatenate into an array ()
kwargs dict passed to numpy.concatenate {}

Examples:

arr = sc.cat(4, np.ones(3))
arr = sc.cat(np.array([1,2,3]), [4,5], 6)
arr = sc.cat(np.random.rand(2,4), np.random.rand(2,6), axis=1)
  • New in version 1.0.0.
  • New in version 1.1.0: “copy” and keyword arguments.
  • New in version 2.0.2: removed “copy” argument; changed default axis of 0; arguments passed to np.concatenate()

convolve

sc_math.convolve(a, v)

Like numpy.convolve, but always returns an array the size of the first array (equivalent to mode=‘same’), and solves the boundary problem present in numpy.convolve by adjusting the edges by the weight of the convolution kernel.

Parameters

Name Type Description Default
a arr the input array required
v arr the convolution kernel required

Example:

a = np.ones(5)
v = np.array([0.3, 0.5, 0.2])
c1 = np.convolve(a, v, mode='same') # Returns array([0.8, 1.  , 1.  , 1.  , 0.7])
c2 = sc.convolve(a, v)              # Returns array([1., 1., 1., 1., 1.])
  • New in version 1.3.0.
  • New in version 1.3.1: handling the case where len(a) < len(v)

count

sc_math.count(arr=None, val=None, eps=1e-06, **kwargs)

Count the number of matching elements.

Similar to numpy.count_nonzero(), but allows for slight mismatches (e.g., floats vs. ints). Equivalent to len(sc.findinds()).

Parameters

Name Type Description Default
arr array the array to find values in None
val float if provided, the value to match None
eps float the precision for matching (default 1e-6, equivalent to numpy.isclose’s atol) 1e-06
kwargs dict passed to numpy.isclose() {}

Examples:

sc.count(rand(10)<0.5) # returns e.g. 4
sc.count([2,3,6,3], 3) # returns 2

New in version 2.0.0.

dataindex

sc_math.dataindex(dataarray, index)

Take an array of data and return either the first or last (or some other) non-NaN entry.

This function is deprecated.

fillnans

sc_math.fillnans(data=None, replacenans=True, **kwargs)

Alias for sc.sanitize(..., replacenans=True) with nearest interpolation (or a specified value).

New in version 2.0.0.

findfirst

sc_math.findfirst(*args, **kwargs)

Alias for sc.findinds(..., first=True). New in version 1.0.0.

findinds

sc_math.findinds(
    arr=None,
    val=None,
    *args,
    eps=1e-06,
    first=False,
    last=False,
    ind=None,
    die=True,
    **kwargs,
)

Find matches even if two things aren’t eactly equal (e.g. floats vs. ints).

If one argument, find nonzero values. With two arguments, check for equality using eps (by default 1e-6, to handle single-precision floating point). Returns a tuple of arrays if val1 is multidimensional, else returns an array. Similar to calling np.nonzero(np.isclose(arr, val))[0].

Parameters

Name Type Description Default
arr array the array to find values in None
val float if provided, the value to match None
args list if provided, additional boolean arrays ()
eps float the precision for matching (default 1e-6, equivalent to numpy.isclose’s atol) 1e-06
first bool whether to return the first matching value (equivalent to ind=0) False
last bool whether to return the last matching value (equivalent to ind=-1) False
ind int index of match to retrieve None
die bool whether to raise an exception if first or last is true and no matches were found True
kwargs dict passed to numpy.isclose() {}

Examples:

data = np.random.rand(10)
sc.findinds(data<0.5) # Standard usage; returns e.g. array([2, 4, 5, 9])
sc.findinds(data>0.1, data<0.5) # Multiple arguments

sc.findinds([2,3,6,3], 3) # Returs array([1,3])
sc.findinds([2,3,6,3], 3, first=True) # Returns 1
  • New in version 1.2.3: “die” argument
  • New in version 2.0.0: fix string matching; allow multiple arguments
  • New in version 3.0.0: multidimensional arrays now return a list of tuples

findlast

sc_math.findlast(*args, **kwargs)

Alias for sc.findinds(..., last=True). New in version 1.0.0.

findnans

sc_math.findnans(data=None, **kwargs)

Alias for sc.findinds(np.isnan(data)).

Examples:

data = [0, 1, 2, np.nan, 4, np.nan, 6, np.nan, np.nan, np.nan, 10]
sc.findnans(data) # Returns array([3, 5, 7, 8, 9])
  • New in version 3.0.0.
  • New in version 3.1.0: replaced np.isnan with pd.isna for robustness

findnearest

sc_math.findnearest(series=None, value=None)

Return the index of the nearest match in series to value – like sc.findinds(), but always returns an object with the same type as value (i.e. findnearest with a number returns a number, findnearest with an array returns an array).

Parameters

Name Type Description Default
series array the array of numbers to look for nearest matches in None
value scalar or array the number or numbers to compare against None

Examples:

sc.findnearest(rand(10), 0.5) # returns whichever index is closest to 0.5
sc.findnearest([2,3,6,3], 6) # returns 2
sc.findnearest([2,3,6,3], 6) # returns 2
sc.findnearest([0,2,4,6,8,10], [3, 4, 5]) # returns array([1, 2, 2])

gauss1d

sc_math.gauss1d(x=None, y=None, xi=None, scale=None, use32=True)

Gaussian 1D smoothing kernel.

Create smooth interpolation of input points at interpolated points. If no points are supplied, use the same as the input points.

Parameters

Name Type Description Default
x arr 1D list of x coordinates None
y arr 1D list of y values at each of the x coordinates None
xi arr 1D list of points to calculate the interpolated y None
scale float how much smoothing to apply (by default, width of 5 data points) None
use32 bool convert arrays to 32-bit floats (doubles speed for large arrays) True

Examples:

# Setup
import numpy as np
import matplotlib.pyplot as plt
import sciris as sc

x = np.random.rand(40)
y = (x-0.3)**2 + 0.2*np.random.rand(40)

# Smooth
yi = sc.gauss1d(x, y)
yi2 = sc.gauss1d(x, y, scale=0.3)
xi3 = np.linspace(0,1)
yi3 = sc.gauss1d(x, y, xi)

# Plot original and interpolated versions
plt.scatter(x, y,     label='Original')
plt.scatter(x, yi,    label='Default smoothing')
plt.scatter(x, yi2,   label='More smoothing')
plt.scatter(xi3, yi3, label='Uniform spacing')
plt.show()

# Simple usage
sc.gauss1d(y)

New in version 1.3.0.

gauss2d

sc_math.gauss2d(
    x=None,
    y=None,
    z=None,
    xi=None,
    yi=None,
    scale=1.0,
    xscale=1.0,
    yscale=1.0,
    grid=False,
    use32=True,
)

Gaussian 2D smoothing kernel.

Create smooth interpolation of input points at interpolated points. Can handle either 1D or 2D inputs.

Parameters

Name Type Description Default
x arr 1D or 2D array of x coordinates (if None, take from z) None
y arr ditto, for y None
z arr 1D or 2D array of z values at each of the (x,y) points None
xi arr 1D or 2D array of points to calculate the interpolated Z; if None, same as x None
yi arr ditto, for y None
scale float overall scale factor 1.0
xscale float ditto, just for x 1.0
yscale float ditto, just for y 1.0
grid bool if True, then return Z at a grid of (xi,yi) rather than at points False
use32 bool convert arrays to 32-bit floats (doubles speed for large arrays) True

Examples:

# Setup
import numpy as np
import matplotlib.pyplot as plt

x = np.random.rand(40)
y = np.random.rand(40)
z = 1-(x-0.5)**2 + (y-0.5)**2 # Make a saddle

# Simple usage -- only works if z is 2D
zi0 = sc.gauss2d(np.random.rand(10,10))
sc.surf3d(zi0)

# Method 1 -- form grid
xi = np.linspace(0,1,20)
yi = np.linspace(0,1,20)
zi = sc.gauss2d(x, y, z, xi, yi, scale=0.1, grid=True)

# Method 2 -- use points directly
xi2 = np.random.rand(400)
yi2 = np.random.rand(400)
zi2 = sc.gauss2d(x, y, z, xi2, yi2, scale=0.1)

# Plot oiginal and interpolated versions
sc.scatter3d(x, y, z, c=z)
sc.surf3d(zi)
sc.scatter3d(xi2, yi2, zi2, c=zi2)
plt.show()
  • New in version 1.3.0.
  • New in version 1.3.1: default arguments; support for 2D inputs

getvaliddata

sc_math.getvaliddata(data=None, filterdata=None, defaultind=0)

Return the data value indices that are valid based on the validity of the input data.

This function is deprecated; see sc.sanitize() instead.

Example:

sc.getvaliddata(array([3,5,8,13]), array([2000, nan, nan, 2004])) # Returns array([3,13])

getvalidinds

sc_math.getvalidinds(data=None, filterdata=None)

Return the indices that are valid based on the validity of the input data from an arbitrary number of 1-D vector inputs. Note, closely related to sc.getvaliddata().

This function is deprecated.

Example:

sc.getvalidinds([3,5,8,13], [2000, nan, nan, 2004]) # Returns array([0,3])

inclusiverange

sc_math.inclusiverange(*args, stretch=False, **kwargs)

Like numpy.arange/numpy.linspace, but includes the start and stop points. Accepts 0-3 args, or the kwargs start, stop, step.

In most cases, equivalent to np.linspace(start, stop, int((stop-start)/step)+1).

Parameters

Name Type Description Default
start float value to start at required
stop float value to stop at required
step float step size required
stretch bool if True, adjust the step size to end exactly at stop if needed False
kwargs dict passed to numpy.linspace {}

Examples:

x = sc.inclusiverange(10)        # Like np.arange(11)
x = sc.inclusiverange(3,5,0.2)   # Like np.linspace(3, 5, int((5-3)/0.2+1))
x = sc.inclusiverange(stop=5)    # Like np.arange(6)
x = sc.inclusiverange(6, step=2) # Like np.arange(0, 7, 2)
x = sc.inclusiverange(0, 10, 3) # Like np.arange(0, 10, 3)
x = sc.inclusiverange(0, 10, 3, stretch=True) # Like np.linspace(0,10,int(10/3)+1)
  • New in version 3.2.0: “stretch” argument

isprime

sc_math.isprime(n, verbose=False)

Determine if a number is prime.

From https://stackoverflow.com/questions/15285534/isprime-function-for-python-language

Example:

for i in range(100): print(i) if sc.isprime(i) else None

linregress

sc_math.linregress(x, y, full=False, **kwargs)

Simple linear regression returning the line of best fit and R value. Similar to `scipy.stats.linregress`` but simpler.

Parameters

Name Type Description Default
x array the x coordinates required
y array the y coordinates required
full bool whether to return a full data structure False
kwargs dict passed to numpy.polyfit {}

Examples:

x = range(10)
y = sorted(2*np.random.rand(10) + 1)
m,b = sc.linregress(x, y) # Simple usage
out = sc.linregress(x, y, full=True) # Has out.m, out.b, out.x, out.y, out.corr, etc.
plt.scatter(x, y)
plt.plot(x, m*x+b)
plt.bar(x, out.residuals)
plt.title(f'R² = {out.r2}')

nanequal

sc_math.nanequal(arr, *args, scalar=False, equal_nan=True)

Compare two or more arrays for equality element-wise, treating NaN values as equal.

Unnlike numpy.array_equal, this function works even if the arrays cannot be cast to float.

Parameters

Name Type Description Default
arr array the array to use as the base for the comparison required
args list one or more arrays to compare to ()
scalar bool whether to return a true/false value (else return the array) False

Examples:

arr1 = np.array([1, 2, np.nan])
arr2 = [1, 2, np.nan]
sc.nanequal(arr1, arr2) # Returns array([ True,  True,  True])

arr3 = [3, np.nan, 'foo']
sc.nanequal(arr3, arr3, arr3, scalar=True) # Returns True

New in version 3.1.0.

normalize

sc_math.normalize(arr, minval=0.0, maxval=1.0)

Rescale an array between a minimum value and a maximum value.

Parameters

Name Type Description Default
arr array array to normalize required
minval float minimum value in rescaled array 0.0
maxval float maximum value in rescaled array 1.0

Example:

normarr = sc.normalize([2,3,7,27]) # Returns array([0.  , 0.04, 0.2 , 1.  ])

normsum

sc_math.normsum(arr, total=None)

Multiply a list or array by some normalizing factor so that its sum is equal to the total. Formerly called “scaleratio”.

Parameters

Name Type Description Default
arr array array (or list) to normalize required
total float amount to sum to (default 1) None

Example:

normarr = sc.normsum([2,5,3,10], 100) # Scale so sum equals 100; returns [10.0, 25.0, 15.0, 50.0]

Renamed in version 1.0.0.

numdigits

sc_math.numdigits(n, *args, count_minus=False, count_decimal=False)

Count the number of digits in a number (or list of numbers).

Useful for e.g. knowing how long a string needs to be to fit a given number.

If a number is less than 1, return the number of digits until the decimal place.

Reference: https://stackoverflow.com/questions/22656345/how-to-count-the-number-of-digits-in-python

Parameters

Name Type Description Default
n int / float / list / array number or list of numbers required
args list additional numbers ()
count_minus bool whether to count the minus sign as a digit False
count_decimal bool whether to count the decimal point as a digit False

Examples:

sc.numdigits(12345) # Returns 5
sc.numdigits(12345.5) # Returns 5
sc.numdigits(0) # Returns 1
sc.numdigits(-12345) # Returns 5
sc.numdigits(-12345, count_minus=True) # Returns 6
sc.numdigits(12, 123, 12345) # Returns [2, 3, 5]
sc.numdigits(0.01) # Returns -2
sc.numdigits(0.01, count_decimal=True) # Returns -4

New in version 2.0.0.

perturb

sc_math.perturb(*args, n=1, span=0.5, randseed=None, normal=False)

Define an array of numbers uniformly perturbed with a mean of 1.

Note: if called with a single argument, this is intepreted as “span”, not “n”.

Parameters

Name Type Description Default
n int number of points; or an array to perturb 1
span float width of distribution on either side of 1 (or standard deviation if normal=True) 0.5
randseed int seed passed to the reseed Numpy’s random number generator None
normal bool whether to use a normal distribution instead of uniform False

Example:

sc.perturb() # Returns a random number on (0.5, 1.5)
sc.perturb(0.1) # Returns a random number on (0.9, 1.1)
sc.perturb(5, 0.3) # Returns e.g. array([0.73852362, 0.7088094 , 0.93713658, 1.13150755, 0.87183371])
sc.perturb([1,2,3], 0.1, normal=True) # Returns e.g. array([1.03574377, 2.00286363, 3.53437126])
  • New in version 3.0.0: Uses a separate random number stream
  • New in version 3.2.1: Allows use with a single argument; allows “n” to be an array

randround

sc_math.randround(x)

Round a float, list, or array probabilistically to the nearest integer. Works for both positive and negative values.

Adapted from

https://stackoverflow.com/questions/19045971/random-rounding-to-integer-in-python

Parameters

Name Type Description Default
x (int, list, arr) the floating point numbers to probabilistically convert to the nearest integer required

Returns

Name Type Description
Array of integers

Example:

sc.randround(np.random.randn(8)) # Returns e.g. array([-1,  0,  1, -2,  2,  0,  0,  0])
  • New in version 1.0.0.
  • New in version 3.0.0: allow arrays of arbitrary shape

rolling

sc_math.rolling(data, window=7, operation='mean', replacenans=None, **kwargs)

Alias to pandas.Series.rolling() (window) method to smooth a series.

Parameters

Name Type Description Default
data list / arr the 1D or 2D data to be smoothed required
window int the length of the window 7
operation str the operation to perform: ‘mean’ (default), ‘median’, ‘sum’, or ‘none’ 'mean'
replacenans bool / float if None, leave NaNs; if False, remove them; if a value, replace with that value; if the string ‘nearest’ or ‘linear’, do interpolation (see sc.rmnans() for details) None
kwargs dict passed to pandas.Series.rolling() {}

Example:

data = [5,5,5,0,0,0,0,7,7,7,7,0,0,3,3,3]
rolled = sc.rolling(data, replacenans='nearest')

safedivide

sc_math.safedivide(
    numerator=None,
    denominator=None,
    default=None,
    eps=None,
    warn=False,
)

Handle divide-by-zero and divide-by-nan elegantly.

Examples:

sc.safedivide(numerator=0, denominator=0, default=1, eps=0) # Returns 1
sc.safedivide(numerator=5, denominator=2.0, default=1, eps=1e-3) # Returns 2.5
sc.safedivide(3, np.array([1,3,0]), -1, warn=True) # Returns array([ 3,  1, -1])

sanitize

sc_math.sanitize(
    data=None,
    returninds=False,
    replacenans=None,
    defaultval=None,
    die=True,
    verbose=False,
    label=None,
)

Sanitize input to remove NaNs. (NB: sc.sanitize() and sc.rmnans() are aliases.)

Returns an array with the sanitized data. If replacenans=True, the sanitized array is of the same length/size as data. If replacenans=False, the sanitized array may be shorter than data.

Parameters

Name Type Description Default
data (arr/list) array or list with numbers to be sanitized required
returninds (bool) whether to return indices of non-nan/valid elements, indices are with respect the shape of data required
replacenans (float/str) whether to replace the NaNs with the specified value, or if True or a string, using interpolation required
defaultval (float) value to return if the sanitized array is empty required
die (bool) whether to raise an exception if the sanitization failed (otherwise return an empty array) required
verbose (bool) whether to print out a warning if no valid values are found required
label (str) human readable label for data (for use with verbose mode only) required

Examples:

data = [3, 4, np.nan, 8, 2, np.nan, np.nan, 8]
sanitized1, inds = sc.sanitize(data, returninds=True) # Remove NaNs
sanitized2 = sc.sanitize(data, replacenans=True) # Replace NaNs using nearest neighbor interpolation
sanitized3 = sc.sanitize(data, replacenans='nearest') # Eequivalent to replacenans=True
sanitized4 = sc.sanitize(data, replacenans='linear') # Replace NaNs using linear interpolation
sanitized5 = sc.sanitize(data, replacenans=0) # Replace NaNs with 0
  • New in version 2.0.0: handle multidimensional arrays
  • New in version 3.0.0: return zero-length arrays if all NaN

sem

sc_math.sem(a, axis=None, *args, **kwargs)

Calculate the standard error of the mean (SEM).

Shortcut (for a 1D array) to array.std()/np.sqrt(len(array)).

Parameters

Name Type Description Default
a arr array to calculate the SEM of required
axis int axis to calculate the SEM along None
kwargs dict passed to numpy.std {}

Example:

data = np.random.randn(100)
sem = sc.sem(data) # Roughly 0.1
  • New in version 3.2.0.

similarity

sc_math.similarity(*args, method='jaccard')

Compute pair-wise similarity for two or more sets

If two arguments, returns a scalar similarity between 0 and 1. If three or more, compute the matrix of similarities. Similarity is computed by default via the Jaccard index, which is the length of the intersection of the sets divided by the length of their union.

Parameters

Name Type Description Default
*args set / list / arr Two or more iterables to compute the ()
method str Similarity metric: ‘jaccard’ (default) or ‘dice’ 'jaccard'
  • New in version 3.2.4.

smooth

sc_math.smooth(data, repeats=None, kernel=None, legacy=False)

Very simple function to smooth a 1D or 2D array.

See also sc.gauss1d() for simple Gaussian smoothing.

Parameters

Name Type Description Default
data arr 1D or 2D array to smooth required
repeats int number of times to apply smoothing (by default, scale to be 1/5th the length of data) None
kernel arr the smoothing kernel to use (default: [0.25, 0.5, 0.25]) None
legacy bool if True, use the old (pre-1.3.0) method of calculation that doesn’t correct for edge effects False

Example:

data = np.random.randn(5,5)
smoothdata = sc.smooth(data)

New in version 1.3.0: Fix edge effects.

smoothinterp

sc_math.smoothinterp(
    newx=None,
    origx=None,
    origy=None,
    smoothness=None,
    growth=None,
    ensurefinite=True,
    keepends=True,
    method='linear',
)

Smoothly interpolate over values

Unlike np.interp(), this function does exactly pass through each data point:

Parameters

Name Type Description Default
newx arr the points at which to interpolate None
origx arr the original x coordinates None
origy arr the original y coordinates None
smoothness float how much to smooth None
growth float the growth rate to apply past the ends of the data None
ensurefinite bool ensure all values are finite (including skipping NaNs) True
method str the type of interpolation to use (options are ‘linear’ or ‘nearest’) 'linear'

Returns

Name Type Description
newy arr the new y coordinates

Example:

import sciris as sc
import numpy as np
from scipy import interpolate

origy = np.array([0,0.2,0.1,0.9,0.7,0.8,0.95,1])
origx = np.linspace(0,1,len(origy))
newx = np.linspace(0,1,5*len(origy))
sc_y = sc.smoothinterp(newx, origx, origy, smoothness=5)
np_y = np.interp(newx, origx, origy)
si_y = interpolate.interp1d(origx, origy, 'cubic')(newx)
kw = dict(lw=3, alpha=0.7)
plt.plot(newx, np_y, '--', label='NumPy', **kw)
plt.plot(newx, si_y, ':',  label='SciPy', **kw)
plt.plot(newx, sc_y, '-',  label='Sciris', **kw)
plt.scatter(origx, origy, s=50, c='k', label='Data')
plt.legend()
plt.show()
  • New in verison 3.0.0: “ensurefinite” now defaults to True; removed “skipnans” argument