Skip to content

curvefit.uncertainty.predictive_validity

Out of sample predictive validity

Runs out of sample predictive validity for all groups in the data with the specified model and solver. Uses a Residuals object for each group to store predictions and compute out of sample residuals at each time point.

Arguments

  • evaluation_space (Callable): the space in which to do predictive validity, can be different than the space that you're fitting the model in
  • debug_mode (bool): whether or not to store all of the model data and results used for every single group-specific model (saves a lot of memory when debug_mode = False)

Attributes

  • theta (int): the amount of scaling for the residuals relative to the predictions. If the evaluation space is in log space, then theta is set to 0. so that they are absolute residuals. If the evaluation space is in linear space, then theta is set to 1. so that they are relative residuals to the prediction magnitude.
  • `group_residuals (Dict[str: curvefit.uncertainty.residuals.Residual]): residual objects for each group in the dataset
  • `group_records (Dict[str: List[curvefit.solvers.solver.Solver]]): a list of solvers at each time point in the predictive validity analysis for a particular group

Methods

_make_group_info

Creates a ResidualInfo object based on the data passed in and for a particular group.

  • data (curvefit.core.data.DataSpecs): data specifications
  • group_name (str): the name of the group

_run_group_pv

Runs the predictive validity analysis for only one group using the data, model and solver from the run_predictive_validity() function.

run_predictive_validity

Runs predictive validity for all groups in the data.

  • data (curvefit.core.data.Data): data and specifications for the whole analysis
  • group (str): name of the group
  • model (curvefit.models.base.Model): a model object that may be copied depending on whether or not in debug mode
  • solver (curvefit.solvers.solver.Solver): a solver used to fit the model

get_residual_data

Return the out of sample residuals for all groups from the data argument to run_predictive_validity. Important input to the _ResidualModel for eventually creating uncertainty.