data_requirements.md

# Project Resilience Data Requirements and Tips

## Format and Features

Data features in each row of the data should include columns that can be cast
as **Context**, **Actions** and **Outcomes** of a decision pertaining to the
unit of decision-making.

For example, if the problem is carbon emissions decisions per power plant:
- the unit of decision making is a power plant, so each row should represent 
a decision made for a power plant
- Context features are features about the plant that can't be changed
(e.g., location, weather, reactor type)
- Actions are policies for the plant that can be changed within reasonable
time so that the effect can be observed and associated to the action
(e.g., generator setup config, carbon capture level, change in generation
hours)
- Outcomes are quantifiable values that can be attributed to a single region
within a reasonable lag (e.g., carbon emissions, cost of actions, energy
- output)

## Predictability

We need some a priori theory of why/how Actions could affect Outcomes,
and why we should expect prediction of Outcomes to be easier from
Context/Actions rather than from a Context alone. A human being should,
just by looking at the context / action data, be able to predict more or less
what the outcome should be. At least be able to reason about it.
Alternatively, a basic predictor model mapping Context/Actions to Outcomes
should be able to show that it uses the Actions to make predictions better
than with Context alone. This simple predictor model does not need to use
the full data or input/output spaces, it just needs to make it clear that
there's something there.

## Rules of Thumb

### Time-series

Either (1) we have an outcome value at each time step, in which case the row
should indicate the time step; or (2) we have an outcome value only at
particular time steps (e.g., if we have daily power plant CO2 output,
but only monthly cost reports). In any case, if there are time steps which
are missing some values (context, action, or outcome), it's ok if they are
NA in the row for that time step: we can still construct time series to train
on from this dataset.

### Missing Data

To give the project the best chance of success, the amount of missing data
should be minimal and/or structured, e.g., we only get cost reports monthly.

### Data Sufficiency

Data rows should cover variations of decisions sufficiently, and so, in the
case of time-series data, we need historical decision instances that include
different actions taken for similar context.

A single row should represent one observation, which includes context, 
actions, outcomes for that observation.

We need enough cases for our predictor to learn something about how Actions
affect Outcomes. If we have thousands of samples to begin with, that certainly
gives us a better shot. A quick-and-dirty check for correlations between
actions and outcomes could be used as a gating function, i.e., the
correlation matrix should not look like noise. If it looks like noise,
the project may be possible but hard.

Data requirement grows exponentially with number of outcome objectives.

### Consistency

Same context and actions should result in similar outcomes. Contradicting
samples should be minimal. In other words, not too many rows with same
Context and Actions resulting in different Outcomes.

It should be possible to observe the outcome of an action in a reasonable
amount of time (e.g., less than 3 months)

### Availability and Updates to the Data

As a rule of thumb, the number of new samples should be at least on the order
of the problem dimension, or (dim(A) + dim(C)) x (dim(O)). More important
than the number of new samples is which data is sampled: one sample in a
previously-unknown region of interest may be more useful than thousands
in a region we already know well or don't care about. So, if we control
which data is sampled, we don't need as much of it.

### Transparency/Accountability

Data should come from reliable, trusted, scientific, ethical sources.
(e.g. not blackboxes or your mom's Facebook surveys).