final push to this branch

README.qmd

@@ -10,7 +10,7 @@ This repository contains all code, notebooks, data and empirical results for our
 
 The chart below illustrates what we define as macrodynamics in Algorithmic Recourse: (a) we have a simple linear classifier trained for binary classification where samples from the negative class ($y=0$) are marked in blue and samples of the positive class ($y=1$) are marked in orange; (b) the implementation of AR for a random subset of individuals leads to a noticable domain shift; (c) as the classifier is retrained we observe a corresponding model shift; (d) as this process is repeated, the decision boundary moves away from the target class.
 
-![](cover.png)
+![](paper/www/poc.png)
 
 ## Abstract
 

_quarto.yml

@@ -43,7 +43,7 @@ format:
   html:
     theme: cosmo
   epub:
-    cover-image: cover.png
+    cover-image: paper/www/poc.png
 
 execute: 
   freeze: auto

intro.qmd

@@ -15,22 +15,17 @@ To start with, we will look at a proof-of-concept that demonstrates the main obs
 
 We begin by generating the synthetic data for a simple binary classification problem. For illustrative purposes, we will use data that is linearly separable. The chart below shows the data $\mathcal{D}$ at time zero, before any implementation of recourse.
 
+
 ```{julia}
 #| output: true
 #| label: fig-data
 #| fig-cap: "Linearly separable synthetic data"
 
-N = 1000
-xmax = 2
-X, ys = make_blobs(
-    N, 2; 
-    centers=2, as_table=false, center_box=(-xmax => xmax), cluster_std=0.1
-)
-ys .= ys.==2
-X = X'
-xs = Flux.unstack(X,2)
-data = zip(xs,ys)
-counterfactual_data = CounterfactualData(X,ys')
+max_obs = 1000
+catalogue = load_synthetic(max_obs)
+counterfactual_data = catalogue[:linearly_separable]
+X = counterfactual_data.X
+ys = vec(counterfactual_data.y)
 plot()
 scatter!(counterfactual_data)
 ```
@@ -44,6 +39,7 @@ n_epochs = 100
 model = Chain(Dense(2,1))
 mod = FluxModel(model)
 Models.train(mod, counterfactual_data; n_epochs=n_epochs)
+mod_orig = deepcopy(mod)
 ```
 
 @fig-model below shows the linear separation of the two classes.
@@ -72,7 +68,7 @@ Markdown.parse(
 ```
 
 ```{julia}
-opt = Flux.Adam(0.01)
+opt = Flux.Descent(0.01)
 gen = GenericGenerator(;decision_threshold=γ, opt=opt)
 ```
 
@@ -143,5 +139,45 @@ plt_single_repeat = plot(mod,counterfactual_data′;zoom=0,colorbar=false,title=
 
 plt = plot(plt_original, plt_single, plt_single_retrained, plt_single_repeat, layout=(1,4), legend=false, axis=nothing, size=(600,165))
 savefig(plt, joinpath(www_path, "poc.png"))
+savefig(plt, "paper/www/poc.png")
+display(plt)
+```
+
+## Mitigation Strategies
+
+In the paper, we propose three simple mitigation strategies:
+
+1. More Conservative Decision Thresholds
+2. Classifier Preserving ROAR
+3. Gravitational Counterfactual Explanations 
+
+@fig-mitigate shows an illustrative example that demonstrates the differences in counterfactual outcomes when using the various mitigation strategies compared to the baseline approach, that is, Wachter with $\gamma=0.5$: choosing a higher decision threshold pushes the counterfactual a little further into the target domain; this effect is even stronger for ClaPROAR; finally, using the Gravitational generator the counterfactual ends up all the way inside the target domain. 
+
+```{julia}
+#| output: true
+#| label: fig-mitigate
+#| fig-cap: "Mitigation strategies."
+
+# Generators:
+generators = Dict(
+    "Generic (γ=0.5)" => GenericGenerator(opt = opt, decision_threshold=0.5),
+    "Generic (γ=0.9)" => GenericGenerator(opt = opt, decision_threshold=0.9),
+    "Gravitational" => GravitationalGenerator(opt = opt),
+    "ClaPROAR" => ClapROARGenerator(opt = opt)
+)
+
+# Counterfactuals
+x = select_factual(counterfactual_data, rand(candidates)) 
+counterfactuals = Dict([name => generate_counterfactual(x, 1, counterfactual_data, mod_orig, gen;) for (name, gen) in generators])
+
+# Plots:
+plts = []
+for (name,ce) ∈ counterfactuals
+    plt = plot(ce; title=name, colorbar=false, ticks = false, legend=false, zoom=0)
+    plts = vcat(plts..., plt)
+end
+plt = plot(plts..., size=(750,200), layout=(1,4))
+savefig(plt, joinpath(www_path, "mitigation.png"))
+savefig(plt, "paper/www/mitigation.png")
 display(plt)
 ```

paper/paper.Rmd

@@ -125,7 +125,7 @@ knitr::opts_chunk$set(
 ```{r, child=child_docs}
 ```
 
-# Acknowledgement {-}
+# Acknowledgements {-}
 
 Some of the members of TU Delft were partially funded by ICAI AI for Fintech Research, an ING --- TU Delft collaboration. 
 

paper/sections/conclusion.rmd

 # Concluding Remarks {#conclusion}
 
-This work has revisited and extended some of the most general and defining concepts underlying the literature on Counterfactual Explanations and, in particular, Algorithmic Recourse. We demonstrate that long-held beliefs as to what defines optimality in AR, may not always be suitable. Specifically, we run experiments that simulate the application of recourse in practice using various state-of-the-art counterfactual generators and find that all of them induce substantial domain and model shifts. We argue that these shifts should be considered as an expected external cost of individual recourse and call for a paradigm shift from individual to collective recourse in these types of situations. By proposing an adapted counterfactual search objective that incorporates this cost, we make that paradigm shift explicit. We show that this modified objective lends itself to mitigation strategies that can be used to effectively decrease the magnitude of induced domain and model shifts. Through our work, we hope to inspire future research on this important topic. To this end we have open-sourced all of our code along with a Julia package: [`AlgorithmicRecourseDynamics.jl`](https://anonymous.4open.science/r/AlgorithmicRecourseDynamics/README.md). Future researchers should find it relatively easy to replicate, modify and extend the simulation experiments presented here and apply to their own custom counterfactual generators. 
+This work has revisited and extended some of the most general and defining concepts underlying the literature on Counterfactual Explanations and, in particular, Algorithmic Recourse. We demonstrate that long-held beliefs as to what defines optimality in AR, may not always be suitable. Specifically, we run experiments that simulate the application of recourse in practice using various state-of-the-art counterfactual generators and find that all of them induce substantial domain and model shifts. We argue that these shifts should be considered as an expected external cost of individual recourse and call for a paradigm shift from individual to collective recourse in these types of situations. By proposing an adapted counterfactual search objective that incorporates this cost, we make that paradigm shift explicit. We show that this modified objective lends itself to mitigation strategies that can be used to effectively decrease the magnitude of induced domain and model shifts. Through our work, we hope to inspire future research on this important topic. To this end we have open-sourced all of our code along with a Julia package: [`AlgorithmicRecourseDynamics.jl`](https://anonymous.4open.science/r/AlgorithmicRecourseDynamics/README.md). Future researchers should find it easy to replicate, modify and extend the simulation experiments presented here and apply them to their own custom counterfactual generators. 

paper/sections/empirical.rmd

 # Experiment Setup {#empirical}
 
-This section presents the exact ingredients and parameter choices describing the simulation experiments we ran to produce the findings presented in the next section (\@ref(empirical-2)). For convenience, we use Algorithm \ref{algo-experiment} as a template to guide us through this section. A few high-level details upfront: each experiment is run for a total of $T=50$ rounds, where in each round we provide recourse to five per cent of all individuals in the non-target class, so $B_t=0.05 * N_t^{\mathcal{D}_0}$^[As mentioned in the previous section, we end up providing recourse to a total of $\approx50\%$ by the end of round $T=50$.]. All classifiers and generative models are retrained for 10 epochs in each round $t$ of the experiment. Rather than retraining models from scratch, we initialize all parameters at their previous levels ($t-1$) and compute backpropagate for 10 epochs using the new training data as inputs into the existing model. Evaluation metrics are computed and stored every 10 rounds. To account for noise, each individual experiment is repeated five times.^[In the current implementation, we use the train-test split each time to only account for stochasticity associated with randomly selecting individuals for recourse. An interesting alternative may be to also perform data splitting each time, thereby adding an additional layer of randomness.]
+This section presents the exact ingredients and parameter choices describing the simulation experiments we ran to produce the findings presented in the next section (\@ref(empirical-2)). For convenience, we use Algorithm \ref{algo-experiment} as a template to guide us through this section. A few high-level details upfront: each experiment is run for a total of $T=50$ rounds, where in each round we provide recourse to five per cent of all individuals in the non-target class, so $B_t=0.05 * N_t^{\mathcal{D}_0}$. All classifiers and generative models are retrained for 10 epochs in each round $t$ of the experiment. Rather than retraining models from scratch, we initialize all parameters at their previous levels ($t-1$) and backpropagate for 10 epochs using the new training data as inputs into the existing model. Evaluation metrics are computed and stored every 10 rounds. To account for noise, each individual experiment is repeated five times.^[In the current implementation, we use the same train-test split each time to only account for stochasticity associated with randomly selecting individuals for recourse. An interesting alternative may be to also perform data splitting each time, thereby adding an additional layer of randomness.]
 
 ## $M$---Classifiers and Generative Models {#empirical-classifiers}
 
@@ -49,21 +49,21 @@ plt = plot(plts..., layout=(1,4), size=(850,200))
 savefig(plt, "paper/www/synthetic_data.png")
 ```
 
-We use four synthetic binary classification datasets consisting of 1000 samples each: **Overlapping**, **Linearly Separable**, **Circles** and **Moons** (\@ref(fig:synthetic-data)).
+We use four synthetic binary classification datasets consisting of 1000 samples each: **Overlapping**, **Linearly Separable**, **Circles** and **Moons** (Figure \@ref(fig:synthetic-data)).
 
 ```{r synthetic-data, fig.cap="Synthetic classification datasets used in our experiments. Samples from the negative class ($y=0$) are marked in blue while samples of the positive class ($y=1$) are marked in orange."}
 knitr::include_graphics("www/synthetic_data.png")
 ```
 
-Ex-ante we expect to see that by construction, Wachter will create a new cluster of counterfactual instances in the proximity of the initial decision boundary as we saw in Figure \@ref(fig:poc). Thus, the choice of a black-box model may have an impact on the paths of the recourse. For generators that use latent space search (REVISE @joshi2019realistic, CLUE @antoran2020getting) or rely on (and have access to) probabilistic models (CLUE @antoran2020getting, Greedy @schut2021generating) we expect that counterfactuals will end up in regions of the target domain that are densely populated by training samples. Of course, this expectation hinges on how effective said probabilistic models are at capturing predictive uncertainty. Finally, we expect to see the counterfactuals generated by DiCE to be uniformly spread around the feature space inside the target class^[As we mentioned earlier, the diversity constraint used by DiCE is only effective when at least two counterfactuals are being generated. We have therefore decided to always generate 5 counterfactuals for each generator and randomly pick one of them.]. In summary, we expect that the endogenous shifts induced by Wachter outsize those of all other generators since Wachter is not explicitly concerned with generating what we have defined as meaningful counterfactuals. 
+Ex-ante we expect to see that by construction, Wachter will create a new cluster of counterfactual instances in the proximity of the initial decision boundary as we saw in Figure \@ref(fig:poc). Thus, the choice of a black-box model may have an impact on the counterfactual paths. For generators that use latent space search (REVISE @joshi2019realistic, CLUE @antoran2020getting) or rely on (and have access to) probabilistic models (CLUE @antoran2020getting, Greedy @schut2021generating) we expect that counterfactuals will end up in regions of the target domain that are densely populated by training samples. Of course, this expectation hinges on how effective said probabilistic models are at capturing predictive uncertainty. Finally, we expect to see the counterfactuals generated by DiCE to be diversely spread around the feature space inside the target class^[As we mentioned earlier, the diversity constraint used by DiCE is only effective when at least two counterfactuals are being generated. We have therefore decided to always generate 5 counterfactuals for each generator and randomly pick one of them.]. In summary, we expect that the endogenous shifts induced by Wachter outsize those of all other generators since Wachter is not explicitly concerned with generating what we have defined as meaningful counterfactuals. 
 
 ### Real-world data
 
-We use three different real-world datasets from the Finance and Economics domain, all of which are tabular and can be used for binary classification. Firstly, we use the **Give Me Some Credit** dataset which was open-sourced on Kaggle for the task to predict whether a borrower is likely to experience financial difficulties in the next two years [@kaggle2011give], originally consisting of 250,000 instances with 11 numerical attributes. Secondly, we use the **UCI defaultCredit** dataset [@yeh2009comparisons], a benchmark dataset that can be used to train binary classifiers to predict the binary outcome variable of whether credit card clients default on their payment. In its raw form, it consists of 23 explanatory variables: 4 categorical features relating to demographic attributes^[These have been omitted from the analysis. See Section \@ref(limit-data) for details.] and 19 continuous features largely relating to individuals' payment histories and amount of credit outstanding. Both datasets have been used in the literature on AR before (see for example @pawelczyk2021carla, @joshi2019realistic and @ustun2019actionable), presumably because they constitute real-world classification tasks involving individuals that compete for access to credit. 
+We use three different real-world datasets from the Finance and Economics domain, all of which are tabular and can be used for binary classification. Firstly, we use the **Give Me Some Credit** dataset which was open-sourced on Kaggle for the task to predict whether a borrower is likely to experience financial difficulties in the next two years [@kaggle2011give], originally consisting of 250,000 instances with 11 numerical attributes. Secondly, we use the **UCI defaultCredit** dataset [@yeh2009comparisons], a benchmark dataset that can be used to train binary classifiers to predict the binary outcome variable of whether credit card clients default on their payment. In its raw form, it consists of 23 explanatory variables: 4 categorical features relating to demographic attributes and 19 continuous features largely relating to individuals' payment histories and amount of credit outstanding. Both datasets have been used in the literature on AR before (see for example @pawelczyk2021carla, @joshi2019realistic and @ustun2019actionable), presumably because they constitute real-world classification tasks involving individuals that compete for access to credit. 
 
-As a third dataset, we include the **California Housing** dataset derived from the 1990 U.S. census and sourced through scikit-learn [@pedregosa2011scikitlearn, @pace1997sparse]. It consists of 8 continuous features that can be used to predict the median house price for California districts. The continuous outcome variable is binarized as $\tilde{y}=\mathbb{I}_{y>\text{median}(Y)}$ indicating whether or not the median house price of a given district is above or below the median of all districts. While we have not seen this dataset used in the previous literature on AR, others have used the Boston Housing dataset in a similar fashion @schut2021generating. While we initially also conducted experiments on that dataset, we eventually discarded this dataset due to surrounding ethical concerns [@carlisle2019racist].
+As a third dataset, we include the **California Housing** dataset derived from the 1990 U.S. census and sourced through scikit-learn [@pedregosa2011scikitlearn, @pace1997sparse]. It consists of 8 continuous features that can be used to predict the median house price for California districts. The continuous outcome variable is binarized as $\tilde{y}=\mathbb{I}_{y>\text{median}(Y)}$ indicating whether or not the median house price of a given district is above the median of all districts. While we have not seen this dataset used in the previous literature on AR, others have used the Boston Housing dataset in a similar fashion @schut2021generating. We initially also conducted experiments on that dataset, but eventually discarded it due to surrounding ethical concerns [@carlisle2019racist].
 
-Since the simulations involve generating counterfactuals for a significant proportion of the entire sample of individuals, we have randomly undersampled each dataset to yield balanced subsamples consisting of 5,000 individuals each. We have also standardized all explanatory features since our chosen classifiers are sensitive to scale.
+Since the simulations involve generating counterfactuals for a significant proportion of the entire sample of individuals, we have randomly undersampled each dataset to yield balanced subsamples consisting of 5,000 individuals each. We have also standardized all continuous explanatory features since our chosen classifiers are sensitive to scale.
 
 ## $G$---Generators
 

paper/sections/empirical_2.rmd

@@ -6,7 +6,7 @@ Below, we first present our main experimental findings regarding these questions
 
 We start this section off with the key high-level observations. Across all datasets (synthetic and real), classifiers and counterfactual generators we observe either most or all of the following dynamics at varying degrees:
 
-- Statistically significant domain and model shift as measured by MMD. 
+- Statistically significant domain and model shifts as measured by MMD. 
 - A deterioration in out-of-sample model performance as measured by the F-Score evaluated on a test sample. In many cases this drop in performance is substantial.
 - Significant perturbations to the model parameters as well as an increase in the model's decisiveness.
 - Disagreement between the original and retrained model, in some cases large.

paper/sections/introduction.rmd

 # Introduction {#intro}
 
-Recent advances in Artificial Intelligence (AI) have propelled its adoption in scientific domains outside of Computer Science including Healthcare, Bioinformatics, Genetics and the Social Sciences. While this has in many cases brought benefits in terms of efficiency, state-of-the-art models like Deep Neural Networks (DNN) have also given rise to a new type of problem in the context of data-driven decision-making. They are essentially **black boxes**: so complex, opaque and underspecified in the data that it is often impossible to understand how they actually arrive at their decision without auxiliary tools. Despite this shortcoming, black-box models have grown in popularity in recent years and have at times created undesirable societal outcomes [@oneil2016weapons]. The scientific community has tackled this issue from two different angles: while some have appealed for a strict focus on inherently interpretable models [@rudin2019stop], others have investigated different ways to explain the behaviour of black-box models. These two sub-domains can be broadly referred to as **interpretable AI** and **explainable AI** (XAI), respectively. 
+Recent advances in Artificial Intelligence (AI) have propelled its adoption in scientific domains outside of Computer Science including Healthcare, Bioinformatics, Genetics and the Social Sciences. While this has in many cases brought benefits in terms of efficiency, state-of-the-art models like Deep Neural Networks (DNN) have also given rise to a new type of problem in the context of data-driven decision-making. They are essentially **black boxes**: so complex, opaque and underspecified in the data that it is often impossible to understand how they actually arrive at their decisions without auxiliary tools. Despite this shortcoming, black-box models have grown in popularity in recent years and have at times created undesirable societal outcomes [@oneil2016weapons]. The scientific community has tackled this issue from two different angles: while some have appealed for a strict focus on inherently interpretable models [@rudin2019stop], others have investigated different ways to explain the behaviour of black-box models. These two sub-domains can be broadly referred to as **interpretable AI** and **explainable AI** (XAI), respectively. 
 
 Among the approaches to XAI that have recently grown in popularity are **Counterfactual Explanations** (CE). They explain how inputs into a model need to change for it to produce different outputs. Counterfactual Explanations that involve realistic and actionable changes can be used for the purpose of **Algorithmic Recourse** (AR) to help individuals who face adverse outcomes. An example relevant to the Social Sciences is consumer credit: in this context, AR can be used to guide individuals in improving their creditworthiness, should they have previously been denied access to credit based on an automated decision-making system. A meaningful recourse recommendation for a denied applicant could be: *"If your net savings rate had been 10% of your monthly income instead of the actual 8%, your application would have been successful. See if you can temporarily cut down on consumption."* In the remainder of this paper, we will use both terminologies---recourse and counterfactual---interchangeably to refer to situations where counterfactuals are generated with the intent to provide individual recourse.
 
@@ -8,23 +8,23 @@ Existing work in this field has largely worked in a static setting: various appr
 
 Research on Algorithmic Recourse has also so far typically addressed the issue from the perspective of a single individual. Arguably though, most real-world applications that warrant AR involve potentially large groups of individuals typically competing for scarce resources. Our work demonstrates that in such scenarios, choices made by or for a single individual are likely to affect the broader collective of individuals in ways that many current approaches to AR fail to account for. More specifically, we argue that a strict focus on minimizing the private costs to individuals may be too narrow an objective.
 
-Figure \@ref(fig:poc) illustrates this idea for a binary problem involving a linear classifier and the counterfactual generator proposed by Wachter et al. @wachter2017counterfactual: the implementation of AR for a subset of individuals immediately leads to a visible domain shift in the (orange) target class (b), which in turn triggers a model shift (c). As this game of implementing AR and updating the classifier is repeated, the decision boundary moves away from training samples that were originally in the target class (d). We refer to these types of dynamics as **endogenous** because they are induced by the implementation of recourse itself. The term **macrodynamics** is borrowed from the economics literature and used to describe processes involving whole groups or societies.
+Figure \@ref(fig:poc) illustrates this idea for a binary problem involving a linear classifier and the counterfactual generator proposed by Wachter et al. @wachter2017counterfactual: the implementation of AR for a subset of individuals immediately leads to a visible domain shift in the (blue) target class (b), which in turn triggers a model shift (c). As this game of implementing AR and updating the classifier is repeated, the decision boundary moves away from training samples that were originally in the target class (d). We refer to these types of dynamics as **endogenous** because they are induced by the implementation of recourse itself. The term **macrodynamics** is borrowed from the economics literature and used to describe processes involving whole groups or societies.
 
 ```{r poc, fig.cap="Dynamics in Algorithmic Recourse: (a) we have a simple linear classifier trained for binary classification where samples from the negative class ($y=0$) are marked in orange and samples of the positive class ($y=1$) are marked in blue; (b) the implementation of AR for a random subset of individuals leads to a noticeable domain shift; (c) as the classifier is retrained we observe a corresponding model shift; (d) as this process is repeated, the decision boundary moves away from the target class."}
 knitr::include_graphics("www/poc.png")
 ```
 
-We think that these types of endogenous dynamics may be problematic and deserve our attention. From a purely technical perspective, we note the following: firstly, model shifts may inadvertently change classification outcomes for individuals who never received and implemented recourse. Secondly, we observe in Figure \@ref(fig:poc) that as the decision boundary moves in the direction of the non-target class, counterfactual paths become shorter. We think that in some practical applications, this can be expected to generate costs for involved stakeholders. To follow our argument, consider the following two examples:
+We think that these types of endogenous dynamics may be problematic and deserve our attention. From a purely technical perspective, we note the following: firstly, model shifts may inadvertently change classification outcomes for individuals who never received recourse. Secondly, we observe in Figure \@ref(fig:poc) that as the decision boundary moves in the direction of the non-target class, counterfactual paths become shorter. We think that in some practical applications, this can be expected to generate costs for involved stakeholders. To follow our argument, consider the following two examples:
 
 ::: {.example #consumer name="Consumer Credit"}
-Suppose Figure \@ref(fig:poc) relates to an automated decision-making system used by a retail bank to evaluate credit applicants with respect to their creditworthiness. Assume that the two features are meaningful in the sense that creditworthiness increases in the South-East direction. Then we can think of the outcome in panel (d) as representing a situation where the bank supplies credit to more borrowers (orange), but these borrowers are on average less creditworthy and more of them can be expected to default on their loan. This represents a cost to the retail bank.
+Suppose Figure \@ref(fig:poc) relates to an automated decision-making system used by a retail bank to evaluate credit applicants with respect to their creditworthiness. Assume that the two features are meaningful in the sense that creditworthiness increases in the South-East direction. Then we can think of the outcome in panel (d) as representing a situation where the bank supplies credit to more borrowers (blue), but these borrowers are on average less creditworthy and more of them can be expected to default on their loan. This represents a cost to the retail bank.
 :::
 
 ::: {.example #student name="Student Admission"}
-Suppose Figure \@ref(fig:poc) relates to an automated decision-making system used by a university in its student admission process. Assume that the two features are meaningful in the sense that the likelihood of students completing their degree increases in the South-East direction. Then we can think of the outcome in panel (b) as representing a situation where more students are admitted to university (orange), but they are more likely to fail their degree than students that were admitted in previous years. The university admission committee catches on to this and suspends its efforts to offer Algorithmic Recourse. This represents an opportunity cost to future student applicants, that may have derived utility from being offered recourse.
+Suppose Figure \@ref(fig:poc) relates to an automated decision-making system used by a university in its student admission process. Assume that the two features are meaningful in the sense that the likelihood of students completing their degree increases in the South-East direction. Then we can think of the outcome in panel (b) as representing a situation where more students are admitted to university (blue), but they are more likely to fail their degree than students that were admitted in previous years. The university admission committee catches on to this and suspends its efforts to offer Algorithmic Recourse. This represents an opportunity cost to future student applicants, that may have derived utility from being offered recourse.
 :::
 
 Both examples are exaggerated simplifications of potential real-world scenarios, but they serve to illustrate the point that recourse for one single individual may exert negative externalities on other individuals.
 
-To the best of our knowledge, this is the first work investigating endogenous macrodynamics in AR. Our contributions to the state of knowledge are as follows: firstly, we posit a compelling argument that calls for a novel perspective on Algorithmic Recourse extending our focus from single individuals to groups (Sections \@ref(related) and \@ref(method)). Secondly, we introduce an experimental framework extending previous work by Altmeyer @altmeyer2022counterfactualexplanations, which enables us to study macrodynamics of Algorithmic Recourse through simulations that can be fully parallelized (Section \@ref(method-2)). Thirdly, we use this framework to provide a first in-depth analysis of endogenous recourse dynamics induced by various popular counterfactual generators including @wachter2017counterfactual, @schut2021generating, @joshi2019realistic, @mothilal2020explaining and @antoran2020getting (Sections \@ref(empirical) and \@ref(empirical-2)). Fourthly, given that we find a substantial impact of recourse, we propose key mitigation strategies and measure their impact experimentally (Section \@ref(mitigate)). Finally, we discuss our findings in the broader context of the literature in Section \@ref(discussion), before pointing to some of the limitations of our work as well as avenues for future research in Section \@ref(limit). Section \@ref(conclusion) concludes.
+To the best of our knowledge, this is the first work investigating endogenous macrodynamics in AR. Our contributions to the state of knowledge are as follows: firstly, we posit a compelling argument that calls for a novel perspective on Algorithmic Recourse extending our focus from single individuals to groups (Sections \@ref(related) and \@ref(method)). Secondly, we introduce an experimental framework extending previous work by Altmeyer @altmeyer2022counterfactualexplanations, which enables us to study macrodynamics of Algorithmic Recourse through simulations that can be fully parallelized (Section \@ref(method-2)). Thirdly, we use this framework to provide a first in-depth analysis of endogenous recourse dynamics induced by various popular counterfactual generators proposed in @wachter2017counterfactual, @schut2021generating, @joshi2019realistic, @mothilal2020explaining and @antoran2020getting (Sections \@ref(empirical) and \@ref(empirical-2)). Fourthly, given that we find a substantial impact of recourse, we propose and assess various mitigation strategies (Section \@ref(mitigate)). Finally, we discuss our findings in the broader context of the literature in Section \@ref(discussion), before pointing to some of the limitations of our work as well as avenues for future research in Section \@ref(limit). Section \@ref(conclusion) concludes.
 

paper/sections/limitations.rmd

 # Limitations and Future Work {#limit}
 
-While we believe that this work constitutes a valuable starting point for addressing existing issues in Algorithmic Recourse from a fresh perspective, we are aware of several of its limitations. In the following, we highlight some of these limitations and point to avenues for future research.
+While we believe that this work constitutes a valuable starting point for addressing existing issues in Algorithmic Recourse from a fresh perspective, we are aware of several of its limitations. In the following, we highlight some of these and point to avenues for future research.
 
 ## Private vs. External Costs
 
-Perhaps the most crucial shortcoming of our work is that we merely point out that there exists a trade-off between private costs to the individual and external costs to the collective of stakeholders. We fall short of providing any definitive answers as to how that trade-off may be resolved in practice. The mitigation strategies we have proposed here provide a good starting point, but they are ad-hoc, mechanical extensions of the existing AR framework. An interesting idea to explore in future work could be the potential for Pareto optimal Algorithmic Recourse, that is, a collective recourse outcome in which no single individual can be made better off, without making at least one other individual worse off. This type of work would be interdisciplinary and could help to formalize some of the concepts presented in this work.
+Perhaps the most crucial shortcoming of our work is that we merely point out that there exists a trade-off between private costs to the individual and external costs to the collective of stakeholders. We fall short of providing any definitive answers as to how that trade-off may be resolved in practice. The mitigation strategies we have proposed here provide a good starting point, but they are ad-hoc extensions of the existing AR framework. An interesting idea to explore in future work could be the potential for Pareto optimal Algorithmic Recourse, that is, a collective recourse outcome in which no single individual can be made better off, without making at least one other individual worse off. This type of work would be interdisciplinary and could help to formalize some of the concepts presented in this work.
 
 ## Experimental Setup
 

paper/sections/methodology.rmd

 # Gradient-Based Recourse Revisited {#method}
 
-In this section, we first set out a generalized framework for gradient-based counterfactual search that encapsulates the various individual recourse methods we have chosen to use in our experiments (Section \@ref(method-general)). We then introduce the notion of a hidden external cost in algorithmic recourse and extend the existing framework to explicitly address this cost in the counterfactual search objective (Section \@ref(method-collective)).
+In this section, we first set out a generalized framework for gradient-based counterfactual search that encapsulates the various Individual Recourse methods we have chosen to use in our experiments (Section \@ref(method-general)). We then introduce the notion of a hidden external cost in Algorithmic Recourse and extend the existing framework to explicitly address this cost in the counterfactual search objective (Section \@ref(method-collective)).
 
-## From individual recourse ... {#method-general}
+## From Individual Recourse ... {#method-general}
 
-We have chosen to focus on gradient-based counterfactual search for two reasons: firstly, they can be seen as direct descendants of our baseline method (Wachter); secondly, gradient-based search is particularly well-suited for differentiable black-box models like deep neural networks, which we focus on in this work. In particular, we include the following generators in our simulation experiments below: **REVISE** [@joshi2019realistic], **CLUE** [@antoran2020getting], **DiCE** [@mothilal2020explaining] and a greedy approach that relies on probabilistic models [@schut2021generating]. Our motivation for including these different generators in our analysis is that they all offer slightly different approaches to generating meaningful counterfactuals for differentiable black-box models. We hypothesize that generating more **meaningful** counterfactuals should mitigate the endogenous dynamics illustrated in Figure \@ref(fig:poc) in Section \@ref(intro). This intuition stems from the underlying idea that more meaningful counterfactuals are generated by the same or at least a very similar data-generating process as the training data. All else equal, counterfactuals that fulfil this basic requirement should be less prone to trigger shifts. 
+We have chosen to focus on gradient-based counterfactual search for two reasons: firstly, they can be seen as direct descendants of our baseline method (Wachter); secondly, gradient-based search is particularly well-suited for differentiable black-box models like deep neural networks, which we focus on in this work. In particular, we include the following generators in our simulation experiments below: **REVISE** [@joshi2019realistic], **CLUE** [@antoran2020getting], **DiCE** [@mothilal2020explaining] and a greedy approach that relies on probabilistic models [@schut2021generating]. Our motivation for including these different generators in our analysis is that they all offer slightly different approaches to generating meaningful counterfactuals for differentiable black-box models. We hypothesize that generating more **meaningful** counterfactuals should mitigate the endogenous dynamics illustrated in Figure \@ref(fig:poc) in Section \@ref(intro). This intuition stems from the underlying idea that more meaningful counterfactuals are generated by the same or at least a very similar data-generating process as the observed data. All else equal, counterfactuals that fulfil this basic requirement should be less prone to trigger shifts. 
 
 As we will see next, all of them can be described by the following generalized form of Equation \@ref(eq:general):
 
@@ -18,11 +18,11 @@ Here $\mathbf{s}^\prime=\left\{s_k^\prime\right\}_K$ is a $K$-dimensional array
 
 CLUE distinguishes itself from REVISE and other counterfactual generators in that it aims to minimize the predictive uncertainty of the model in question, $M$. To quantify predictive uncertainty, Antoran et al. [@antoran2020getting] rely on entropy estimates for probabilistic models. The greedy approach proposed by Schut et al. @schut2021generating, which we refer to as **Greedy**, also works with the subclass of models $\tilde{\mathcal{M}}\subset\mathcal{M}$ that can produce predictive uncertainty estimates. The authors show that in this setting the cost function $\text{cost}(\cdot)$ in Equation \@ref(eq:general) is redundant and meaningful counterfactuals can be generated in a fast and efficient manner through a modified Jacobian-based Saliency Map Attack (JSMA). Schut et al. [@schut2021generating] also show that by maximizing the predicted probability of $x^\prime$ being assigned to target class $y^*$, we also implicitly minimize predictive entropy (as in CLUE). In that sense, CLUE can be seen as equivalent to REVISE in the Bayesian context and we shall therefore refer to both approaches collectively as **Latent Space** generators^[In fact, there are several other recently proposed approaches to counterfactual search that also broadly fall in this same category. They largely differ with respect to the chosen generative model: for example, the generator proposed by Dombrowski et al. @dombrowski2021diffeomorphic relies on normalizing flows.]. 
 
-Finally, DiCE [@mothilal2020explaining] distinguishes itself from all other generators considered here in that it aims to generate a diverse set of $K>1$ counterfactuals. Wachter et al. [@wachter2017counterfactual] show that diverse outcomes can in principle be achieved simply by rerunning counterfactual search multiple times using stochastic gradient descent (or by randomly initializing the counterfactual)^[Note that \@ref(eq:general) naturally lends itself to that idea: setting $K$ to some value greater than one and using the Wachter objective essentially boils down to computing multiple counterfactuals in parallel. Here, $yloss(\cdot)$ is first broadcasted over elements of $\mathbf{s}^\prime$ and then aggregated. This is exactly how counterfactual search is implemented in [`CounterfactualExplanations.jl`](https://github.com/pat-alt/CounterfactualExplanations.jl).]. In @mothilal2020explaining diversity is explicitly proxied via Determinantal Point Processes (DDP): the authors simply introduce DDP as a component of the cost function $\text{cost}(\mathbf{s}^\prime)$ and thereby produce counterfactuals $s_1, ..., s_K$ that look as different from each other as possible. The implementation of DiCE in our library of choice---[`CounterfactualExplanations.jl`](https://github.com/pat-alt/CounterfactualExplanations.jl)---uses that exact approach. It is worth noting that for $k=1$, DiCE reduces to Wachter since the DDP is constant and therefore does not affect the objective function in Equation \@ref(eq:general).
+Finally, DiCE [@mothilal2020explaining] distinguishes itself from all other generators considered here in that it aims to generate a diverse set of $K>1$ counterfactuals. Wachter et al. [@wachter2017counterfactual] show that diverse outcomes can in principle be achieved simply by rerunning counterfactual search multiple times using stochastic gradient descent (or by randomly initializing the counterfactual)^[Note that \@ref(eq:general) naturally lends itself to that idea: setting $K$ to some value greater than one and using the Wachter objective essentially boils down to computing multiple counterfactuals in parallel. Here, $yloss(\cdot)$ is first broadcasted over elements of $\mathbf{s}^\prime$ and then aggregated. This is exactly how counterfactual search is implemented in [`CounterfactualExplanations.jl`](https://github.com/pat-alt/CounterfactualExplanations.jl).]. In @mothilal2020explaining diversity is explicitly proxied via Determinantal Point Processes (DDP): the authors introduce DDP as a component of the cost function $\text{cost}(\mathbf{s}^\prime)$ and thereby produce counterfactuals $s_1, ..., s_K$ that look as different from each other as possible. The implementation of DiCE in our library of choice---[`CounterfactualExplanations.jl`](https://github.com/pat-alt/CounterfactualExplanations.jl)---uses that exact approach. It is worth noting that for $k=1$, DiCE reduces to Wachter since the DDP is constant and therefore does not affect the objective function in Equation \@ref(eq:general).
 
-## ... towards collective recourse {#method-collective}
+## ... towards Collective Recourse {#method-collective}
 
-All of the different approaches introduced above tackle the problem of Algorithmic Recourse from the perspective of one single individual^[DiCE recognizes that different individuals may have different objective functions, but it does not address the interdependencies between different individuals.]. To explicitly address the issue that individual recourse may affect the outcome and prospect of other individuals, we propose to extend Equation \@ref(eq:general) as follows:
+All of the different approaches introduced above tackle the problem of Algorithmic Recourse from the perspective of one single individual^[DiCE recognizes that different individuals may have different objective functions, but it does not address the interdependencies between different individuals.]. To explicitly address the issue that Individual Recourse may affect the outcome and prospect of other individuals, we propose to extend Equation \@ref(eq:general) as follows:
 
 \begin{equation}
 \begin{aligned}
@@ -30,7 +30,7 @@ All of the different approaches introduced above tackle the problem of Algorithm
 \end{aligned} 
 \end{equation}
 
-Here $\text{cost}(f(\mathbf{s}^\prime))$ denotes the proxy for private costs faced by the individual as before and $\lambda_1$ governs to what extent that private cost ought to be penalized. The newly introduced term $\text{extcost}(f(\mathbf{s}^\prime))$ is meant to capture and address external costs incurred by the collective of individuals in response to changes in $\mathbf{s}^\prime$. The underlying concept of private and external costs is borrowed from Economics and well-established in that field: when the decisions or actions by some individual market participant generate external costs, then the market is said to suffer from negative externalities and is considered inefficient [@pindyck2014microeconomics]. We think that this concept describes the endogenous dynamics of algorithmic recourse observed here very well. As with individual recourse, the exact choice of $\text{extcost}(\cdot)$ is not obvious, nor do we intend to provide a definite answer in this work, if such even exists. That being said, we do propose a few potential mitigation strategies in Section \@ref(mitigate).
+Here $\text{cost}(f(\mathbf{s}^\prime))$ denotes the proxy for private costs faced by the individual as before and $\lambda_1$ governs to what extent that private cost ought to be penalized. The newly introduced term $\text{extcost}(f(\mathbf{s}^\prime))$ is meant to capture and address external costs incurred by the collective of individuals in response to changes in $\mathbf{s}^\prime$. The underlying concept of private and external costs is borrowed from Economics and well-established in that field: when the decisions or actions by some individual market participant generate external costs, then the market is said to suffer from negative externalities and is considered inefficient [@pindyck2014microeconomics]. We think that this concept describes the endogenous dynamics of algorithmic recourse observed here very well. As with Individual Recourse, the exact choice of $\text{extcost}(\cdot)$ is not obvious, nor do we intend to provide a definitive answer in this work, if such even exists. That being said, we do propose a few potential mitigation strategies in Section \@ref(mitigate).
 
 
 

paper/sections/methodology_2.rmd

-# Modeling Endogenous Macrodynamics in Algorithmic Recourse {#method-2}
+# Modelling Endogenous Macrodynamics in Algorithmic Recourse {#method-2}
 
 In the following, we describe the framework we propose for modelling and analyzing endogenous macrodynamics in Algorithmic Recourse. We introduce this framework with the ambition to shed light on the following research questions:
 
@@ -46,13 +46,13 @@ In order to simulate the dynamic process, we suppose that the model $M$ is retra
 
 Note that the operation in line 4 is an assignment, rather than a copy operation, so any updates to 'batch' will also affect $\mathcal{D}$. The function $\text{eval}(M,\mathcal{D})$ loosely denotes the computation of various evaluation metrics introduced below. In practice, these metrics can also be computed at regular intervals as opposed to every round. 
 
-Along with any other fixed parameters affecting the counterfactual search, the parameters $T$ and $B$ are assumed as given in Algorithm \ref{algo-experiment}. Still, it is worth noting that the higher these values, the more factual instances undergo recourse throughout the entire experiment. Of course, this is likely to lead to more pronounced domain and model shifts by time $T$. In our experiments, we choose the values such that $T \cdot B$ corresponds to the application of recourse on $\approx50\%$ of the negative instances from the initial dataset. As we compute evaluation metrics at regular intervals throughout the procedure, we can also verify the impact of recourse when it is implemented for a smaller number of individuals. 
+Along with any other fixed parameters affecting the counterfactual search, the parameters $T$ and $B$ are assumed as given in Algorithm \ref{algo-experiment}. Still, it is worth noting that the higher these values, the more factual instances undergo recourse throughout the entire experiment. Of course, this is likely to lead to more pronounced domain and model shifts by time $T$. In our experiments, we choose the values such that the majority of the negative instances from the initial dataset receive recourse. As we compute evaluation metrics at regular intervals throughout the procedure, we can also verify the impact of recourse when it is implemented for a smaller number of individuals. 
 
 Algorithm \ref{algo-experiment} summarizes the proposed simulation experiment for a given dataset $\mathcal{D}$, model $M$ and generator $G$, but naturally, we are interested in comparing simulation outcomes for different sources of data, models and generators. The framework we have built facilitates this, making use of multi-threading in order to speed up computations. Holding the initial model and dataset constant, the experiments are run for all generators, since our primary concern is to benchmark different recourse methods. To ensure that each generator is faced with the same initial conditions in each round $t$, the candidate batch of individuals from the non-target class is randomly drawn from the intersection of all non-target class individuals across all experiments $\left\{\textsc{Experiment}(M,\mathcal{D},G)\right\}_{j=1}^J$ where $J$ is the total number of generators. 
 
 ## Evaluation Metrics {#method-2-metrics}
 
-We formulate two desiderata for the set of metrics used to measure domain and model shifts induced by recourse. First, the metrics should be applicable regardless of the dataset or classification technique so that they allow for the meaningful comparison of the generators in various scenarios. As knowledge of the underlying probability distribution is rarely available, the metrics should be empirical and non-parametric. This further ensures that we can also measure large datasets by sampling from the available data. Moreover, while our study was conducted in a two-class classification setting, our choice of metrics should remain applicable in future research on multi-class recourse problems. Second, the set of metrics should allow capturing various aspects of the previously mentioned magnitude, path, and tempo of changes while remaining as small as possible.
+We formulate two desiderata for the set of metrics used to measure domain and model shifts induced by recourse. First, the metrics should be applicable regardless of the dataset or classification technique so that they allow for the meaningful comparison of the generators in various scenarios. As knowledge of the underlying probability distribution is rarely available, the metrics should be empirical and non-parametric. This further ensures that we can also measure large datasets by sampling from the available data. Moreover, while our study was conducted in a two-class classification setting, our choice of metrics should remain applicable in future research on multi-class recourse problems. Second, the set of metrics should allow capturing various aspects of the previously mentioned magnitude, path, and pace of changes while remaining as small as possible.
 
 ### Domain Shifts
 
@@ -66,7 +66,7 @@ MMD({X}^\prime,\tilde{X}^\prime) &= \frac{1}{m(m-1)}\sum_{i=1}^m\sum_{j\neq i}^m
 
 where $X=\{x_1,...,x_m\}$, $\tilde{X}=\{\tilde{x}_1,...,\tilde{x}_n\}$ represent independent and identically distributed samples drawn from probability distributions $\mathcal{X}$ and $\mathcal{\tilde{X}}$ respectively @gretton2012kernel. MMD is a measure of the distance between the kernel mean embeddings of $\mathcal{X}$ and $\mathcal{\tilde{X}}$ in a Reproducing Kernel Hilbert Space, $\mathcal{H}$ [@berlinet2011reproducing]. An important consideration is the choice of the kernel function $k(\cdot,\cdot)$. In our implementation, we make use of a Gaussian kernel with a constant length-scale parameter of $0.5$. As the Gaussian kernel captures all moments of distributions $\mathcal{X}$ and $\mathcal{\tilde{X}}$, we have that $MMD(X,\tilde{X})=0$ if and only if $X=\tilde{X}$. Conversely, larger values $MMD(X,\tilde{X})>0$ indicate that it is more likely that $\mathcal{X}$ and $\mathcal{\tilde{X}}$ are different distributions. In our context, large values, therefore, indicate that a domain shift indeed seems to have occurred.
 
-To assess the statistical significance of the observed shifts under the null hypothesis that samples $X$ and $\tilde{X}$ were drawn from the same probability distribution, we follow @arcones1992bootstrap. To that end, we combine the two samples and generate a large number of permutations of $X + \tilde{X}$. Then, we split the permuted data into two new samples $X^\prime$ and $\tilde{X}^\prime$ having the same size as the original samples. Then under the null hypothesis, we should have that $MMD(X^\prime,\tilde{X}^\prime)$ be approximately equal to $MMD(X,\tilde{X})$. The corresponding $p$-value can then be calculated by counting how these two quantities are not equal.
+To assess the statistical significance of the observed shifts under the null hypothesis that samples $X$ and $\tilde{X}$ were drawn from the same probability distribution, we follow @arcones1992bootstrap. To that end, we combine the two samples and generate a large number of permutations of $X + \tilde{X}$. Then, we split the permuted data into two new samples $X^\prime$ and $\tilde{X}^\prime$ having the same size as the original samples. Under the null hypothesis, we should have that $MMD(X^\prime,\tilde{X}^\prime)$ be approximately equal to $MMD(X,\tilde{X})$. The corresponding $p$-value can then be calculated by counting how often these two quantities are not equal.
 
 We calculate the MMD for both classes individually based on the ground truth labels, i.e. the labels that samples were assigned in time $t=0$. Throughout our experiments, we generally do not expect the distribution of the negative class to change over time – application of recourse reduces the size of this class, but since individuals are sampled uniformly the distribution should remain unaffected. Conversely, unless a recourse generator can perfectly replicate the original probability distribution, we expect the MMD of the positive class to increase. Thus, when discussing MMD, we generally mean the shift in the distribution of the positive class.
 

paper/sections/mitigation.rmd

@@ -6,7 +6,7 @@ Having established in the previous section that endogenous macrodynamics in AR a
 What are potential mitigation strategies with respect to endogenous macrodynamics in AR?
 :::
 
-We proposed and test several simple mitigation strategies. All of them essentially boil down to one simple principle: to avoid substantial domain and model shifts, the generated counterfactuals should comply as much as possible with the true data-generating process. This principle is really at the core of Latent Space (LS) generators, and hence it is not surprising that we have found these types of generators to perform comparably well in the previous section. But as we have mentioned earlier, generators that rely on separate generative models carry an additional computational burden and, perhaps more importantly, their performance hinges on the performance of said generative models. Fortunately, it turns out that we can use a number of other, much simpler strategies.
+We propose and test several simple mitigation strategies. All of them essentially boil down to one simple principle: to avoid domain and model shifts, the generated counterfactuals should comply as much as possible with the true data-generating process. This principle is really at the core of Latent Space (LS) generators, and hence it is not surprising that we have found these types of generators to perform comparably well in the previous section. But as we have mentioned earlier, generators that rely on separate generative models carry an additional computational burden and, perhaps more importantly, their performance hinges on the performance of said generative models. Fortunately, it turns out that we can use a number of other, much simpler strategies.
 
 ## More Conservative Decision Thresholds
 
@@ -34,9 +34,9 @@ Yet another strategy extends Wachter as follows: instead of only penalizing the
 \end{aligned}
 \end{equation}
 
-Once again we can put this in the context of Equation \@ref(eq:collective): the former penalty can be thought of here as the private cost incurred by the individual, while the latter reflects the external cost incurred by other individuals. Higher choices of $\lambda_2$ relative to $\lambda_1$ will lead counterfactuals to gravitate towards the specified point $\bar{x}$ in the target domain. In the remainder of this paper, we will therefore refer to this approach as **Gravitational** generator, when we investigate its usefulness for mitigating endogenous macrodynamics^[Note that despite the naming conventions, our goal here is not to provide yet more counterfactual generators. Rather than looking at them as isolated entities, we believe and demonstrate that different approaches can be effectively combined.]. 
+Once again we can put this in the context of Equation \@ref(eq:collective): the former penalty can be thought of here as the private cost incurred by the individual, while the latter reflects the external cost incurred by other individuals. Higher choices of $\lambda_2$ relative to $\lambda_1$ will lead counterfactuals to gravitate towards the specified point $\bar{x}^*$ in the target domain. In the remainder of this paper, we will therefore refer to this approach as **Gravitational** generator, when we investigate its usefulness for mitigating endogenous macrodynamics^[Note that despite the naming conventions, our goal here is not to provide yet more counterfactual generators. Rather than looking at them as isolated entities, we believe and demonstrate that different approaches can be effectively combined.]. 
 
-Figure \@ref(fig:mitigation) shows an illustrative example that demonstrates the differences in counterfactual outcomes when using the various mitigation strategies compared to the baseline approach, that is, Wachter with $\gamma=0.5$: choosing a higher decision threshold pushes the counterfactual a little further into the target domain; this effect is even stronger for ClaPROAR; finally, using the Gravitational generator the counterfactual ends up all the way inside the target domain in the neighbourhood of $\bar{x}$^[In order for the Gravitational generator and ClaPROAR to work as expected, one needs to ensure that counterfactual search continues, independent of the threshold probability $\gamma$.]. Linking these ideas back to Example \@ref(exm:student), the mitigation strategies help ensure that the recommended recourse actions are substantial enough to truly lead to an increase in the probability that the admitted student eventually graduates.
+Figure \@ref(fig:mitigation) shows an illustrative example that demonstrates the differences in counterfactual outcomes when using the various mitigation strategies compared to the baseline approach, that is, Wachter with $\gamma=0.5$: choosing a higher decision threshold pushes the counterfactual a little further into the target domain; this effect is even stronger for ClaPROAR; finally, using the Gravitational generator the counterfactual ends up all the way inside the target domain in the neighbourhood of $\bar{x}^*$^[In order for the Gravitational generator and ClaPROAR to work as expected, one needs to ensure that counterfactual search continues, independent of the threshold probability $\gamma$.]. Linking these ideas back to Example \@ref(exm:student), the mitigation strategies help ensure that the recommended recourse actions are substantial enough to truly lead to an increase in the probability that the admitted student eventually graduates.
 
 ```{julia, eval=FALSE, echo=FALSE}
 using Random
@@ -83,7 +83,7 @@ savefig(plt, "paper/www/mitigation.png")
 ```
 
 
-```{r mitigation, fig.cap="Illustrative example demonstrating the properties of the various mitigation strategies. Samples from the negative class ($y=0$) are marked in blue while samples of the positive class ($y=1$) are marked in orange."}
+```{r mitigation, fig.cap="Illustrative example demonstrating the properties of the various mitigation strategies. Samples from the negative class ($y=0$) are marked in orange while samples of the positive class ($y=1$) are marked in blue."}
 knitr::include_graphics("www/mitigation.png")
 ```
 

paper/sections/related.rmd

 # Background {#related}
 
-In this section, we provide a review of the relevant literature. First, Subsection \@ref(related-recourse) discusses the existing research within the domain of Counterfactual Explanations and Algorithmic Recourse. Then, Subsection \@ref(related-shifts) presents some of the previous work on the measurement of dataset and model shifts.
+In this section, we provide a review of the relevant literature. First, Subsection \@ref(related-recourse) discusses the existing research within the domain of Counterfactual Explanations and Algorithmic Recourse. Then, Subsection \@ref(related-shifts) presents some of the previous work on the measurement of data and model shifts.
 
 ## Algorithmic Recourse {#related-recourse}
 
-A framework for Counterfactual Explanations was first proposed in 2017 by Wachter et al. @wachter2017counterfactual and has served as the baseline for most methodologies that have been proposed since then. Let $M: \mathcal{X} \mapsto \mathcal{Y}$ denote some pre-trained model that maps from inputs $X \in \mathcal{X}$ to outputs $Y \in \mathcal{Y}$. Then we are interested in minimizing the cost^[Equivalently, others have referred to this quantity as *complexity* or simply *distance*.] $C=\text{cost}(x^\prime)$ incurred by individual $x$ when moving to a counterfactual state $x^\prime$ such that the predicted outcome $M(x^\prime)$ corresponds to some target outcome $y^*$:
+A framework for Counterfactual Explanations was first proposed in 2017 by Wachter et al. @wachter2017counterfactual and has served as the baseline for many methodologies that have been proposed since then. Let $M: \mathcal{X} \mapsto \mathcal{Y}$ denote some pre-trained model that maps from inputs $X \in \mathcal{X}$ to outputs $Y \in \mathcal{Y}$. Then we are interested in minimizing the cost^[Equivalently, others have referred to this quantity as *complexity* or simply *distance*.] $C=\text{cost}(x^\prime)$ incurred by individual $x$ when moving to a counterfactual state $x^\prime$ such that the predicted outcome $M(x^\prime)$ corresponds to some target outcome $y^*$:
 
 \begin{equation}
 \min_{x^\prime \in \mathcal{X}} \text{cost}(x^\prime) \ \ \ \mbox{s. t.} \ \ \ M(x^\prime) = y^* (\#eq:obj)
@@ -33,6 +33,6 @@ In the context of Algorithmic Recourse, domain and model shifts were first broug
 
 ## Benchmarking Counterfactual Generators {#related-benchmark}
 
-Despite the large and growing number of approaches to counterfactual search, there have been surprisingly few benchmark studies that compare different methodologies. This may be partially due to limited software availability in this space. Recent work has started to address this gap: firstly, @deoliveira2021framework run a large benchmarking study using different algorithmic approaches and numerous tabular datasets; secondly, @pawelczyk2021carla introduce a Python framework---CARLA---that can be used to apply and benchmark different methodologies; finally, [`CounterfactualExplanations.jl`](https://github.com/pat-alt/CounterfactualExplanations.jl) [@altmeyer2022counterfactualexplanations] provides an extensible, fast and language-agnostic implementation in Julia. Since the experiments presented here involve extensive simulations, we have relied on and extended the Julia implementation due to the associated performance benefits. In particular, we have built a framework on top of [`CounterfactualExplanations.jl`](https://github.com/pat-alt/CounterfactualExplanations.jl) that extends the functionality from static benchmarks to simulation experiments: [`AlgorithmicRecourseDynamics.jl`]((https://github.com/pat-alt/AlgorithmicRecourseDynamics.jl))^[The code has been released as a package: [https://github.com/pat-alt/AlgorithmicRecourseDynamics.jl](https://github.com/pat-alt/AlgorithmicRecourseDynamics.jl).]. The core concepts implemented in that package reflect what is presented in Section \@ref(method-2) of this paper. 
+Despite the large and growing number of approaches to counterfactual search, there have been surprisingly few benchmark studies that compare different methodologies. This may be partially due to limited software availability in this space. Recent work has started to address this gap: firstly, @deoliveira2021framework run a large benchmarking study using different algorithmic approaches and numerous tabular datasets; secondly, @pawelczyk2021carla introduce a Python framework---CARLA---that can be used to apply and benchmark different methodologies; finally, [`CounterfactualExplanations.jl`](https://github.com/pat-alt/CounterfactualExplanations.jl) [@altmeyer2022counterfactualexplanations] provides an extensible and fast implementation in Julia. Since the experiments presented here involve extensive simulations, we have relied on and extended the Julia implementation due to the associated performance benefits. In particular, we have built a framework on top of [`CounterfactualExplanations.jl`](https://github.com/pat-alt/CounterfactualExplanations.jl) that extends the functionality from static benchmarks to simulation experiments: [`AlgorithmicRecourseDynamics.jl`]((https://github.com/pat-alt/AlgorithmicRecourseDynamics.jl))^[The code has been released as a package: [https://github.com/pat-alt/AlgorithmicRecourseDynamics.jl](https://github.com/pat-alt/AlgorithmicRecourseDynamics.jl).]. The core concepts implemented in that package reflect what is presented in Section \@ref(method-2) of this paper.