Merge branch '73-aries-comments' of https://github.com/pat-alt/ECCCo.jl into 73-aries-comments

paper/.gitignore

+/.quarto/

paper/_quarto.yml

+bibliography: bib.bib
\ No newline at end of file

paper/aaai/author_response.qmd

 ---
 title: 'Author Response'
-format: commonmark
+format: 
+   pdf:
+      number-sections: true
+      number-depth: 1
 bibliography: ../bib.bib
 ---
 
-# Author Response
-
 Following the reviews we received for NeurIPS we have taken substantial measures to address reviewer concerns both during the initial rebuttal period and beyond. 
 
 ## Additional Datasets
@@ -14,13 +15,13 @@ A common concern across reviewers was limited evaluation on real-world datasets.
 
 ### A note on image datasets
 
-Related work on plausibility of counterfactuals has largely relied on small image datasets like *MNIST* [@dhurandhar2018explanations,@schut2021generating]. This may be due to the fact that generating counterfactuals for high-dimensional input data is computationally very challenging. An exception to this rule is the work on *REVISE* [@joshi2019realistic], which uses a larger image dataset. *REVISE* is suitable for this task, because it maps counterfactuals to a lower-dimensional latent space. Similarly, our proposed *ECCCo+* should also be applicable to high-dimensional input data. In our benchmarks, however, we include other generators that search directly in the input space. Since our benchmarks required us to generate a very large number of counterfactuals, it was not at this time feasible to include larger image datasets. That is despite our best efforts to optimize the code and parallelize the computations through multi-threading and multi-processing on a high-performance computing cluster.
+Related work on plausibility of counterfactuals has largely relied on small image datasets like *MNIST* [@dhurandhar2018explanations;@schut2021generating;@delaney2023counterfactual]. This may be due to the fact that generating counterfactuals for high-dimensional input data is computationally very challenging. An exception to this rule is the work on *REVISE* [@joshi2019realistic], which uses a larger image dataset. *REVISE* is suitable for this task, because it maps counterfactuals to a lower-dimensional latent space. Similarly, our proposed *ECCCo+* should also be applicable to high-dimensional input data. In our benchmarks, however, we include other generators that search directly in the input space. Since our benchmarks required us to generate a very large number of counterfactuals, it was not at this time feasible to include larger image datasets. That is despite our best efforts to optimize the code and parallelize the computations through multi-threading and multi-processing on a high-performance computing cluster.
 
 ## Constraining Energy Directly
 
-In our initial work we used our unfaithfulness metric directly as a penalty term in *ECCCo*'s counterfactual search objective. This generally achieves the highest levels of faithfulness but it has several disadvantages, some of which were pointed out by the reviewers. Our new approach constrains the energy directly, which is more theoretically grounded and leads to better results across the board. It also addresses the following reviewer concerns:
+In our initial work we used our unfaithfulness metric directly as a penalty term in *ECCCo*'s counterfactual search objective. This generally achieves the highest levels of faithfulness but it has several disadvantages, some of which were pointed out by the reviewers. Our new approach constrains the energy directly, which is more theoretically grounded and leads to better results across the board. Since it does not depend on generating samples through SGLD, our new approach is much more computationally efficient as well. Additionally, it also addresses the following reviewer concerns:
 
-### Results are biased with respect to unfaithfulness metric
+### Results were biased with respect to unfaithfulness metric
 
 One reviewer raised concern about the fact the using the unfaithfulness metric as a penalty biases the results. This is a valid concern which we have addressed now.
 
@@ -28,35 +29,29 @@ One reviewer raised concern about the fact the using the unfaithfulness metric a
 
 Another reviewer pointed out that the counterfactuals generated by *ECCCo* looked homogeneous, which is also a valid concern. The observed homogeneity most likely stemmed form the fact that the samples generated through SGLD for the underlying models were fairly homogenous. With our new approach we no longer rely on SGLD samples and the homogeneity issue is no longer present. 
 
-### Closeness criterium violated
+### Closeness criterium was violated
 
 A related concern was that large perturbations induced by *ECCCo* seemed to violate the closeness criterium. As we discuss in the paper, our findings do not suggest that *ECCCo* yields unnecessarily costly counterfactuals. Indeed, with reference to the vision data, *ECCCo* seems to keep useful parts of the factual largely in tact, which reduces costs. As we already argued during the rebuttal and in the paper, additional costs cannot be avoided entirely when faithfulness and plausibility are prioritized. This applies to *ECCCo* as much as to other generators like *REVISE*.
 
-### Sampling Overhead
-
-
-
-1. Applied to additional commonly used tabular real-world datasets
-2. Constraining energy directly
-   1. Better results across the board, in particular for image data
-   2. Derived from JEM loss function -> more theoretically grounded
-   3. No sampling overhead.
-   4. Energy does not depend on differentiability.
-   5. Benchmarks no longer biased with respect to unfaithfulness metric (addressing reviewer concern).
-3. Counterfactual explanations do not scale well to high-dimensional input data
-   1. We have added native support for multi-processing and multi-threading.
-   2. We have run more extensive experiments including fine-tuning hyperparameter choices.
-   3. For image data we use PCA to map counterfactuals to a smaller dimenionsional latent space, which not only reduces costs of gradient computations but also leads to higher plausibility.
-   4. PCA is much less costly and interventionist than a VAE: pricipal component merely represent variation in the data; nothing else about the data is learned by the surrogate. 
-      1. ECCCo-$\Delta$ (latent) remains faithful, although not as faithful as standard ECCCo-$\Delta$.
-4. We have revisited the mathematical notation.
-5. We have moved the introduction of conformal prediction forward and added more detail in line with reviewer feedback.
-6. We have extended the limitations section. 
-7. Distance metric
-   1. We have revisited the distance metrics and decided to use the L2 Norm for plausibility and faithfulness
-   2. Orginially, we used the L1 Norm in line with how the the closeness criterium is commonly evaluated. But in this context the L1 Norm implicitly addresses the desire for sparsity.
-   3. In the case of image data, we investigated various additional distance metrics:
-      1. Cosine similarity
-      2. Euclidean distance
-      3. Ultimately we chose to rely on structural dissimilarity.
+### Generalizability
+
+This was not an explicit concern but some reviewers wondered if *ECCCo* could also be applied to non-differentiable models. While our initial approach that relied on SGLD samples was not suitable for non-differentiable models, our new approach is. This is because none of its penalties rely on differentiability. Of course, we still framed *ECCCo* in terms of gradient-based optimization, but the proposed penalties could be applied to other, non-gradient-based counterfactual generators as well such as *FeatureTweak*, for example [@tolomei2017interpretable]. 
+
+## Mathematical notation and concepts
+
+One reviewer complained about the mathematical notation, a concern that was not shared by any of the other reviewers. Nonetheless, we have revisited the notation and hope that it is now more clear. That same reviewer also raised concern about our definitions of plausibility and faithfulness that rely on distributional properties. We have extensively argued our case during the rebuttal and pointed to a potential reviwer misunderstanding in this context. None of the other reviewers found any issue with our definitions and we have made no changes in this regard. We did, however, make a minor change with respect to the related evaluation metrics. We are now more careful about our choice of the distance function. In particular, we investigated various distance metrics for image data and decided to rely on structural dissimilarity. For all other data we use the L2 Norm, where we previously used the L1 Norm. This has no impact on the results, but there was no obvious reason to use the L1 Norm in the first place other than the fact that it is typically used to assess closeness. 
+
+## Conformal prediction was introduced too suddenly
+
+One reviewer pointed out that conformal prediction was introduced too suddenly. We have moved the introduction of conformal prediction forward and added more detail in line with reviewer feedback.
+
+## Limitations section
+
+We have extended the limitations section to address reviewer concerns.
+
+## Other improvements
+
+As discussed above, counterfactual explanations do not scale very well to high-dimensional input data. The NeurIPS feedback has motivated us to work on this issue by enabling intuitive support for multi-threading and multi-processing to our code. This has not only allowed us to include additional datasets but also to run extensive experiments to fine-tune hyperparameter choices. All of our code will be open-sourced as a package and we hope that it will be as useful to the community as was to us during our research.
+
+## References
    
\ No newline at end of file

paper/bib.bib

@@ -3139,6 +3139,15 @@
   year    = {2018},
 }
 
+@Article{delaney2023counterfactual,
+  author    = {Delaney, Eoin and Pakrashi, Arjun and Greene, Derek and Keane, Mark T},
+  title     = {Counterfactual explanations for misclassified images: How human and machine explanations differ},
+  pages     = {103995},
+  journal   = {Artificial Intelligence},
+  publisher = {Elsevier},
+  year      = {2023},
+}
+
 @Comment{jabref-meta: databaseType:biblatex;}
 
 @Comment{jabref-meta: keypatterndefault:[auth:lower][year][veryshorttitle:lower];}