diff --git a/.gitignore b/.gitignore index e4420b30c1c50c3d6cd4fe1943f860a3f05ffbc7..8bb53c52731c961c4c3e6efd796133dc0420be21 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,7 @@ /Manifest.toml /replicated/ */.CondaPkg/ +/dev/rebuttal/www # Tex diff --git a/dev/rebuttal/6zGr.md b/dev/rebuttal/6zGr.md index 57be54c8ac9b60b7109bfda86b811e7298e4002a..ca9e0798543723fa27a79e8f5bf9b449621952b6 100644 --- a/dev/rebuttal/6zGr.md +++ b/dev/rebuttal/6zGr.md @@ -1,12 +1,12 @@ Thank you! In this individual response, we will refer back to the main points discussed in the global response where relevant and discuss any other specific points the reviewer has raised. Below we will go through individual points where quotations trace back to reviewer remarks. -#### Low plausibility (real-world data) +### Low plausibility (real-world data) > "The major weakness of this work is that plausibility for non-JEM-based classifiers is very low on 'real-world' datasets (Table 2)." Please refer to **Point 3** (and to some extent also **Point 2**) of the global rebuttal. -#### Visual quality (MNIST) +### Visual quality (MNIST) > "[...] visual quality of generated counterfactuals seems to be low. [Results] hint to low diversity of generated counterfactuals." @@ -17,7 +17,7 @@ Again, we kindly point to the global rebuttal (**Point 2** and **Point 3**) in t We will discuss this more thoroughly in the paper. -#### Closeness desideratum +### Closeness desideratum > "ECCCos seems to generate counterfactuals that heavily change the initial image [...] thereby violating the closeness desideratum." @@ -28,19 +28,19 @@ We will discuss this more thoroughly in the paper. We will highlight this tradeoff in section 7. -#### Datasets +### Datasets > "The experiments are only conducted on small-scale datasets." In short, we have relied on illustrative datasets commonly used in similar studies. Please refer to our global rebuttal (in particular **Point 1**) for additional details. -#### Conformal Prediction (ablation) +### Conformal Prediction (ablation) > "[...] it is unclear if conformal prediction is actually required for ECCCos." Please refer to **Point 4** in the global rebuttal. -#### Bias towards faithfulness +### Bias towards faithfulness > "Experimental results for faithfulness are biased since (un)faithfulness is already used during counterfactual optimization as regularizer." @@ -49,7 +49,7 @@ Please refer to **Point 4** in the global rebuttal. We will make this point more explicit in section 7. -#### Other questions +### Other questions Finally, let us try to answer the specific questions that were raised: diff --git a/dev/rebuttal/ZaU8.md b/dev/rebuttal/ZaU8.md index 81e2add332e3a0e6f73c7e57355480531f911930..820c9031788d747a52468320ad19f4641c4b89da 100644 --- a/dev/rebuttal/ZaU8.md +++ b/dev/rebuttal/ZaU8.md @@ -1,16 +1,16 @@ Thank you! In this individual response, we will refer back to the main points discussed in the global response where relevant and discuss any other specific points the reviewer has raised. Below we will go through individual points where quotations trace back to reviewer remarks. -#### Summary +### Summary The reviewer has nicely summarised our work and we are happy to see that the main messages of the paper evidently came across. We also appreciate the mentioning of "honest acknowledgment of method limitations" as one of the strengths of the paper, which has indeed been important to us. -#### Citation of Welling \& Teh (2011) +### Citation of Welling \& Teh (2011) > "It may be good to add a citation to [Welling & Teh, 2011] for SGLD on line 144." Regarding the specific question/suggestion raised by the reviewer, we do actually cite Welling \& Teh (2011) in line 145, but we can move that up to line 144 to make it clearer. -#### Need for gradient-access +### Need for gradient-access > "Need for gradient access, e.g. through autodiff, for black-box model under investigation." diff --git a/dev/rebuttal/pekM.md b/dev/rebuttal/pekM.md index c06c9afd19b505b0de222d1bfb2c3f515d85e9fe..c697751443ecaf864b5fe4ab374145909f206f84 100644 --- a/dev/rebuttal/pekM.md +++ b/dev/rebuttal/pekM.md @@ -1,11 +1,11 @@ Thank you! In this individual response, we will refer back to the main points discussed in the global response where relevant and discuss any other specific points the reviewer has raised. Below we will go through individual points where quotations trace back to reviewer remarks. -#### Mathematical notation +### Mathematical notation > "Some notions are lacking descriptions and explanations" We will make a full pass over all notation, and improve where needed. -#### Conditional distribution +### Conditional distribution > "[...] the class-condition distribution $p(\mathbf{x}|\mathbf{y^{+}})$ is existed but unknown and learning this distribution is very challenging especially for structural data" @@ -17,14 +17,14 @@ We do not see this as a weakness of our paper. Instead: We will revisit section 2 to clarify this. -#### Implausibility metric +### Implausibility metric > "Additionally, the implausibility metric seems not general and rigorous [...]" - We agree it is not perfect and speak to this in the paper (e.g. lines 297 to 299). But we think that it is an improved, more robust version of the metric that was previously proposed and used in the literature (lines 159 to 166). Nonetheless, we will make this limitation clearer also in section 7. - The rule-based unary constraint metric proposed in Vo et al. (2023) looks interesting, but the paper will be presented for the first time at KDD in August 2023 and we were not aware of it at the time of writing. Thanks for bringing it to our attention, we will mention it in the same context in section 7. -#### Definiton of "faithfulness" +### Definiton of "faithfulness" > "Faithfulness [...] can be understood as the validity and fidelity of counterfactual examples. [...] The definition 4.1 is fine but missing of the details of $p_{\theta}(\mathbf{x}|\mathbf{y^{+}})$. [...] However, it is not clear to me how to [...] use it in [SGLD]." @@ -36,7 +36,7 @@ We wish to highlight a possible reviewer misunderstanding with regard to a funda We will revisit sections 3 and 4 of the paper to better explain this. -#### Conformal Prediction (CP) +### Conformal Prediction (CP) CP in this context is mentioned both as a strength @@ -54,6 +54,6 @@ We reiterate our motivation here: Following the suggestion from reviewer 6zGr we will smoothen the introduction Conformal Prediction and better motivate it beforehand. -#### Experiments +### Experiments -Please see **Points 1** and **4** of our global rebuttal \ No newline at end of file +Please see **Points 1** and **4** of our global rebuttal. \ No newline at end of file diff --git a/dev/rebuttal/uCjw.md b/dev/rebuttal/uCjw.md index d2f3d7f38125c0adbceede27be8492592c9a2524..f67481f25e3db26fa8d7a189aefbb46a3d8862b9 100644 --- a/dev/rebuttal/uCjw.md +++ b/dev/rebuttal/uCjw.md @@ -1,16 +1,16 @@ Thank you! In this individual response, we will refer back to the main points discussed in the global response where relevant and discuss any other specific points the reviewer has raised. Below we will go through individual points where quotations trace back to reviewer remarks. -#### Q1 and Q3: Data and models +### Q1 and Q3: Data and models Please refer to **Point 1** and **Point 2** in the global response, respectively. -#### Q2: Generalisability +### Q2: Generalisability > "Is the ECCCos approach adaptable to a broad range of black-box models beyond those discussed?" Our approach should generalise to any classifier that is differentiable with respect to inputs, consistent with other gradient-based counterfactual generators (Equation 1). Our actual implementation is currently compatible with neural networks trained in Julia and has experimental support for `torch` trained in either Python or R. Even though it is possible to generate counterfactuals for non-differentiable models, it is not immediately obvious to us how SGLD can be applied in this context. An interesting question for future research would be if other scalable and gradient-free methods can be used to sample from the conditional distribution learned by the model. -#### Q4: Link to causality +### Q4: Link to causality > "There’s a broad literature on causal abstractions and causal model explanations that seems related."