search.json

[
  {
    "objectID": "index.html",
    "href": "index.html",
    "title": "Endogenous Macrodynamics in Algorithmic Recourse",
    "section": "",
    "text": "Preface\nThis is an online auxiliary companion to our research paper: Endogenous Macrodynamics in Algorithmic Recourse."
  },
  {
    "objectID": "intro.html#data",
    "href": "intro.html#data",
    "title": "1  Introduction",
    "section": "1.1 Data",
    "text": "1.1 Data\nWe begin by generating the synthetic data for a simple binary classification problem. For illustrative purposes we will use data that is linearly separable. The chart below shows the data \\(\\mathcal{D}\\) at time zero, before any implementation of recourse.\n\nN = 1000\nxmax = 2\nX, ys = make_blobs(\n    N, 2; \n    centers=2, as_table=false, center_box=(-xmax => xmax), cluster_std=0.1\n)\nys .= ys.==2\nX = X'\nxs = Flux.unstack(X,2)\ndata = zip(xs,ys)\ncounterfactual_data = CounterfactualData(X,ys')\nplot()\nscatter!(counterfactual_data)\n\n\n\n\nFigure 1.1: Linearly separable synthetic data"
  },
  {
    "objectID": "intro.html#classifier",
    "href": "intro.html#classifier",
    "title": "1  Introduction",
    "section": "1.2 Classifier",
    "text": "1.2 Classifier\nTo model this data \\(\\mathcal{D}\\) we will use a linear classifier. In particular, as in the paper, we will build a logistic regression model in Flux.jl: a single layer with sigmoid activation.\n\nn_epochs = 100\nmodel = Chain(Dense(2,1))\nmod = FluxModel(model)\nModels.train(mod, counterfactual_data; n_epochs=n_epochs)\n\nFigure 1.2 below shows the linear separation of the two classes.\n\nplt_original = plot(mod, counterfactual_data; zoom=0, colorbar=false, title=\"(a)\")\ndisplay(plt_original)\n\n\n\n\nFigure 1.2: The baseline model: contours indicate the predicted label; dots indicate observed data points."
  },
  {
    "objectID": "intro.html#implementation-of-recourse",
    "href": "intro.html#implementation-of-recourse",
    "title": "1  Introduction",
    "section": "1.3 Implementation of Recourse",
    "text": "1.3 Implementation of Recourse"
  },
  {
    "objectID": "intro.html#generate-counterfactual",
    "href": "intro.html#generate-counterfactual",
    "title": "1  Introduction",
    "section": "1.4 Generate counterfactual",
    "text": "1.4 Generate counterfactual\n\nγ = 0.50\nμ = 0.10\nMarkdown.parse(\n    \"\"\"\n    To generate counterfactual explanations we will rely on the most generic approach. As our decision threshold we will use $(γ*100)% here. In other words, the counterfactual is considered as valid, as soon as the classifier is more convinced that it belongs to the target class (blue) than the non-target class (orange). In each round we will implement recourse for $(μ * 100)% of the individuals in the non-target class. \n    \"\"\"\n)\n\n\nopt = Flux.Adam(0.01)\ngen = GenericGenerator(;decision_threshold=γ, opt=opt)\n\nFigure 1.3 below shows the recourse outcome, which we denote here as \\(\\mathcal{D}^{\\prime}\\). The obvious observation at this point is that the resulting counterfactuals, while valid, are clearly distinguishable from the factuals that were always in the target class. This is not a new observation and nor is it entirely surprising. In fact, a lot of recent work in this field has tried to address this issue. In this work we wonder what happens when we let these sorts of dynamics play out further in practice. While the outcome in (b) is not surprising, it may be much harder to observe so clearly it in practice (when the data is more complex).\n\ncandidates = findall(ys.==0)\nchosen_individuals = rand(candidates, Int(round(μ*length(candidates))))\nX′ = copy(X)\ny′ = copy(ys)\nfactuals = select_factual(counterfactual_data,chosen_individuals)\noutcome = generate_counterfactual(factuals, 1, counterfactual_data, mod, gen; initialization=:identity)\nX′[:,chosen_individuals] = reduce(hcat, @.(selectdim(counterfactual(outcome), 3, 1)))\ny′[chosen_individuals] = reduce(vcat,@.(selectdim(counterfactual_label(outcome),3,1)))\ncounterfactual_data′ = CounterfactualData(X′,y′')\nplt_single = plot(mod,counterfactual_data′;zoom=0,colorbar=false,title=\"(b)\")\ndisplay(plt_single)\n\n\n\n\nFigure 1.3: The recourse outcome after one round."
  },
  {
    "objectID": "intro.html#retrain",
    "href": "intro.html#retrain",
    "title": "1  Introduction",
    "section": "1.5 Retrain",
    "text": "1.5 Retrain\nSuppose the agent in charge of the black-box system has provided recourse to a share of individuals leading to the outcome in Figure 1.3. In practice, models are regularly updated through retraining to account for concept drift, for example. For our experiments, we assume that the agent accepts \\(\\mathcal{D}^{\\prime}\\) as its new ground truth. To isolate the endogenous effects we are interested in here from any other effect, we further assume away any exogenous changes to the data that we might expect to occur in practice. Retraining the model on \\(\\mathcal{D}^{\\prime}\\) leads to a shift of the decision boundary in the direction of the non-target class (Figure 1.4).\n\nmod = Models.train(mod, counterfactual_data′)\nplt_single_retrained = plot(mod,counterfactual_data′;zoom=0,colorbar=false,title=\"(c)\")\ndisplay(plt_single_retrained)\n\n\n\n\nFigure 1.4: The retrained model."
  },
  {
    "objectID": "intro.html#repeat",
    "href": "intro.html#repeat",
    "title": "1  Introduction",
    "section": "1.6 Repeat",
    "text": "1.6 Repeat\nWe finally go on to repeat this process of recourse followed by model updates for multiple round. Figure 1.5 below presents the different stages of the experiment side-by-side, where panel (d) represents the outcome after ten rounds.\nAt first glance it seems that costs to individuals seeking recourse are gradually reduced as the decision boundary moves into the direction of the non-target class: they need to exert less effort to move to valid counterfactual states. The problem with this idea is, of course, that there is no free lunch. This reduction inflicts a burden on the agent in charge of the black-box: the group of individuals that is now classified as target class individuals looks entirely different from the original group.\nWhy is this a problem? Let’s, for example, that the two synthetic features accurately describe the credit worthiness of individual seeking loans, where credit-worthiness increases in the South-West direction. Non-target class individuals (orange) are denied credit, while target class individuals (blue) receive a loan. Then the population of borrowers in (d) is much more risky than in (a). Clearly, any lender (bank) aware of such dynamics would avoid them in practice. They might choose not to offer recourse in the first place, generating a cost to all individuals seeking recourse. Alternatively, they may reward first movers, but stop offering recourse after a few rounds.\nThis last point makes it clear that the implementation of recourse by one individual may generate external costs for other individuals. This notion motivates the ideas set out in the paper.\n\ni = 2\nwhile i <= 10\n    counterfactual_data′ = CounterfactualData(X′,y′')\n    candidates = findall(y′.==0)\n    chosen_individuals = rand(candidates, Int(round(μ*length(candidates))))\n    Models.train(mod, counterfactual_data′)\n    factuals = select_factual(counterfactual_data′,chosen_individuals)\n    outcome = generate_counterfactual(factuals, 1, counterfactual_data′, mod, gen; initialization=:identity)\n    X′[:,chosen_individuals] = reduce(hcat, @.(selectdim(counterfactual(outcome), 3, 1)))\n    y′[chosen_individuals] = reduce(vcat,@.(selectdim(counterfactual_label(outcome),3,1)))\n    i += 1\nend\nplt_single_repeat = plot(mod,counterfactual_data′;zoom=0,colorbar=false,title=\"(d)\")\n\n\nplt = plot(plt_original, plt_single, plt_single_retrained, plt_single_repeat, layout=(1,4), legend=false, axis=nothing, size=(600,165))\nsavefig(plt, joinpath(www_path, \"poc.png\"))\ndisplay(plt)\n\n\n\n\nFigure 1.5: The different stages of the experiment."
  },
  {
    "objectID": "sections/data_preprocessing/index.html#generating-synthetic-data",
    "href": "sections/data_preprocessing/index.html#generating-synthetic-data",
    "title": "2  Data Preprocessing",
    "section": "2.1 Generating Synthetic Data",
    "text": "2.1 Generating Synthetic Data\n\nn = 1000\np = 2\n\n# Linearly separable:\nX, y = make_blobs(n, p; centers=2, center_box=(-2 => 2), cluster_std=0.1)\ndf = DataFrame(X)\ndf.target .= ifelse.(y.==1,0,1)\nCSV.write(joinpath(data_path, \"linearly_separable.csv\"),df)\n\n# Overlapping:\nX, y = make_blobs(n, p; centers=2, center_box=(-2 => 2), cluster_std=0.5)\ndf = DataFrame(X)\ndf.target .= ifelse.(y.==1,0,1)\nCSV.write(joinpath(data_path, \"overlapping.csv\"),df)\n\n# Circles:\nX, y = make_circles(n; noise=0.15, factor=0.01)\ndf = DataFrame(X)\ndf.target = y\nCSV.write(joinpath(data_path, \"circles.csv\"),df)\n\n# Moon:\nX, y = make_moons(n)\ndf = DataFrame(X)\ndf.target = y\nCSV.write(joinpath(data_path, \"moons.csv\"),df)\n\n\ngenerate_artifacts(data_path)\n\n\ncatalogue = load_synthetic()\nfunction plot_data(data,title)\n    plt = plot(title=uppercasefirst(replace(string(title),\"_\" => \" \")))\n    scatter!(data)\n    return plt\nend\nplts = [plot_data(data,name) for (name, data) in catalogue]\nplt = plot(plts..., layout=(1,4), size=(850,200))\nsavefig(plt, \"paper/www/synthetic_data.png\")\ndisplay(plt)"
  },
  {
    "objectID": "sections/data_preprocessing/index.html#preprocessing-real-world-data",
    "href": "sections/data_preprocessing/index.html#preprocessing-real-world-data",
    "title": "2  Data Preprocessing",
    "section": "2.2 Preprocessing Real-World Data",
    "text": "2.2 Preprocessing Real-World Data\n\n2.2.1 California Housing Data\nFetching the data using Python’s sklearn (run this in the Python REPL):\nfrom sklearn.datasets import fetch_california_housing\ndf, y = fetch_california_housing(return_X_y=True, as_frame=True)\ndf[\"target\"] = y.values\ndata_path = \"dev/artifacts/upload/data/real_world\"\nimport os\nif not os.path.isdir(os.path.join(data_path,\"raw\")):\n    os.makedirs(os.path.join(data_path,\"raw\"))\ndf.to_csv(os.path.join(data_path,\"raw/cal_housing.csv\"), index=False)\nLoading the data into Julia session.\n\ndf = CSV.read(joinpath(data_path, \"raw/cal_housing.csv\"), DataFrame)\n# Target:\ny = df.target\ny = Float64.(y .>= median(y)); # binary target (positive outcome)\n# Data:\ndf.target = y\n\nRandom undersampling to balance the data:\n\ndf_balanced = getobs(undersample(df, df.target; shuffle=true))[1]\n\nAll features are continuous:\n\nschema(df_balanced)\n\n\n┌────────────┬────────────┬─────────┐\n│ names      │ scitypes   │ types   │\n├────────────┼────────────┼─────────┤\n│ MedInc     │ Continuous │ Float64 │\n│ HouseAge   │ Continuous │ Float64 │\n│ AveRooms   │ Continuous │ Float64 │\n│ AveBedrms  │ Continuous │ Float64 │\n│ Population │ Continuous │ Float64 │\n│ AveOccup   │ Continuous │ Float64 │\n│ Latitude   │ Continuous │ Float64 │\n│ Longitude  │ Continuous │ Float64 │\n│ target     │ Continuous │ Float64 │\n└────────────┴────────────┴─────────┘\n\n\n\n\nFeature transformation:\n\ntransformer = Standardizer(count=true)\nmach = MLJBase.fit!(machine(transformer, df_balanced[:,Not(:target)]))\nX = MLJBase.transform(mach, df_balanced[:,Not(:target)])\nschema(X)\n\n\n┌────────────┬────────────┬─────────┐\n│ names      │ scitypes   │ types   │\n├────────────┼────────────┼─────────┤\n│ MedInc     │ Continuous │ Float64 │\n│ HouseAge   │ Continuous │ Float64 │\n│ AveRooms   │ Continuous │ Float64 │\n│ AveBedrms  │ Continuous │ Float64 │\n│ Population │ Continuous │ Float64 │\n│ AveOccup   │ Continuous │ Float64 │\n│ Latitude   │ Continuous │ Float64 │\n│ Longitude  │ Continuous │ Float64 │\n└────────────┴────────────┴─────────┘\n\n\n\n\nTurning the data into CounterfactualData:\n\nX = Matrix(X)\nX = permutedims(X)\ny = permutedims(df_balanced.target)\ndata = CounterfactualData(X,y)\n\nSaving the data:\n\nCSV.write(joinpath(data_path, \"cal_housing.csv\"), df_balanced)              # binary file\nSerialization.serialize(joinpath(data_path,\"cal_housing.jls\"), data)        # CounterfactualData\n\n\n\n2.2.2 Give Me Some Credit\nLoading and basic preprocessing:\n\ndf = CSV.read(joinpath(data_path, \"raw/cs-training.csv\"), DataFrame)\nselect!(df, Not([:Column1]))\nrename!(df, :SeriousDlqin2yrs => :target)\nmapcols!(x -> [ifelse(x_==\"NA\", missing, x_) for x_ in x], df)\ndropmissing!(df)\nmapcols!(x -> eltype(x) <: AbstractString ? parse.(Int, x) : x, df)\ndf.target .= map(y -> y == 0 ? 1 : 0, df.target)        # postive outcome = no delinquency\n\nBalancing:\n\ndf_balanced = getobs(undersample(df, df.target;shuffle=true))[1]\n\nAll features are continuous:\n\nschema(df_balanced)\n\nFeature transformation:\n\ntransformer = Standardizer(count=true)\nmach = MLJBase.fit!(machine(transformer, df_balanced[:,Not(:target)]))\nX = MLJBase.transform(mach, df_balanced[:,Not(:target)])\nschema(X)\n\nTurning the data into CounterfactualData:\n\nX = Matrix(X)\nX = permutedims(X)\ny = permutedims(df_balanced.target)\ndata = CounterfactualData(X,y)\n\nSaving:\n\nCSV.write(joinpath(data_path, \"gmsc.csv\"), df_balanced)         # binary file\nSerialization.serialize(joinpath(data_path,\"gmsc.jls\"), data)   # CounterfactualData\n\n\n\n2.2.3 UCI Credit Card Default\nLoading and basic preprocessing:\n\ndf = CSV.read(joinpath(data_path, \"raw/UCI_Credit_Card.csv\"), DataFrame)\nselect!(df, Not([:ID]))\nrename!(df, \"default.payment.next.month\" => :target)\ndropmissing!(df)\ndf.SEX = categorical(df.SEX)\ndf.EDUCATION = categorical(df.EDUCATION)\ndf.MARRIAGE = categorical(df.MARRIAGE)\nmapcols!(x -> eltype(x) <: AbstractString ? parse.(Int, x) : x, df)\ndf.target .= map(y -> y == 0 ? 1 : 0, df.target)        # postive outcome = no default\n\nBalancing:\n\ndf_balanced = getobs(undersample(df, df.target;shuffle=true))[1]\n\nNot all features are continuous:\n\nschema(df_balanced)\n\n\n┌───────────┬───────────────┬─────────────────────────────────┐\n│ names     │ scitypes      │ types                           │\n├───────────┼───────────────┼─────────────────────────────────┤\n│ LIMIT_BAL │ Continuous    │ Float64                         │\n│ SEX       │ Multiclass{2} │ CategoricalValue{Int64, UInt32} │\n│ EDUCATION │ Multiclass{7} │ CategoricalValue{Int64, UInt32} │\n│ MARRIAGE  │ Multiclass{4} │ CategoricalValue{Int64, UInt32} │\n│ AGE       │ Count         │ Int64                           │\n│ PAY_0     │ Count         │ Int64                           │\n│ PAY_2     │ Count         │ Int64                           │\n│ PAY_3     │ Count         │ Int64                           │\n│ PAY_4     │ Count         │ Int64                           │\n│ PAY_5     │ Count         │ Int64                           │\n│ PAY_6     │ Count         │ Int64                           │\n│ BILL_AMT1 │ Continuous    │ Float64                         │\n│ BILL_AMT2 │ Continuous    │ Float64                         │\n│ BILL_AMT3 │ Continuous    │ Float64                         │\n│ BILL_AMT4 │ Continuous    │ Float64                         │\n│ BILL_AMT5 │ Continuous    │ Float64                         │\n│ BILL_AMT6 │ Continuous    │ Float64                         │\n│ PAY_AMT1  │ Continuous    │ Float64                         │\n│ PAY_AMT2  │ Continuous    │ Float64                         │\n│ PAY_AMT3  │ Continuous    │ Float64                         │\n│ PAY_AMT4  │ Continuous    │ Float64                         │\n│ PAY_AMT5  │ Continuous    │ Float64                         │\n│ PAY_AMT6  │ Continuous    │ Float64                         │\n│ target    │ Count         │ Int64                           │\n└───────────┴───────────────┴─────────────────────────────────┘\n\n\n\n\nFeature transformation:\n\ntransformer = Standardizer(count=true) |> ContinuousEncoder()\nmach = MLJBase.fit!(machine(transformer, df_balanced[:,Not(:target)]))\nX = MLJBase.transform(mach, df_balanced[:,Not(:target)])\nschema(X)\n\n\n┌──────────────┬────────────┬─────────┐\n│ names        │ scitypes   │ types   │\n├──────────────┼────────────┼─────────┤\n│ LIMIT_BAL    │ Continuous │ Float64 │\n│ SEX__1       │ Continuous │ Float64 │\n│ SEX__2       │ Continuous │ Float64 │\n│ EDUCATION__0 │ Continuous │ Float64 │\n│ EDUCATION__1 │ Continuous │ Float64 │\n│ EDUCATION__2 │ Continuous │ Float64 │\n│ EDUCATION__3 │ Continuous │ Float64 │\n│ EDUCATION__4 │ Continuous │ Float64 │\n│ EDUCATION__5 │ Continuous │ Float64 │\n│ EDUCATION__6 │ Continuous │ Float64 │\n│ MARRIAGE__0  │ Continuous │ Float64 │\n│ MARRIAGE__1  │ Continuous │ Float64 │\n│ MARRIAGE__2  │ Continuous │ Float64 │\n│ MARRIAGE__3  │ Continuous │ Float64 │\n│ AGE          │ Continuous │ Float64 │\n│ PAY_0        │ Continuous │ Float64 │\n│ PAY_2        │ Continuous │ Float64 │\n│ PAY_3        │ Continuous │ Float64 │\n│ PAY_4        │ Continuous │ Float64 │\n│ PAY_5        │ Continuous │ Float64 │\n│ PAY_6        │ Continuous │ Float64 │\n│ BILL_AMT1    │ Continuous │ Float64 │\n│      ⋮       │     ⋮      │    ⋮    │\n└──────────────┴────────────┴─────────┘\n                        11 rows omitted\n\n\n\n\nCategorical indices:\n\nfeatures_categorical = [\n    [2,3],\n    collect(4:10),\n    collect(11:14)\n]\n\nPreparing for use with CounterfactualExplanations.jl:\n\nX = Matrix(X)\nX = permutedims(X)\ny = permutedims(df_balanced.target)\ndata = CounterfactualData(\n    X, y;\n    features_categorical = features_categorical\n)\n\nSaving:\n\nCSV.write(joinpath(data_path, \"credit_default.csv\"), df_balanced)         # binary file\nSerialization.serialize(joinpath(data_path,\"credit_default.jls\"), data)   # CounterfactualData\n\n\ngenerate_artifacts(data_path)"
  },
  {
    "objectID": "sections/experiments/index.html",
    "href": "sections/experiments/index.html",
    "title": "3  Experimental Results",
    "section": "",
    "text": "4 Synthetic Data\nThis notebook was used to run the experiments for the synthetic datasets and can be used to reproduce the results in the paper. In the following we first run the experiments and then generate visualizations and tables."
  },
  {
    "objectID": "sections/experiments/index.html#experiment",
    "href": "sections/experiments/index.html#experiment",
    "title": "3  Experimental Results",
    "section": "4.1 Experiment",
    "text": "4.1 Experiment\n\nmodels = [\n    :LogisticRegression, \n    :FluxModel, \n    :FluxEnsemble,\n]\nopt = Flux.Descent(0.01) \ngenerators = Dict(\n    :Greedy=>GreedyGenerator(), \n    :Generic=>GenericGenerator(opt = opt),\n    :REVISE=>REVISEGenerator(opt = opt),\n    :DICE=>DiCEGenerator(opt = opt),\n)\n\n\nmax_obs = 1000\ncatalogue = load_synthetic(max_obs)\nchoices = [\n    :linearly_separable, \n    :overlapping, \n    :circles, \n    :moons,\n]\ndata_sets = filter(p -> p[1] in choices, catalogue)\n\n\nexperiments = set_up_experiments(data_sets,models,generators)\n\n\nplts = []\nfor (exp_name, exp_) in experiments\n    for (M_name, M) in exp_.models\n        score = round(model_evaluation(M, exp_.test_data),digits=2)\n        plt = plot(M, exp_.test_data, title=\"$exp_name;\\n $M_name ($score)\")\n        # Errors:\n        ids = findall(vec(round.(probs(M, exp_.test_data.X)) .!= exp_.test_data.y))\n        x_wrongly_labelled = exp_.test_data.X[:,ids]\n        scatter!(plt, x_wrongly_labelled[1,:], x_wrongly_labelled[2,:], ms=7.5, color=:red, label=\"\")\n        plts = vcat(plts..., plt)\n    end\nend\nplt = plot(plts..., layout=(length(choices),length(models)),size=(length(choices)*300,length(models)*300))\nsavefig(plt, joinpath(www_path,\"models_test_before.png\"))\n\n\nusing AlgorithmicRecourseDynamics.Models: model_evaluation\nplts = []\nfor (exp_name, exp_) in experiments\n    for (M_name, M) in exp_.models\n        score = round(model_evaluation(M, exp_.train_data),digits=2)\n        plt = plot(M, exp_.train_data, title=\"$exp_name;\\n $M_name ($score)\")\n        # Errors:\n        ids = findall(vec(round.(probs(M, exp_.train_data.X)) .!= exp_.train_data.y))\n        x_wrongly_labelled = exp_.train_data.X[:,ids]\n        scatter!(plt, x_wrongly_labelled[1,:], x_wrongly_labelled[2,:], ms=7.5, color=:red, label=\"\")\n        plts = vcat(plts..., plt)\n    end\nend\nplt = plot(plts..., layout=(length(choices),length(models)),size=(length(choices)*300,length(models)*300))\nsavefig(plt, joinpath(www_path,\"models_train_before.png\"))\n\n\nn_evals = 5\nn_rounds = 50\nevaluate_every = Int(round(n_rounds/n_evals))\nn_folds = 5\nT = 100\nresults = run_experiments(\n    experiments;\n    save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, T=T\n)\nSerialization.serialize(joinpath(output_path,\"results.jls\"),results)\n\n\nplot_dict = Dict(key => Dict() for (key,val) in results)\nfold = 1\nfor (name, res) in results\n    exp_ = res.experiment\n    plot_dict[name] = Dict(key => [] for (key,val) in exp_.generators)\n    rec_sys = exp_.recourse_systems[fold]\n    sys_ids = collect(exp_.system_identifiers)\n    M = length(rec_sys)\n    for m in 1:M\n        model_name, generator_name = sys_ids[m]\n        M = rec_sys[m].model\n        score = round(model_evaluation(M, exp_.test_data),digits=2)\n        plt = plot(M, exp_.test_data, title=\"$name;\\n $model_name ($score)\")\n        # Errors:\n        ids = findall(vec(round.(probs(M, exp_.test_data.X)) .!= exp_.test_data.y))\n        x_wrongly_labelled = exp_.test_data.X[:,ids]\n        scatter!(plt, x_wrongly_labelled[1,:], x_wrongly_labelled[2,:], ms=7.5, color=:red, label=\"\")\n        plot_dict[name][generator_name] = vcat(plot_dict[name][generator_name], plt)\n    end\nend\nplot_dict = Dict(key => reduce(vcat, [plots[key] for plots in values(plot_dict)]) for (key, value) in generators)\nfor (name, plts) in plot_dict\n    plt = plot(plts..., layout=(length(choices),length(models)),size=(length(choices)*300,length(models)*300))\n    savefig(plt, joinpath(www_path,\"models_test_after_$(name).png\"))\nend\n\n\nusing AlgorithmicRecourseDynamics.Models: model_evaluation\nplot_dict = Dict(key => Dict() for (key,val) in results)\nfold = 1\nfor (name, res) in results\n    exp_ = res.experiment\n    plot_dict[name] = Dict(key => [] for (key,val) in exp_.generators)\n    rec_sys = exp_.recourse_systems[fold]\n    sys_ids = collect(exp_.system_identifiers)\n    M = length(rec_sys)\n    for m in 1:M\n        model_name, generator_name = sys_ids[m]\n        M = rec_sys[m].model\n        data = rec_sys[m].data\n        score = round(model_evaluation(M, data),digits=2)\n        plt = plot(M, data, title=\"$name;\\n $model_name ($score)\")\n        # Errors:\n        ids = findall(vec(round.(probs(M, data.X)) .!= data.y))\n        x_wrongly_labelled = data.X[:,ids]\n        scatter!(plt, x_wrongly_labelled[1,:], x_wrongly_labelled[2,:], ms=7.5, color=:red, label=\"\")\n        plot_dict[name][generator_name] = vcat(plot_dict[name][generator_name], plt)\n    end\nend\nplot_dict = Dict(key => reduce(vcat, [plots[key] for plots in values(plot_dict)]) for (key, value) in generators)\nfor (name, plts) in plot_dict\n    plt = plot(plts..., layout=(length(choices),length(models)),size=(length(choices)*300,length(models)*300))\n    savefig(plt, joinpath(www_path,\"models_train_after_$(name).png\"))\nend"
  },
  {
    "objectID": "sections/experiments/index.html#plots",
    "href": "sections/experiments/index.html#plots",
    "title": "3  Experimental Results",
    "section": "4.2 Plots",
    "text": "4.2 Plots\n\nresults = Serialization.deserialize(joinpath(output_path,\"results.jls\"));\n\n\nusing Images\nline_charts = Dict()\nerrorbar_charts = Dict()\nfor (data_name, res) in results\n    plt = plot(res)\n    Images.save(joinpath(www_path, \"line_chart_$(data_name).png\"), plt)\n    line_charts[data_name] = plt\n    plt = plot(res,maximum(res.output.n))\n    Images.save(joinpath(www_path, \"errorbar_chart_$(data_name).png\"), plt)\n    errorbar_charts[data_name] = plt\nend\n\n\n4.2.1 Line Charts\nFigure 4.1 shows the evolution of the evaluation metrics over the course of the experiment.\n\n\n\n\n\n\n(a) Circles\n\n\n\n\n\n\n\n(b) Linearly Separable\n\n\n\n\n\n\n\n(c) Moons\n\n\n\n\n\n\n\n(d) Overlapping\n\n\n\nFigure 4.1: Line Charts\n\n\n\n\n4.2.2 Error Bar Charts\nFigure 4.2 shows the evaluation metrics at the end of the experiments.\n\n\n\n\n\n\n(a) Circles\n\n\n\n\n\n\n\n(b) Linearly Separable\n\n\n\n\n\n\n\n(c) Moons\n\n\n\n\n\n\n\n(d) Overlapping\n\n\n\nFigure 4.2: Error Bar Charts"
  },
  {
    "objectID": "sections/experiments/index.html#bootstrap",
    "href": "sections/experiments/index.html#bootstrap",
    "title": "3  Experimental Results",
    "section": "4.3 Bootstrap",
    "text": "4.3 Bootstrap\n\nn_bootstrap = 100\ndf = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,\"bootstrap.csv\"))\n\n\n\n┌──────────┬─────────┬────────────────────┬────────────────────┬───────────┬──────────────┐\n│     name │   scope │               data │              model │ generator │ p_value_mean │\n│ String31 │ String7 │           String31 │           String31 │   String7 │      Float64 │\n├──────────┼─────────┼────────────────────┼────────────────────┼───────────┼──────────────┤\n│      mmd │  domain │        overlapping │ LogisticRegression │   Generic │          0.0 │\n│      mmd │  domain │        overlapping │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │  domain │        overlapping │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │  domain │        overlapping │ LogisticRegression │      DICE │          0.0 │\n│      mmd │  domain │        overlapping │          FluxModel │   Generic │          0.0 │\n│      mmd │  domain │        overlapping │          FluxModel │    Greedy │          0.0 │\n│      mmd │  domain │        overlapping │          FluxModel │    REVISE │          0.0 │\n│      mmd │  domain │        overlapping │          FluxModel │      DICE │          0.0 │\n│      mmd │  domain │        overlapping │       FluxEnsemble │   Generic │          0.0 │\n│      mmd │  domain │        overlapping │       FluxEnsemble │    Greedy │          0.0 │\n│      mmd │  domain │        overlapping │       FluxEnsemble │    REVISE │          0.0 │\n│      mmd │  domain │        overlapping │       FluxEnsemble │      DICE │          0.0 │\n│      mmd │  domain │ linearly_separable │ LogisticRegression │   Generic │          0.0 │\n│      mmd │  domain │ linearly_separable │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │  domain │ linearly_separable │ LogisticRegression │    REVISE │        0.238 │\n│      mmd │  domain │ linearly_separable │ LogisticRegression │      DICE │          0.0 │\n│      mmd │  domain │ linearly_separable │          FluxModel │   Generic │          0.0 │\n│      mmd │  domain │ linearly_separable │          FluxModel │    Greedy │          0.0 │\n│      mmd │  domain │ linearly_separable │          FluxModel │    REVISE │        0.188 │\n│      mmd │  domain │ linearly_separable │          FluxModel │      DICE │          0.0 │\n│      mmd │  domain │ linearly_separable │       FluxEnsemble │   Generic │          0.0 │\n│      mmd │  domain │ linearly_separable │       FluxEnsemble │    Greedy │          0.0 │\n│      mmd │  domain │ linearly_separable │       FluxEnsemble │    REVISE │        0.158 │\n│      mmd │  domain │ linearly_separable │       FluxEnsemble │      DICE │          0.0 │\n│      mmd │  domain │            circles │ LogisticRegression │   Generic │        0.104 │\n│      mmd │  domain │            circles │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │  domain │            circles │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │  domain │            circles │ LogisticRegression │      DICE │       0.1275 │\n│      mmd │  domain │            circles │          FluxModel │   Generic │          0.0 │\n│      mmd │  domain │            circles │          FluxModel │    Greedy │          0.0 │\n│      mmd │  domain │            circles │          FluxModel │    REVISE │        0.996 │\n│      mmd │  domain │            circles │          FluxModel │      DICE │          0.0 │\n│      mmd │  domain │            circles │       FluxEnsemble │   Generic │          0.0 │\n│      mmd │  domain │            circles │       FluxEnsemble │    Greedy │          0.0 │\n│      mmd │  domain │            circles │       FluxEnsemble │    REVISE │          1.0 │\n│      mmd │  domain │            circles │       FluxEnsemble │      DICE │          0.0 │\n│      mmd │  domain │              moons │ LogisticRegression │   Generic │          0.0 │\n│      mmd │  domain │              moons │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │  domain │              moons │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │  domain │              moons │ LogisticRegression │      DICE │          0.0 │\n│      mmd │  domain │              moons │          FluxModel │   Generic │          0.0 │\n│      mmd │  domain │              moons │          FluxModel │    Greedy │          0.0 │\n│      mmd │  domain │              moons │          FluxModel │    REVISE │          0.0 │\n│      mmd │  domain │              moons │          FluxModel │      DICE │          0.0 │\n│      mmd │  domain │              moons │       FluxEnsemble │   Generic │          0.0 │\n│      mmd │  domain │              moons │       FluxEnsemble │    Greedy │          0.0 │\n│      mmd │  domain │              moons │       FluxEnsemble │    REVISE │          0.0 │\n│      mmd │  domain │              moons │       FluxEnsemble │      DICE │          0.0 │\n│      mmd │   model │        overlapping │ LogisticRegression │   Generic │          0.0 │\n│      mmd │   model │        overlapping │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │   model │        overlapping │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │   model │        overlapping │ LogisticRegression │      DICE │          0.0 │\n│      mmd │   model │        overlapping │          FluxModel │   Generic │          0.0 │\n│      mmd │   model │        overlapping │          FluxModel │    Greedy │          0.0 │\n│      mmd │   model │        overlapping │          FluxModel │    REVISE │          0.0 │\n│      mmd │   model │        overlapping │          FluxModel │      DICE │          0.0 │\n│      mmd │   model │        overlapping │       FluxEnsemble │   Generic │          0.0 │\n│      mmd │   model │        overlapping │       FluxEnsemble │    Greedy │          0.0 │\n│      mmd │   model │        overlapping │       FluxEnsemble │    REVISE │          0.0 │\n│      mmd │   model │        overlapping │       FluxEnsemble │      DICE │          0.0 │\n│      mmd │   model │ linearly_separable │ LogisticRegression │   Generic │          0.0 │\n│      mmd │   model │ linearly_separable │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │   model │ linearly_separable │ LogisticRegression │    REVISE │        0.084 │\n│      mmd │   model │ linearly_separable │ LogisticRegression │      DICE │          0.0 │\n│      mmd │   model │ linearly_separable │          FluxModel │   Generic │         0.87 │\n│      mmd │   model │ linearly_separable │          FluxModel │    Greedy │        0.616 │\n│      mmd │   model │ linearly_separable │          FluxModel │    REVISE │         0.65 │\n│      mmd │   model │ linearly_separable │          FluxModel │      DICE │         0.82 │\n│      mmd │   model │ linearly_separable │       FluxEnsemble │   Generic │        0.892 │\n│      mmd │   model │ linearly_separable │       FluxEnsemble │    Greedy │        0.528 │\n│      mmd │   model │ linearly_separable │       FluxEnsemble │    REVISE │        0.642 │\n│      mmd │   model │ linearly_separable │       FluxEnsemble │      DICE │        0.878 │\n│      mmd │   model │            circles │ LogisticRegression │   Generic │          0.0 │\n│      mmd │   model │            circles │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │   model │            circles │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │   model │            circles │ LogisticRegression │      DICE │          0.0 │\n│      mmd │   model │            circles │          FluxModel │   Generic │        0.392 │\n│      mmd │   model │            circles │          FluxModel │    Greedy │        0.078 │\n│      mmd │   model │            circles │          FluxModel │    REVISE │        0.836 │\n│      mmd │   model │            circles │          FluxModel │      DICE │        0.516 │\n│      mmd │   model │            circles │       FluxEnsemble │   Generic │        0.456 │\n│      mmd │   model │            circles │       FluxEnsemble │    Greedy │         0.16 │\n│      mmd │   model │            circles │       FluxEnsemble │    REVISE │        0.872 │\n│      mmd │   model │            circles │       FluxEnsemble │      DICE │        0.572 │\n│      mmd │   model │              moons │ LogisticRegression │   Generic │          0.0 │\n│      mmd │   model │              moons │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │   model │              moons │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │   model │              moons │ LogisticRegression │      DICE │          0.0 │\n│      mmd │   model │              moons │          FluxModel │   Generic │        0.716 │\n│      mmd │   model │              moons │          FluxModel │    Greedy │         0.17 │\n│      mmd │   model │              moons │          FluxModel │    REVISE │         0.67 │\n│      mmd │   model │              moons │          FluxModel │      DICE │        0.588 │\n│      mmd │   model │              moons │       FluxEnsemble │   Generic │        0.634 │\n│      mmd │   model │              moons │       FluxEnsemble │    Greedy │        0.092 │\n│      mmd │   model │              moons │       FluxEnsemble │    REVISE │         0.75 │\n│      mmd │   model │              moons │       FluxEnsemble │      DICE │         0.61 │\n│ mmd_grid │   model │        overlapping │ LogisticRegression │   Generic │          0.0 │\n│ mmd_grid │   model │        overlapping │ LogisticRegression │    Greedy │          0.0 │\n│ mmd_grid │   model │        overlapping │ LogisticRegression │    REVISE │          0.0 │\n│ mmd_grid │   model │        overlapping │ LogisticRegression │      DICE │          0.0 │\n│ mmd_grid │   model │        overlapping │          FluxModel │   Generic │          0.0 │\n│ mmd_grid │   model │        overlapping │          FluxModel │    Greedy │          0.0 │\n│ mmd_grid │   model │        overlapping │          FluxModel │    REVISE │        0.004 │\n│ mmd_grid │   model │        overlapping │          FluxModel │      DICE │          0.0 │\n│ mmd_grid │   model │        overlapping │       FluxEnsemble │   Generic │          0.0 │\n│ mmd_grid │   model │        overlapping │       FluxEnsemble │    Greedy │          0.0 │\n│ mmd_grid │   model │        overlapping │       FluxEnsemble │    REVISE │        0.024 │\n│ mmd_grid │   model │        overlapping │       FluxEnsemble │      DICE │          0.0 │\n│ mmd_grid │   model │ linearly_separable │ LogisticRegression │   Generic │          0.0 │\n│ mmd_grid │   model │ linearly_separable │ LogisticRegression │    Greedy │          0.0 │\n│ mmd_grid │   model │ linearly_separable │ LogisticRegression │    REVISE │          0.0 │\n│ mmd_grid │   model │ linearly_separable │ LogisticRegression │      DICE │          0.0 │\n│ mmd_grid │   model │ linearly_separable │          FluxModel │   Generic │          0.0 │\n│ mmd_grid │   model │ linearly_separable │          FluxModel │    Greedy │          0.0 │\n│ mmd_grid │   model │ linearly_separable │          FluxModel │    REVISE │        0.018 │\n│ mmd_grid │   model │ linearly_separable │          FluxModel │      DICE │          0.0 │\n│ mmd_grid │   model │ linearly_separable │       FluxEnsemble │   Generic │          0.0 │\n│ mmd_grid │   model │ linearly_separable │       FluxEnsemble │    Greedy │          0.0 │\n│ mmd_grid │   model │ linearly_separable │       FluxEnsemble │    REVISE │        0.014 │\n│ mmd_grid │   model │ linearly_separable │       FluxEnsemble │      DICE │          0.0 │\n│ mmd_grid │   model │            circles │ LogisticRegression │   Generic │          0.0 │\n│ mmd_grid │   model │            circles │ LogisticRegression │    Greedy │          0.0 │\n│ mmd_grid │   model │            circles │ LogisticRegression │    REVISE │          0.0 │\n│ mmd_grid │   model │            circles │ LogisticRegression │      DICE │          0.0 │\n│ mmd_grid │   model │            circles │          FluxModel │   Generic │          0.0 │\n│ mmd_grid │   model │            circles │          FluxModel │    Greedy │          0.0 │\n│ mmd_grid │   model │            circles │          FluxModel │    REVISE │        0.152 │\n│ mmd_grid │   model │            circles │          FluxModel │      DICE │        0.004 │\n│ mmd_grid │   model │            circles │       FluxEnsemble │   Generic │          0.0 │\n│ mmd_grid │   model │            circles │       FluxEnsemble │    Greedy │          0.0 │\n│ mmd_grid │   model │            circles │       FluxEnsemble │    REVISE │         0.14 │\n│ mmd_grid │   model │            circles │       FluxEnsemble │      DICE │        0.002 │\n│ mmd_grid │   model │              moons │ LogisticRegression │   Generic │          0.0 │\n│ mmd_grid │   model │              moons │ LogisticRegression │    Greedy │          0.0 │\n│ mmd_grid │   model │              moons │ LogisticRegression │    REVISE │          0.0 │\n│ mmd_grid │   model │              moons │ LogisticRegression │      DICE │          0.0 │\n│ mmd_grid │   model │              moons │          FluxModel │   Generic │        0.004 │\n│ mmd_grid │   model │              moons │          FluxModel │    Greedy │          0.0 │\n│ mmd_grid │   model │              moons │          FluxModel │    REVISE │        0.074 │\n│ mmd_grid │   model │              moons │          FluxModel │      DICE │        0.016 │\n│ mmd_grid │   model │              moons │       FluxEnsemble │   Generic │        0.002 │\n│ mmd_grid │   model │              moons │       FluxEnsemble │    Greedy │          0.0 │\n│ mmd_grid │   model │              moons │       FluxEnsemble │    REVISE │        0.044 │\n│ mmd_grid │   model │              moons │       FluxEnsemble │      DICE │          0.0 │\n└──────────┴─────────┴────────────────────┴────────────────────┴───────────┴──────────────┘"
  },
  {
    "objectID": "sections/experiments/index.html#sec-app-synthetic-paper",
    "href": "sections/experiments/index.html#sec-app-synthetic-paper",
    "title": "3  Experimental Results",
    "section": "4.4 Chart in paper",
    "text": "4.4 Chart in paper\nFigure 4.3 shows the chart that went into the paper.\n\nImages.load(joinpath(www_artifact_path,\"paper_synthetic_results.png\"))\n\n\n\nFigure 4.3: Chart in paper\n\n\n\n\n# echo: false\n\ngenerate_artifacts(output_path)\ngenerate_artifacts(www_path)"
  },
  {
    "objectID": "sections/experiments/index.html#experiment-1",
    "href": "sections/experiments/index.html#experiment-1",
    "title": "3  Experimental Results",
    "section": "5.1 Experiment",
    "text": "5.1 Experiment\n\nn_evals = 5\nn_rounds = 50\nevaluate_every = Int(round(n_rounds/n_evals))\nn_folds = 5\nn_samples = 10000\nT = 100\ngenerative_model_params = (epochs=250, latent_dim=8)\nresults = run_experiments(\n    experiments;\n    save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, T=T, n_samples=n_samples,\n    generative_model_params=generative_model_params\n)\nSerialization.serialize(joinpath(output_path,\"results.jls\"),results)\n\n\n5.1.1 Plots\n\nresults = Serialization.deserialize(joinpath(output_path,\"results.jls\"))\n\n\nusing Images\nline_charts = Dict()\nerrorbar_charts = Dict()\nfor (data_name, res) in results\n    plt = plot(res)\n    Images.save(joinpath(www_path, \"line_chart_$(data_name).png\"), plt)\n    line_charts[data_name] = plt\n    plt = plot(res,maximum(res.output.n))\n    Images.save(joinpath(www_path, \"errorbar_chart_$(data_name).png\"), plt)\n    errorbar_charts[data_name] = plt\nend\n\n\n\n5.1.2 Line Charts\nFigure 5.1 shows the evolution of the evaluation metrics over the course of the experiment.\n\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"line_chart\")]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n    display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Credit Default\n\n\n\n\n\n\n\n(c) GMSC\n\n\n\nFigure 5.1: Line Charts\n\n\n\n\n\n5.1.3 Error Bar Charts\nFigure 5.2 shows the evaluation metrics at the end of the experiments.\n\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"errorbar_chart\")]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n    display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Credit Default\n\n\n\n\n\n\n\n(c) GMSC\n\n\n\nFigure 5.2: Error Bar Charts"
  },
  {
    "objectID": "sections/experiments/index.html#bootstrap-1",
    "href": "sections/experiments/index.html#bootstrap-1",
    "title": "3  Experimental Results",
    "section": "5.2 Bootstrap",
    "text": "5.2 Bootstrap\n\nn_bootstrap = 100\ndf = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,\"bootstrap.csv\"))\n\n\n\n┌──────────┬─────────┬────────────────┬────────────────────┬───────────┬──────────────┐\n│     name │   scope │           data │              model │ generator │ p_value_mean │\n│ String31 │ String7 │       String15 │           String31 │   String7 │      Float64 │\n├──────────┼─────────┼────────────────┼────────────────────┼───────────┼──────────────┤\n│      mmd │  domain │ credit_default │ LogisticRegression │   Generic │        0.594 │\n│      mmd │  domain │ credit_default │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │  domain │ credit_default │ LogisticRegression │      DICE │        0.268 │\n│      mmd │  domain │ credit_default │ LogisticRegression │    Greedy │        0.388 │\n│      mmd │  domain │ credit_default │          FluxModel │   Generic │        0.668 │\n│      mmd │  domain │ credit_default │          FluxModel │    REVISE │          0.0 │\n│      mmd │  domain │ credit_default │          FluxModel │      DICE │        0.466 │\n│      mmd │  domain │ credit_default │          FluxModel │    Greedy │        0.998 │\n│      mmd │  domain │ credit_default │       FluxEnsemble │   Generic │        0.738 │\n│      mmd │  domain │ credit_default │       FluxEnsemble │    REVISE │          0.0 │\n│      mmd │  domain │ credit_default │       FluxEnsemble │      DICE │        0.634 │\n│      mmd │  domain │ credit_default │       FluxEnsemble │    Greedy │          1.0 │\n│      mmd │  domain │    cal_housing │ LogisticRegression │   Generic │          0.0 │\n│      mmd │  domain │    cal_housing │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │  domain │    cal_housing │ LogisticRegression │      DICE │          0.0 │\n│      mmd │  domain │    cal_housing │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │  domain │    cal_housing │          FluxModel │   Generic │          0.0 │\n│      mmd │  domain │    cal_housing │          FluxModel │    REVISE │          0.0 │\n│      mmd │  domain │    cal_housing │          FluxModel │      DICE │          0.0 │\n│      mmd │  domain │    cal_housing │          FluxModel │    Greedy │          0.0 │\n│      mmd │  domain │    cal_housing │       FluxEnsemble │   Generic │          0.0 │\n│      mmd │  domain │    cal_housing │       FluxEnsemble │    REVISE │          0.0 │\n│      mmd │  domain │    cal_housing │       FluxEnsemble │      DICE │          0.0 │\n│      mmd │  domain │    cal_housing │       FluxEnsemble │    Greedy │          0.0 │\n│      mmd │  domain │           gmsc │ LogisticRegression │   Generic │          0.0 │\n│      mmd │  domain │           gmsc │ LogisticRegression │    REVISE │        0.112 │\n│      mmd │  domain │           gmsc │ LogisticRegression │      DICE │          0.0 │\n│      mmd │  domain │           gmsc │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │  domain │           gmsc │          FluxModel │   Generic │          0.0 │\n│      mmd │  domain │           gmsc │          FluxModel │    REVISE │          0.0 │\n│      mmd │  domain │           gmsc │          FluxModel │      DICE │          0.0 │\n│      mmd │  domain │           gmsc │          FluxModel │    Greedy │          0.0 │\n│      mmd │  domain │           gmsc │       FluxEnsemble │   Generic │          0.0 │\n│      mmd │  domain │           gmsc │       FluxEnsemble │    REVISE │          0.0 │\n│      mmd │  domain │           gmsc │       FluxEnsemble │      DICE │          0.0 │\n│      mmd │  domain │           gmsc │       FluxEnsemble │    Greedy │          0.0 │\n│      mmd │   model │ credit_default │ LogisticRegression │   Generic │          0.0 │\n│      mmd │   model │ credit_default │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │   model │ credit_default │ LogisticRegression │      DICE │          0.0 │\n│      mmd │   model │ credit_default │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │   model │ credit_default │          FluxModel │   Generic │          0.0 │\n│      mmd │   model │ credit_default │          FluxModel │    REVISE │          0.0 │\n│      mmd │   model │ credit_default │          FluxModel │      DICE │          0.0 │\n│      mmd │   model │ credit_default │          FluxModel │    Greedy │          0.0 │\n│      mmd │   model │ credit_default │       FluxEnsemble │   Generic │          0.0 │\n│      mmd │   model │ credit_default │       FluxEnsemble │    REVISE │          0.0 │\n│      mmd │   model │ credit_default │       FluxEnsemble │      DICE │          0.0 │\n│      mmd │   model │ credit_default │       FluxEnsemble │    Greedy │          0.0 │\n│      mmd │   model │    cal_housing │ LogisticRegression │   Generic │          0.0 │\n│      mmd │   model │    cal_housing │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │   model │    cal_housing │ LogisticRegression │      DICE │          0.0 │\n│      mmd │   model │    cal_housing │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │   model │    cal_housing │          FluxModel │   Generic │          0.0 │\n│      mmd │   model │    cal_housing │          FluxModel │    REVISE │          0.0 │\n│      mmd │   model │    cal_housing │          FluxModel │      DICE │          0.0 │\n│      mmd │   model │    cal_housing │          FluxModel │    Greedy │          0.0 │\n│      mmd │   model │    cal_housing │       FluxEnsemble │   Generic │          0.0 │\n│      mmd │   model │    cal_housing │       FluxEnsemble │    REVISE │          0.0 │\n│      mmd │   model │    cal_housing │       FluxEnsemble │      DICE │          0.0 │\n│      mmd │   model │    cal_housing │       FluxEnsemble │    Greedy │          0.0 │\n│      mmd │   model │           gmsc │ LogisticRegression │   Generic │          0.0 │\n│      mmd │   model │           gmsc │ LogisticRegression │    REVISE │          0.0 │\n│      mmd │   model │           gmsc │ LogisticRegression │      DICE │          0.0 │\n│      mmd │   model │           gmsc │ LogisticRegression │    Greedy │          0.0 │\n│      mmd │   model │           gmsc │          FluxModel │   Generic │          0.0 │\n│      mmd │   model │           gmsc │          FluxModel │    REVISE │          0.0 │\n│      mmd │   model │           gmsc │          FluxModel │      DICE │          0.0 │\n│      mmd │   model │           gmsc │          FluxModel │    Greedy │          0.0 │\n│      mmd │   model │           gmsc │       FluxEnsemble │   Generic │          0.0 │\n│      mmd │   model │           gmsc │       FluxEnsemble │    REVISE │          0.0 │\n│      mmd │   model │           gmsc │       FluxEnsemble │      DICE │          0.0 │\n│      mmd │   model │           gmsc │       FluxEnsemble │    Greedy │          0.0 │\n│ mmd_grid │   model │ credit_default │ LogisticRegression │   Generic │          0.0 │\n│ mmd_grid │   model │ credit_default │ LogisticRegression │    REVISE │          0.0 │\n│ mmd_grid │   model │ credit_default │ LogisticRegression │      DICE │          0.0 │\n│ mmd_grid │   model │ credit_default │ LogisticRegression │    Greedy │          0.0 │\n│ mmd_grid │   model │ credit_default │          FluxModel │   Generic │          0.0 │\n│ mmd_grid │   model │ credit_default │          FluxModel │    REVISE │          0.0 │\n│ mmd_grid │   model │ credit_default │          FluxModel │      DICE │          0.0 │\n│ mmd_grid │   model │ credit_default │          FluxModel │    Greedy │          0.0 │\n│ mmd_grid │   model │ credit_default │       FluxEnsemble │   Generic │          0.0 │\n│ mmd_grid │   model │ credit_default │       FluxEnsemble │    REVISE │          0.0 │\n│ mmd_grid │   model │ credit_default │       FluxEnsemble │      DICE │          0.0 │\n│ mmd_grid │   model │ credit_default │       FluxEnsemble │    Greedy │          0.0 │\n│ mmd_grid │   model │    cal_housing │ LogisticRegression │   Generic │          0.0 │\n│ mmd_grid │   model │    cal_housing │ LogisticRegression │    REVISE │          0.0 │\n│ mmd_grid │   model │    cal_housing │ LogisticRegression │      DICE │          0.0 │\n│ mmd_grid │   model │    cal_housing │ LogisticRegression │    Greedy │          0.0 │\n│ mmd_grid │   model │    cal_housing │          FluxModel │   Generic │          0.0 │\n│ mmd_grid │   model │    cal_housing │          FluxModel │    REVISE │          0.0 │\n│ mmd_grid │   model │    cal_housing │          FluxModel │      DICE │          0.0 │\n│ mmd_grid │   model │    cal_housing │          FluxModel │    Greedy │          0.0 │\n│ mmd_grid │   model │    cal_housing │       FluxEnsemble │   Generic │          0.0 │\n│ mmd_grid │   model │    cal_housing │       FluxEnsemble │    REVISE │          0.0 │\n│ mmd_grid │   model │    cal_housing │       FluxEnsemble │      DICE │          0.0 │\n│ mmd_grid │   model │    cal_housing │       FluxEnsemble │    Greedy │          0.0 │\n│ mmd_grid │   model │           gmsc │ LogisticRegression │   Generic │          0.0 │\n│ mmd_grid │   model │           gmsc │ LogisticRegression │    REVISE │          0.0 │\n│ mmd_grid │   model │           gmsc │ LogisticRegression │      DICE │          0.0 │\n│ mmd_grid │   model │           gmsc │ LogisticRegression │    Greedy │          0.0 │\n│ mmd_grid │   model │           gmsc │          FluxModel │   Generic │         0.06 │\n│ mmd_grid │   model │           gmsc │          FluxModel │    REVISE │          0.0 │\n│ mmd_grid │   model │           gmsc │          FluxModel │      DICE │          0.0 │\n│ mmd_grid │   model │           gmsc │          FluxModel │    Greedy │          0.0 │\n│ mmd_grid │   model │           gmsc │       FluxEnsemble │   Generic │          0.0 │\n│ mmd_grid │   model │           gmsc │       FluxEnsemble │    REVISE │          0.0 │\n│ mmd_grid │   model │           gmsc │       FluxEnsemble │      DICE │        0.002 │\n│ mmd_grid │   model │           gmsc │       FluxEnsemble │    Greedy │          0.0 │\n└──────────┴─────────┴────────────────┴────────────────────┴───────────┴──────────────┘\n\n\n\n5.2.1 Chart in paper\nFigure 5.3 shows the chart that went into the paper.\n\nusing DataFrames, Statistics\nmodel_ = :FluxEnsemble\ndf = DataFrame() \nfor (key, val) in results\n    df_ = deepcopy(val.output)\n    df_.dataset .= key\n    df = vcat(df,df_)\nend\ndf = df[df.n .== maximum(df.n),:]\ndf = df[df.model .== model_,:]\nfilter!(:value => x -> !any(f -> f(x), (ismissing, isnothing, isnan)), df)\ngdf = groupby(df, [:generator, :dataset, :n, :name, :scope])\ndf_plot = combine(gdf, :value => (x -> [(mean(x),mean(x)+std(x),mean(x)-std(x))]) => [:mean, :ymax, :ymin])\ndf_plot = df_plot[[name in [:mmd, :model_performance] for name in df_plot.name],:]\ndf_plot = mapcols(x -> typeof(x) == Vector{Symbol} ? string.(x) : x, df_plot)\ndf_plot.name .= [r[:name] == \"mmd\" ? \"$(r[:name])_$(r[:scope])\" : r[:name] for r in eachrow(df_plot)]\ntransform!(df_plot, :dataset => (X -> [x==\"cal_housing\" ? \"California Housing\" : x for x in X]) => :dataset)\ntransform!(df_plot, :dataset => (X -> [x==\"credit_default\" ? \"Credit Default\" : x for x in X]) => :dataset)\ntransform!(df_plot, :dataset => (X -> [x==\"gmsc\" ? \"GMSC\" : x for x in X]) => :dataset)\ntransform!(df_plot, :name => (X -> [x==\"mmd_domain\" ? \"MMD (domain)\" : x for x in X]) => :name)\ntransform!(df_plot, :name => (X -> [x==\"mmd_model\" ? \"MMD (model)\" : x for x in X]) => :name)\ntransform!(df_plot, :name => (X -> [x==\"model_performance\" ? \"Performance\" : x for x in X]) => :name)\ntransform!(df_plot, :generator => (X -> [x==\"REVISE\" ? \"Latent\" : x for x in X]) => :generator)\n\nncol = length(unique(df_plot.dataset))\nnrow = length(unique(df_plot.name))\n\nusing RCall\nscale_ = 1.75\nR\"\"\"\nlibrary(ggplot2)\nplt <- ggplot($df_plot) +\n    geom_bar(aes(x=n, y=mean, fill=generator), stat=\"identity\", alpha=0.5, position=\"dodge\") +\n    geom_pointrange( aes(x=n, y=mean, ymin=ymin, ymax=ymax, colour=generator), alpha=0.9, position=position_dodge(width=0.9), size=0.5) +\n    facet_grid(\n        rows = vars(name),\n        cols =  vars(dataset), \n        scales = \"free_y\"\n    ) +\n    labs(y = \"Value\") + \n    scale_fill_discrete(name=\"Generator:\") +\n    scale_colour_discrete(name=\"Generator:\") +\n    theme(\n        axis.title.x=element_blank(),\n        axis.text.x=element_blank(),\n        axis.ticks.x=element_blank(),\n        legend.position=\"bottom\"\n    )\ntemp_path <- file.path(tempdir(), \"plot.png\")\nggsave(temp_path,width=$ncol * $scale_,height=$nrow * $scale_ * 0.8) \n\"\"\"\n\nimg = Images.load(rcopy(R\"temp_path\"))\nImages.save(joinpath(www_path,\"paper_real_world_results.png\"), img)\n\n\nImages.load(joinpath(www_artifact_path,\"paper_real_world_results.png\"))\n\n\n\nFigure 5.3: Chart in paper"
  },
  {
    "objectID": "sections/experiments/index.html#synthetic",
    "href": "sections/experiments/index.html#synthetic",
    "title": "3  Experimental Results",
    "section": "6.1 Synthetic",
    "text": "6.1 Synthetic\n\nmax_obs = 1000\ncatalogue = load_synthetic(max_obs)\nchoices = [\n    :linearly_separable, \n    :overlapping, \n    :circles, \n    :moons,\n]\ndata_sets = filter(p -> p[1] in choices, catalogue)\n\n\nexperiments = set_up_experiments(data_sets,models,generators)\n\n\nn_evals = 5\nn_rounds = 50\nevaluate_every = Int(round(n_rounds/n_evals))\nn_folds = 5\nT = 100\nusing Serialization\nresults = run_experiments(\n    experiments;\n    save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, T=T\n)\nSerialization.serialize(joinpath(output_path,\"results_synthetic.jls\"),results)"
  },
  {
    "objectID": "sections/experiments/index.html#plots-2",
    "href": "sections/experiments/index.html#plots-2",
    "title": "3  Experimental Results",
    "section": "6.2 Plots",
    "text": "6.2 Plots\n\nusing Serialization\nresults = Serialization.deserialize(joinpath(output_path,\"results_synthetic.jls\"))\n\n\nusing Images\nline_charts = Dict()\nerrorbar_charts = Dict()\nfor (data_name, res) in results\n    plt = plot(res)\n    Images.save(joinpath(www_path, \"line_chart_$(data_name).png\"), plt)\n    line_charts[data_name] = plt\n    plt = plot(res,maximum(res.output.n))\n    Images.save(joinpath(www_path, \"errorbar_chart_$(data_name).png\"), plt)\n    errorbar_charts[data_name] = plt\nend\n\n\n6.2.1 Line Charts\nFigure 6.1 shows the evolution of the evaluation metrics over the course of the experiment.\n\nchoices = [\n    :linearly_separable, \n    :overlapping, \n    :circles, \n    :moons,\n]\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"line_chart\") .&& .!contains.(readdir(www_artifact_path),\"latent\")]\nimg_files = img_files[Bool.(reduce(+, map(choice -> contains.(img_files, string(choice)), choices)))]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n    display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Circles\n\n\n\n\n\n\n\n(c) Credit Default\n\n\n\n\n\n\n\n(d) GMSC\n\n\n\nFigure 6.1: Line Charts\n\n\n\n\n\n6.2.2 Error Bar Charts\nFigure 6.2 shows the evaluation metrics at the end of the experiments.\n\nchoices = [\n    :linearly_separable, \n    :overlapping, \n    :circles, \n    :moons,\n]\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"errorbar_chart\") .&& .!contains.(readdir(www_artifact_path),\"latent\")]\nimg_files = img_files[Bool.(reduce(+, map(choice -> contains.(img_files, string(choice)), choices)))]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n    display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Circles\n\n\n\n\n\n\n\n(c) Credit Default\n\n\n\n\n\n\n\n(d) GMSC\n\n\n\nFigure 6.2: Error Bar Charts"
  },
  {
    "objectID": "sections/experiments/index.html#bootstrap-2",
    "href": "sections/experiments/index.html#bootstrap-2",
    "title": "3  Experimental Results",
    "section": "6.3 Bootstrap",
    "text": "6.3 Bootstrap\n\nn_bootstrap = 100\ndf = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,\"bootstrap_synthetic.csv\"))\n\n\n\n┌──────────┬─────────┬────────────────────┬────────────────────┬──────────────────────┬──────────────┐\n│     name │   scope │               data │              model │            generator │ p_value_mean │\n│ String31 │ String7 │           String31 │           String31 │             String31 │      Float64 │\n├──────────┼─────────┼────────────────────┼────────────────────┼──────────────────────┼──────────────┤\n│      mmd │  domain │        overlapping │          FluxModel │ Generic_conservative │          0.0 │\n│      mmd │  domain │        overlapping │          FluxModel │        Gravitational │          0.0 │\n│      mmd │  domain │        overlapping │          FluxModel │             ClapROAR │          0.0 │\n│      mmd │  domain │        overlapping │          FluxModel │               Latent │          0.0 │\n│      mmd │  domain │        overlapping │          FluxModel │              Generic │          0.0 │\n│      mmd │  domain │        overlapping │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │  domain │        overlapping │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │  domain │        overlapping │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │  domain │        overlapping │ LogisticRegression │               Latent │          0.0 │\n│      mmd │  domain │        overlapping │ LogisticRegression │              Generic │          0.0 │\n│      mmd │  domain │        overlapping │       FluxEnsemble │ Generic_conservative │          0.0 │\n│      mmd │  domain │        overlapping │       FluxEnsemble │        Gravitational │          0.0 │\n│      mmd │  domain │        overlapping │       FluxEnsemble │             ClapROAR │          0.0 │\n│      mmd │  domain │        overlapping │       FluxEnsemble │               Latent │          0.0 │\n│      mmd │  domain │        overlapping │       FluxEnsemble │              Generic │          0.0 │\n│      mmd │  domain │ linearly_separable │          FluxModel │ Generic_conservative │          0.0 │\n│      mmd │  domain │ linearly_separable │          FluxModel │        Gravitational │        0.184 │\n│      mmd │  domain │ linearly_separable │          FluxModel │             ClapROAR │          0.0 │\n│      mmd │  domain │ linearly_separable │          FluxModel │               Latent │        0.358 │\n│      mmd │  domain │ linearly_separable │          FluxModel │              Generic │          0.0 │\n│      mmd │  domain │ linearly_separable │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │  domain │ linearly_separable │ LogisticRegression │        Gravitational │         0.27 │\n│      mmd │  domain │ linearly_separable │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │  domain │ linearly_separable │ LogisticRegression │               Latent │        0.586 │\n│      mmd │  domain │ linearly_separable │ LogisticRegression │              Generic │          0.0 │\n│      mmd │  domain │ linearly_separable │       FluxEnsemble │ Generic_conservative │          0.0 │\n│      mmd │  domain │ linearly_separable │       FluxEnsemble │        Gravitational │        0.152 │\n│      mmd │  domain │ linearly_separable │       FluxEnsemble │             ClapROAR │          0.0 │\n│      mmd │  domain │ linearly_separable │       FluxEnsemble │               Latent │         0.38 │\n│      mmd │  domain │ linearly_separable │       FluxEnsemble │              Generic │          0.0 │\n│      mmd │  domain │            circles │          FluxModel │ Generic_conservative │          0.0 │\n│      mmd │  domain │            circles │          FluxModel │        Gravitational │          0.0 │\n│      mmd │  domain │            circles │          FluxModel │             ClapROAR │          0.0 │\n│      mmd │  domain │            circles │          FluxModel │               Latent │          1.0 │\n│      mmd │  domain │            circles │          FluxModel │              Generic │          0.0 │\n│      mmd │  domain │            circles │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │  domain │            circles │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │  domain │            circles │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │  domain │            circles │ LogisticRegression │               Latent │          0.0 │\n│      mmd │  domain │            circles │ LogisticRegression │              Generic │          0.0 │\n│      mmd │  domain │            circles │       FluxEnsemble │ Generic_conservative │          0.0 │\n│      mmd │  domain │            circles │       FluxEnsemble │        Gravitational │          0.0 │\n│      mmd │  domain │            circles │       FluxEnsemble │             ClapROAR │          0.0 │\n│      mmd │  domain │            circles │       FluxEnsemble │               Latent │          1.0 │\n│      mmd │  domain │            circles │       FluxEnsemble │              Generic │          0.0 │\n│      mmd │  domain │              moons │          FluxModel │ Generic_conservative │          0.0 │\n│      mmd │  domain │              moons │          FluxModel │        Gravitational │          0.0 │\n│      mmd │  domain │              moons │          FluxModel │             ClapROAR │          0.0 │\n│      mmd │  domain │              moons │          FluxModel │               Latent │          0.0 │\n│      mmd │  domain │              moons │          FluxModel │              Generic │          0.0 │\n│      mmd │  domain │              moons │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │  domain │              moons │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │  domain │              moons │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │  domain │              moons │ LogisticRegression │               Latent │          0.0 │\n│      mmd │  domain │              moons │ LogisticRegression │              Generic │          0.0 │\n│      mmd │  domain │              moons │       FluxEnsemble │ Generic_conservative │          0.0 │\n│      mmd │  domain │              moons │       FluxEnsemble │        Gravitational │          0.0 │\n│      mmd │  domain │              moons │       FluxEnsemble │             ClapROAR │          0.0 │\n│      mmd │  domain │              moons │       FluxEnsemble │               Latent │          0.0 │\n│      mmd │  domain │              moons │       FluxEnsemble │              Generic │          0.0 │\n│      mmd │   model │        overlapping │          FluxModel │ Generic_conservative │          0.0 │\n│      mmd │   model │        overlapping │          FluxModel │        Gravitational │        0.024 │\n│      mmd │   model │        overlapping │          FluxModel │             ClapROAR │        0.012 │\n│      mmd │   model │        overlapping │          FluxModel │               Latent │          0.0 │\n│      mmd │   model │        overlapping │          FluxModel │              Generic │          0.0 │\n│      mmd │   model │        overlapping │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │   model │        overlapping │ LogisticRegression │        Gravitational │        0.006 │\n│      mmd │   model │        overlapping │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │   model │        overlapping │ LogisticRegression │               Latent │          0.0 │\n│      mmd │   model │        overlapping │ LogisticRegression │              Generic │          0.0 │\n│      mmd │   model │        overlapping │       FluxEnsemble │ Generic_conservative │          0.0 │\n│      mmd │   model │        overlapping │       FluxEnsemble │        Gravitational │        0.018 │\n│      mmd │   model │        overlapping │       FluxEnsemble │             ClapROAR │        0.004 │\n│      mmd │   model │        overlapping │       FluxEnsemble │               Latent │          0.0 │\n│      mmd │   model │        overlapping │       FluxEnsemble │              Generic │          0.0 │\n│      mmd │   model │ linearly_separable │          FluxModel │ Generic_conservative │        0.852 │\n│      mmd │   model │ linearly_separable │          FluxModel │        Gravitational │         0.89 │\n│      mmd │   model │ linearly_separable │          FluxModel │             ClapROAR │         0.84 │\n│      mmd │   model │ linearly_separable │          FluxModel │               Latent │        0.722 │\n│      mmd │   model │ linearly_separable │          FluxModel │              Generic │         0.83 │\n│      mmd │   model │ linearly_separable │ LogisticRegression │ Generic_conservative │        0.176 │\n│      mmd │   model │ linearly_separable │ LogisticRegression │        Gravitational │        0.898 │\n│      mmd │   model │ linearly_separable │ LogisticRegression │             ClapROAR │        0.866 │\n│      mmd │   model │ linearly_separable │ LogisticRegression │               Latent │        0.338 │\n│      mmd │   model │ linearly_separable │ LogisticRegression │              Generic │        0.002 │\n│      mmd │   model │ linearly_separable │       FluxEnsemble │ Generic_conservative │        0.868 │\n│      mmd │   model │ linearly_separable │       FluxEnsemble │        Gravitational │        0.912 │\n│      mmd │   model │ linearly_separable │       FluxEnsemble │             ClapROAR │        0.916 │\n│      mmd │   model │ linearly_separable │       FluxEnsemble │               Latent │        0.822 │\n│      mmd │   model │ linearly_separable │       FluxEnsemble │              Generic │        0.818 │\n│      mmd │   model │            circles │          FluxModel │ Generic_conservative │        0.356 │\n│      mmd │   model │            circles │          FluxModel │        Gravitational │         0.48 │\n│      mmd │   model │            circles │          FluxModel │             ClapROAR │        0.484 │\n│      mmd │   model │            circles │          FluxModel │               Latent │        0.802 │\n│      mmd │   model │            circles │          FluxModel │              Generic │        0.482 │\n│      mmd │   model │            circles │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │   model │            circles │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │   model │            circles │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │   model │            circles │ LogisticRegression │               Latent │          0.0 │\n│      mmd │   model │            circles │ LogisticRegression │              Generic │          0.0 │\n│      mmd │   model │            circles │       FluxEnsemble │ Generic_conservative │        0.462 │\n│      mmd │   model │            circles │       FluxEnsemble │        Gravitational │        0.282 │\n│      mmd │   model │            circles │       FluxEnsemble │             ClapROAR │        0.534 │\n│      mmd │   model │            circles │       FluxEnsemble │               Latent │         0.91 │\n│      mmd │   model │            circles │       FluxEnsemble │              Generic │        0.408 │\n│      mmd │   model │              moons │          FluxModel │ Generic_conservative │        0.568 │\n│      mmd │   model │              moons │          FluxModel │        Gravitational │        0.868 │\n│      mmd │   model │              moons │          FluxModel │             ClapROAR │        0.594 │\n│      mmd │   model │              moons │          FluxModel │               Latent │        0.598 │\n│      mmd │   model │              moons │          FluxModel │              Generic │        0.604 │\n│      mmd │   model │              moons │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │   model │              moons │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │   model │              moons │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │   model │              moons │ LogisticRegression │               Latent │          0.0 │\n│      mmd │   model │              moons │ LogisticRegression │              Generic │          0.0 │\n│      mmd │   model │              moons │       FluxEnsemble │ Generic_conservative │         0.48 │\n│      mmd │   model │              moons │       FluxEnsemble │        Gravitational │        0.922 │\n│      mmd │   model │              moons │       FluxEnsemble │             ClapROAR │        0.584 │\n│      mmd │   model │              moons │       FluxEnsemble │               Latent │        0.654 │\n│      mmd │   model │              moons │       FluxEnsemble │              Generic │        0.412 │\n│ mmd_grid │   model │        overlapping │          FluxModel │ Generic_conservative │        0.002 │\n│ mmd_grid │   model │        overlapping │          FluxModel │        Gravitational │        0.034 │\n│ mmd_grid │   model │        overlapping │          FluxModel │             ClapROAR │        0.022 │\n│ mmd_grid │   model │        overlapping │          FluxModel │               Latent │        0.008 │\n│ mmd_grid │   model │        overlapping │          FluxModel │              Generic │          0.0 │\n│ mmd_grid │   model │        overlapping │ LogisticRegression │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │        overlapping │ LogisticRegression │        Gravitational │          0.0 │\n│ mmd_grid │   model │        overlapping │ LogisticRegression │             ClapROAR │        0.002 │\n│ mmd_grid │   model │        overlapping │ LogisticRegression │               Latent │          0.0 │\n│ mmd_grid │   model │        overlapping │ LogisticRegression │              Generic │          0.0 │\n│ mmd_grid │   model │        overlapping │       FluxEnsemble │ Generic_conservative │        0.006 │\n│ mmd_grid │   model │        overlapping │       FluxEnsemble │        Gravitational │        0.034 │\n│ mmd_grid │   model │        overlapping │       FluxEnsemble │             ClapROAR │        0.058 │\n│ mmd_grid │   model │        overlapping │       FluxEnsemble │               Latent │        0.016 │\n│ mmd_grid │   model │        overlapping │       FluxEnsemble │              Generic │          0.0 │\n│ mmd_grid │   model │ linearly_separable │          FluxModel │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │ linearly_separable │          FluxModel │        Gravitational │        0.062 │\n│ mmd_grid │   model │ linearly_separable │          FluxModel │             ClapROAR │          0.0 │\n│ mmd_grid │   model │ linearly_separable │          FluxModel │               Latent │          0.0 │\n│ mmd_grid │   model │ linearly_separable │          FluxModel │              Generic │          0.0 │\n│ mmd_grid │   model │ linearly_separable │ LogisticRegression │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │ linearly_separable │ LogisticRegression │        Gravitational │        0.004 │\n│ mmd_grid │   model │ linearly_separable │ LogisticRegression │             ClapROAR │          0.0 │\n│ mmd_grid │   model │ linearly_separable │ LogisticRegression │               Latent │          0.0 │\n│ mmd_grid │   model │ linearly_separable │ LogisticRegression │              Generic │          0.0 │\n│ mmd_grid │   model │ linearly_separable │       FluxEnsemble │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │ linearly_separable │       FluxEnsemble │        Gravitational │        0.128 │\n│ mmd_grid │   model │ linearly_separable │       FluxEnsemble │             ClapROAR │          0.0 │\n│ mmd_grid │   model │ linearly_separable │       FluxEnsemble │               Latent │          0.0 │\n│ mmd_grid │   model │ linearly_separable │       FluxEnsemble │              Generic │          0.0 │\n│ mmd_grid │   model │            circles │          FluxModel │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │            circles │          FluxModel │        Gravitational │        0.002 │\n│ mmd_grid │   model │            circles │          FluxModel │             ClapROAR │        0.004 │\n│ mmd_grid │   model │            circles │          FluxModel │               Latent │        0.176 │\n│ mmd_grid │   model │            circles │          FluxModel │              Generic │        0.008 │\n│ mmd_grid │   model │            circles │ LogisticRegression │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │            circles │ LogisticRegression │        Gravitational │          0.0 │\n│ mmd_grid │   model │            circles │ LogisticRegression │             ClapROAR │          0.0 │\n│ mmd_grid │   model │            circles │ LogisticRegression │               Latent │          0.0 │\n│ mmd_grid │   model │            circles │ LogisticRegression │              Generic │          0.0 │\n│ mmd_grid │   model │            circles │       FluxEnsemble │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │            circles │       FluxEnsemble │        Gravitational │          0.0 │\n│ mmd_grid │   model │            circles │       FluxEnsemble │             ClapROAR │        0.002 │\n│ mmd_grid │   model │            circles │       FluxEnsemble │               Latent │         0.24 │\n│ mmd_grid │   model │            circles │       FluxEnsemble │              Generic │          0.0 │\n│ mmd_grid │   model │              moons │          FluxModel │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │              moons │          FluxModel │        Gravitational │        0.052 │\n│ mmd_grid │   model │              moons │          FluxModel │             ClapROAR │        0.002 │\n│ mmd_grid │   model │              moons │          FluxModel │               Latent │        0.024 │\n│ mmd_grid │   model │              moons │          FluxModel │              Generic │          0.0 │\n│ mmd_grid │   model │              moons │ LogisticRegression │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │              moons │ LogisticRegression │        Gravitational │          0.0 │\n│ mmd_grid │   model │              moons │ LogisticRegression │             ClapROAR │          0.0 │\n│ mmd_grid │   model │              moons │ LogisticRegression │               Latent │          0.0 │\n│ mmd_grid │   model │              moons │ LogisticRegression │              Generic │          0.0 │\n│ mmd_grid │   model │              moons │       FluxEnsemble │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │              moons │       FluxEnsemble │        Gravitational │        0.164 │\n│ mmd_grid │   model │              moons │       FluxEnsemble │             ClapROAR │          0.0 │\n│ mmd_grid │   model │              moons │       FluxEnsemble │               Latent │        0.058 │\n│ mmd_grid │   model │              moons │       FluxEnsemble │              Generic │          0.0 │\n└──────────┴─────────┴────────────────────┴────────────────────┴──────────────────────┴──────────────┘"
  },
  {
    "objectID": "sections/experiments/index.html#chart-in-paper-1",
    "href": "sections/experiments/index.html#chart-in-paper-1",
    "title": "3  Experimental Results",
    "section": "6.4 Chart in paper",
    "text": "6.4 Chart in paper\nFigure 6.3 shows the chart that went into the paper.\n\nImages.load(joinpath(www_artifact_path,\"paper_synthetic_results.png\"))\n\n\n\nFigure 6.3: Chart in paper"
  },
  {
    "objectID": "sections/experiments/index.html#latent-space-search",
    "href": "sections/experiments/index.html#latent-space-search",
    "title": "3  Experimental Results",
    "section": "6.5 Latent Space Search",
    "text": "6.5 Latent Space Search\n\ngenerators = Dict(\n    :Latent=>GenericGenerator(opt = opt, decision_threshold=0.5),\n    :Latent_conservative=>GenericGenerator(opt = opt, decision_threshold=0.9),\n    :Gravitational=>GravitationalGenerator(opt = opt),\n    :ClapROAR=>ClapROARGenerator(opt = opt)\n)\n\n\nexperiments = set_up_experiments(data_sets,models,generators)\n\n\nn_evals = 5\nn_rounds = 50\nevaluate_every = Int(round(n_rounds/n_evals))\nn_folds = 5\nT = 100\nusing Serialization\nresults = run_experiments(\n    experiments;\n    save_path=output_path, evaluate_every=evaluate_every, n_rounds=n_rounds, n_folds=n_folds, T=T, \n    latent_space = true\n)\nSerialization.serialize(joinpath(output_path,\"results_synthetic_latent.jls\"),results)\n\n\nusing Serialization\nresults = Serialization.deserialize(joinpath(output_path,\"results_synthetic_latent.jls\"))\n\n\nusing Images\nline_charts = Dict()\nerrorbar_charts = Dict()\nfor (data_name, res) in results\n    plt = plot(res)\n    Images.save(joinpath(www_path, \"line_chart_latent_$(data_name).png\"), plt)\n    line_charts[data_name] = plt\n    plt = plot(res,maximum(res.output.n))\n    Images.save(joinpath(www_path, \"errorbar_chart_latent_$(data_name).png\"), plt)\n    errorbar_charts[data_name] = plt\nend"
  },
  {
    "objectID": "sections/experiments/index.html#plots-3",
    "href": "sections/experiments/index.html#plots-3",
    "title": "3  Experimental Results",
    "section": "6.6 Plots",
    "text": "6.6 Plots\n\n6.6.1 Line Charts\nFigure 6.4 shows the evolution of the evaluation metrics over the course of the experiment.\n\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"line_chart\") .&& contains.(readdir(www_artifact_path),\"latent\")]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n    display(load(img))\nend\n\n\n\n\n\n\n(a) Circles\n\n\n\n\n\n\n\n(b) Linearly Separable\n\n\n\n\n\n\n\n(c) Moons\n\n\n\n\n\n\n\n(d) Overlapping\n\n\n\nFigure 6.4: Line Charts\n\n\n\n\n\n6.6.2 Error Bar Charts\nFigure 6.5 shows the evaluation metrics at the end of the experiments.\n\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"errorbar_chart\") .&& contains.(readdir(www_artifact_path),\"latent\")]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n    display(load(img))\nend\n\n\n\n\n\n\n(a) Circles\n\n\n\n\n\n\n\n(b) Linearly Separable\n\n\n\n\n\n\n\n(c) Moons\n\n\n\n\n\n\n\n\n(d) Overlapping\n\n\n\nFigure 6.5: Error Bar Charts"
  },
  {
    "objectID": "sections/experiments/index.html#bootstrap-3",
    "href": "sections/experiments/index.html#bootstrap-3",
    "title": "3  Experimental Results",
    "section": "6.7 Bootstrap",
    "text": "6.7 Bootstrap\n\nn_bootstrap = 100\ndf = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,\"bootstrap_latent.csv\"))"
  },
  {
    "objectID": "sections/experiments/index.html#chart-in-paper-2",
    "href": "sections/experiments/index.html#chart-in-paper-2",
    "title": "3  Experimental Results",
    "section": "6.8 Chart in paper",
    "text": "6.8 Chart in paper\nFigure 6.6 shows the chart that went into the paper.\n\nImages.load(joinpath(www_artifact_path,\"paper_synthetic_latent_results.png\"))\n\n\n\nFigure 6.6: Chart in paper"
  },
  {
    "objectID": "sections/experiments/index.html#real-world",
    "href": "sections/experiments/index.html#real-world",
    "title": "3  Experimental Results",
    "section": "6.9 Real World",
    "text": "6.9 Real World\n\nmodels = [\n    :LogisticRegression, \n    :FluxModel, \n    :FluxEnsemble,\n]\nopt = Flux.Descent(0.01) \ngenerators = Dict(\n    :Generic=>GenericGenerator(opt = opt, decision_threshold=0.5),\n    :Latent=>REVISEGenerator(opt = opt),\n    :Generic_conservative=>GenericGenerator(opt = opt, decision_threshold=0.9),\n    :Gravitational=>GravitationalGenerator(opt = opt),\n    :ClapROAR=>ClapROARGenerator(opt = opt)\n)\n\n\nmax_obs = 2500\ndata_path = data_dir(\"real_world\")\ndata_sets = load_real_world(max_obs)\nchoices = [\n    :cal_housing, \n    :credit_default, \n    :gmsc, \n]\ndata_sets = filter(p -> p[1] in choices, data_sets)\n\n\nusing CounterfactualExplanations.DataPreprocessing: unpack\nbs = 500\nfunction data_loader(data::CounterfactualData)\n    X, y = unpack(data)\n    data = Flux.DataLoader((X,y),batchsize=bs)\n    return data\nend\nmodel_params = (batch_norm=false,n_hidden=64,n_layers=3,dropout=true,p_dropout=0.1)\n\n\nexperiments = set_up_experiments(\n    data_sets,models,generators; \n    pre_train_models=100, model_params=model_params, \n    data_loader=data_loader\n)\n\n\nn_evals = 5\nn_rounds = 50\nevaluate_every = Int(round(n_rounds/n_evals))\nn_folds = 5\nn_samples = 10000\nT = 100\ngenerative_model_params = (epochs=250, latent_dim=8)\nresults = run_experiments(\n    experiments;\n    save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, T=T, n_samples=n_samples,\n    generative_model_params=generative_model_params\n)\nSerialization.serialize(joinpath(output_path,\"results_real_world.jls\"),results)\n\n\nusing Serialization\nresults = Serialization.deserialize(joinpath(output_path,\"results_real_world.jls\"))\n\n\nusing Images\nline_charts = Dict()\nerrorbar_charts = Dict()\nfor (data_name, res) in results\n    plt = plot(res)\n    Images.save(joinpath(www_path, \"line_chart_$(data_name).png\"), plt)\n    line_charts[data_name] = plt\n    plt = plot(res,maximum(res.output.n))\n    Images.save(joinpath(www_path, \"errorbar_chart_$(data_name).png\"), plt)\n    errorbar_charts[data_name] = plt\nend\n\n\n6.9.1 Line Charts\nFigure 5.1 shows the evolution of the evaluation metrics over the course of the experiment.\n\nchoices = [\n    :cal_housing, \n    :credit_default, \n    :gmsc, \n]\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"line_chart\")]\nimg_files = img_files[Bool.(reduce(+, map(choice -> contains.(img_files, string(choice)), choices)))]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n    display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Credit Default\n\n\n\n\n\n\n\n(c) GMSC\n\n\n\nFigure 6.7: Line Charts\n\n\n\n\n\n6.9.2 Error Bar Charts\nFigure 5.2 shows the evaluation metrics at the end of the experiments.\n\nchoices = [\n    :cal_housing, \n    :credit_default, \n    :gmsc, \n]\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"errorbar_chart\")]\nimg_files = img_files[Bool.(reduce(+, map(choice -> contains.(img_files, string(choice)), choices)))]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n    display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Credit Default\n\n\n\n\n\n\n\n\n(c) GMSC\n\n\n\nFigure 6.8: Error Bar Charts\n\n\n\n\n\n6.9.3 Bootstrap\n\nn_bootstrap = 100\ndf = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,\"bootstrap_real_world.csv\"))\n\n\n\n┌──────────┬─────────┬────────────────┬────────────────────┬──────────────────────┬──────────────┐\n│     name │   scope │           data │              model │            generator │ p_value_mean │\n│ String31 │ String7 │       String15 │           String31 │             String31 │      Float64 │\n├──────────┼─────────┼────────────────┼────────────────────┼──────────────────────┼──────────────┤\n│      mmd │  domain │ credit_default │       FluxEnsemble │               Latent │          0.0 │\n│      mmd │  domain │ credit_default │       FluxEnsemble │ Generic_conservative │          1.0 │\n│      mmd │  domain │ credit_default │       FluxEnsemble │             ClapROAR │          1.0 │\n│      mmd │  domain │ credit_default │       FluxEnsemble │        Gravitational │          0.0 │\n│      mmd │  domain │ credit_default │       FluxEnsemble │              Generic │          1.0 │\n│      mmd │  domain │ credit_default │          FluxModel │               Latent │          0.0 │\n│      mmd │  domain │ credit_default │          FluxModel │ Generic_conservative │          1.0 │\n│      mmd │  domain │ credit_default │          FluxModel │             ClapROAR │          1.0 │\n│      mmd │  domain │ credit_default │          FluxModel │        Gravitational │          0.0 │\n│      mmd │  domain │ credit_default │          FluxModel │              Generic │          1.0 │\n│      mmd │  domain │ credit_default │ LogisticRegression │               Latent │          0.0 │\n│      mmd │  domain │ credit_default │ LogisticRegression │ Generic_conservative │        0.256 │\n│      mmd │  domain │ credit_default │ LogisticRegression │             ClapROAR │          1.0 │\n│      mmd │  domain │ credit_default │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │  domain │ credit_default │ LogisticRegression │              Generic │          1.0 │\n│      mmd │  domain │    cal_housing │       FluxEnsemble │               Latent │        0.398 │\n│      mmd │  domain │    cal_housing │       FluxEnsemble │ Generic_conservative │          0.0 │\n│      mmd │  domain │    cal_housing │       FluxEnsemble │             ClapROAR │          0.0 │\n│      mmd │  domain │    cal_housing │       FluxEnsemble │        Gravitational │          0.0 │\n│      mmd │  domain │    cal_housing │       FluxEnsemble │              Generic │          0.0 │\n│      mmd │  domain │    cal_housing │          FluxModel │               Latent │        0.204 │\n│      mmd │  domain │    cal_housing │          FluxModel │ Generic_conservative │          0.0 │\n│      mmd │  domain │    cal_housing │          FluxModel │             ClapROAR │          0.0 │\n│      mmd │  domain │    cal_housing │          FluxModel │        Gravitational │          0.0 │\n│      mmd │  domain │    cal_housing │          FluxModel │              Generic │          0.0 │\n│      mmd │  domain │    cal_housing │ LogisticRegression │               Latent │          0.2 │\n│      mmd │  domain │    cal_housing │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │  domain │    cal_housing │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │  domain │    cal_housing │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │  domain │    cal_housing │ LogisticRegression │              Generic │          0.0 │\n│      mmd │  domain │           gmsc │       FluxEnsemble │               Latent │          0.0 │\n│      mmd │  domain │           gmsc │       FluxEnsemble │ Generic_conservative │          0.0 │\n│      mmd │  domain │           gmsc │       FluxEnsemble │             ClapROAR │          0.0 │\n│      mmd │  domain │           gmsc │       FluxEnsemble │        Gravitational │          0.0 │\n│      mmd │  domain │           gmsc │       FluxEnsemble │              Generic │          0.0 │\n│      mmd │  domain │           gmsc │          FluxModel │               Latent │          0.0 │\n│      mmd │  domain │           gmsc │          FluxModel │ Generic_conservative │          0.0 │\n│      mmd │  domain │           gmsc │          FluxModel │             ClapROAR │          0.0 │\n│      mmd │  domain │           gmsc │          FluxModel │        Gravitational │          0.0 │\n│      mmd │  domain │           gmsc │          FluxModel │              Generic │          0.0 │\n│      mmd │  domain │           gmsc │ LogisticRegression │               Latent │          0.0 │\n│      mmd │  domain │           gmsc │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │  domain │           gmsc │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │  domain │           gmsc │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │  domain │           gmsc │ LogisticRegression │              Generic │          0.0 │\n│      mmd │   model │ credit_default │       FluxEnsemble │               Latent │          0.0 │\n│      mmd │   model │ credit_default │       FluxEnsemble │ Generic_conservative │          0.0 │\n│      mmd │   model │ credit_default │       FluxEnsemble │             ClapROAR │          0.0 │\n│      mmd │   model │ credit_default │       FluxEnsemble │        Gravitational │          0.0 │\n│      mmd │   model │ credit_default │       FluxEnsemble │              Generic │          0.0 │\n│      mmd │   model │ credit_default │          FluxModel │               Latent │          0.0 │\n│      mmd │   model │ credit_default │          FluxModel │ Generic_conservative │          0.0 │\n│      mmd │   model │ credit_default │          FluxModel │             ClapROAR │          0.0 │\n│      mmd │   model │ credit_default │          FluxModel │        Gravitational │          0.0 │\n│      mmd │   model │ credit_default │          FluxModel │              Generic │          0.0 │\n│      mmd │   model │ credit_default │ LogisticRegression │               Latent │          0.0 │\n│      mmd │   model │ credit_default │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │   model │ credit_default │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │   model │ credit_default │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │   model │ credit_default │ LogisticRegression │              Generic │          0.0 │\n│      mmd │   model │    cal_housing │       FluxEnsemble │               Latent │          0.0 │\n│      mmd │   model │    cal_housing │       FluxEnsemble │ Generic_conservative │          0.0 │\n│      mmd │   model │    cal_housing │       FluxEnsemble │             ClapROAR │          0.0 │\n│      mmd │   model │    cal_housing │       FluxEnsemble │        Gravitational │          0.0 │\n│      mmd │   model │    cal_housing │       FluxEnsemble │              Generic │          0.0 │\n│      mmd │   model │    cal_housing │          FluxModel │               Latent │          0.0 │\n│      mmd │   model │    cal_housing │          FluxModel │ Generic_conservative │          0.0 │\n│      mmd │   model │    cal_housing │          FluxModel │             ClapROAR │          0.0 │\n│      mmd │   model │    cal_housing │          FluxModel │        Gravitational │          0.0 │\n│      mmd │   model │    cal_housing │          FluxModel │              Generic │          0.0 │\n│      mmd │   model │    cal_housing │ LogisticRegression │               Latent │          0.0 │\n│      mmd │   model │    cal_housing │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │   model │    cal_housing │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │   model │    cal_housing │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │   model │    cal_housing │ LogisticRegression │              Generic │          0.0 │\n│      mmd │   model │           gmsc │       FluxEnsemble │               Latent │          0.0 │\n│      mmd │   model │           gmsc │       FluxEnsemble │ Generic_conservative │          0.0 │\n│      mmd │   model │           gmsc │       FluxEnsemble │             ClapROAR │          0.0 │\n│      mmd │   model │           gmsc │       FluxEnsemble │        Gravitational │          0.0 │\n│      mmd │   model │           gmsc │       FluxEnsemble │              Generic │          0.0 │\n│      mmd │   model │           gmsc │          FluxModel │               Latent │          0.0 │\n│      mmd │   model │           gmsc │          FluxModel │ Generic_conservative │          0.0 │\n│      mmd │   model │           gmsc │          FluxModel │             ClapROAR │          0.0 │\n│      mmd │   model │           gmsc │          FluxModel │        Gravitational │          0.0 │\n│      mmd │   model │           gmsc │          FluxModel │              Generic │          0.0 │\n│      mmd │   model │           gmsc │ LogisticRegression │               Latent │          0.0 │\n│      mmd │   model │           gmsc │ LogisticRegression │ Generic_conservative │          0.0 │\n│      mmd │   model │           gmsc │ LogisticRegression │             ClapROAR │          0.0 │\n│      mmd │   model │           gmsc │ LogisticRegression │        Gravitational │          0.0 │\n│      mmd │   model │           gmsc │ LogisticRegression │              Generic │          0.0 │\n│ mmd_grid │   model │ credit_default │       FluxEnsemble │               Latent │          0.0 │\n│ mmd_grid │   model │ credit_default │       FluxEnsemble │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │ credit_default │       FluxEnsemble │             ClapROAR │          0.0 │\n│ mmd_grid │   model │ credit_default │       FluxEnsemble │        Gravitational │          0.0 │\n│ mmd_grid │   model │ credit_default │       FluxEnsemble │              Generic │          0.0 │\n│ mmd_grid │   model │ credit_default │          FluxModel │               Latent │          0.0 │\n│ mmd_grid │   model │ credit_default │          FluxModel │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │ credit_default │          FluxModel │             ClapROAR │          0.0 │\n│ mmd_grid │   model │ credit_default │          FluxModel │        Gravitational │          0.0 │\n│ mmd_grid │   model │ credit_default │          FluxModel │              Generic │          0.0 │\n│ mmd_grid │   model │ credit_default │ LogisticRegression │               Latent │          0.0 │\n│ mmd_grid │   model │ credit_default │ LogisticRegression │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │ credit_default │ LogisticRegression │             ClapROAR │          0.0 │\n│ mmd_grid │   model │ credit_default │ LogisticRegression │        Gravitational │          0.0 │\n│ mmd_grid │   model │ credit_default │ LogisticRegression │              Generic │          0.0 │\n│ mmd_grid │   model │    cal_housing │       FluxEnsemble │               Latent │          0.0 │\n│ mmd_grid │   model │    cal_housing │       FluxEnsemble │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │    cal_housing │       FluxEnsemble │             ClapROAR │          0.0 │\n│ mmd_grid │   model │    cal_housing │       FluxEnsemble │        Gravitational │          0.0 │\n│ mmd_grid │   model │    cal_housing │       FluxEnsemble │              Generic │          0.0 │\n│ mmd_grid │   model │    cal_housing │          FluxModel │               Latent │          0.0 │\n│ mmd_grid │   model │    cal_housing │          FluxModel │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │    cal_housing │          FluxModel │             ClapROAR │          0.0 │\n│ mmd_grid │   model │    cal_housing │          FluxModel │        Gravitational │          0.0 │\n│ mmd_grid │   model │    cal_housing │          FluxModel │              Generic │          0.0 │\n│ mmd_grid │   model │    cal_housing │ LogisticRegression │               Latent │          0.0 │\n│ mmd_grid │   model │    cal_housing │ LogisticRegression │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │    cal_housing │ LogisticRegression │             ClapROAR │          0.0 │\n│ mmd_grid │   model │    cal_housing │ LogisticRegression │        Gravitational │          0.0 │\n│ mmd_grid │   model │    cal_housing │ LogisticRegression │              Generic │          0.0 │\n│ mmd_grid │   model │           gmsc │       FluxEnsemble │               Latent │        0.016 │\n│ mmd_grid │   model │           gmsc │       FluxEnsemble │ Generic_conservative │        0.008 │\n│ mmd_grid │   model │           gmsc │       FluxEnsemble │             ClapROAR │          0.0 │\n│ mmd_grid │   model │           gmsc │       FluxEnsemble │        Gravitational │          0.0 │\n│ mmd_grid │   model │           gmsc │       FluxEnsemble │              Generic │          0.0 │\n│ mmd_grid │   model │           gmsc │          FluxModel │               Latent │        0.072 │\n│ mmd_grid │   model │           gmsc │          FluxModel │ Generic_conservative │          0.2 │\n│ mmd_grid │   model │           gmsc │          FluxModel │             ClapROAR │        0.244 │\n│ mmd_grid │   model │           gmsc │          FluxModel │        Gravitational │        0.056 │\n│ mmd_grid │   model │           gmsc │          FluxModel │              Generic │         0.23 │\n│ mmd_grid │   model │           gmsc │ LogisticRegression │               Latent │          0.0 │\n│ mmd_grid │   model │           gmsc │ LogisticRegression │ Generic_conservative │          0.0 │\n│ mmd_grid │   model │           gmsc │ LogisticRegression │             ClapROAR │          0.0 │\n│ mmd_grid │   model │           gmsc │ LogisticRegression │        Gravitational │          0.0 │\n│ mmd_grid │   model │           gmsc │ LogisticRegression │              Generic │          0.0 │\n└──────────┴─────────┴────────────────┴────────────────────┴──────────────────────┴──────────────┘\n\n\n\n\n6.9.4 Chart in paper\nFigure 6.9 shows the chart that went into the paper.\n\nImages.load(joinpath(www_artifact_path,\"paper_real_world_results.png\"))\n\n\n\nFigure 6.9: Chart in paper"
  },
  {
    "objectID": "sections/generators/index.html",
    "href": "sections/generators/index.html",
    "title": "4  Generators",
    "section": "",
    "text": "The generators have been moved into CounterfactualExplanations.jl. For more information so the package documentation."
  }
]