search.json 108.75 KiB
[
{
"objectID": "index.html",
"href": "index.html",
"title": "Endogenous Macrodynamics in Algorithmic Recourse",
"section": "",
"text": "Preface\nThis is an online auxiliary companion to our research paper: Endogenous Macrodynamics in Algorithmic Recourse."
},
{
"objectID": "intro.html#data",
"href": "intro.html#data",
"title": "1 Introduction",
"section": "1.1 Data",
"text": "1.1 Data\nWe begin by generating the synthetic data for a simple binary classification problem. For illustrative purposes we will use data that is linearly separable. The chart below shows the data \\(\\mathcal{D}\\) at time zero, before any implementation of recourse.\n\nN = 1000\nxmax = 2\nX, ys = make_blobs(\n N, 2; \n centers=2, as_table=false, center_box=(-xmax => xmax), cluster_std=0.1\n)\nys .= ys.==2\nX = X'\nxs = Flux.unstack(X,2)\ndata = zip(xs,ys)\ncounterfactual_data = CounterfactualData(X,ys')\nplot()\nscatter!(counterfactual_data)\n\n\n\n\nFigure 1.1: Linearly separable synthetic data"
},
{
"objectID": "intro.html#classifier",
"href": "intro.html#classifier",
"title": "1 Introduction",
"section": "1.2 Classifier",
"text": "1.2 Classifier\nTo model this data \\(\\mathcal{D}\\) we will use a linear classifier. In particular, as in the paper, we will build a logistic regression model in Flux.jl: a single layer with sigmoid activation.\n\nn_epochs = 100\nmodel = Chain(Dense(2,1))\nmod = FluxModel(model)\nModels.train(mod, counterfactual_data; n_epochs=n_epochs)\n\nFigure 1.2 below shows the linear separation of the two classes.\n\nplt_original = plot(mod, counterfactual_data; zoom=0, colorbar=false, title=\"(a)\")\ndisplay(plt_original)\n\n\n\n\nFigure 1.2: The baseline model: contours indicate the predicted label; dots indicate observed data points."
},
{
"objectID": "intro.html#implementation-of-recourse",
"href": "intro.html#implementation-of-recourse",
"title": "1 Introduction",
"section": "1.3 Implementation of Recourse",
"text": "1.3 Implementation of Recourse"
},
{
"objectID": "intro.html#generate-counterfactual",
"href": "intro.html#generate-counterfactual",
"title": "1 Introduction",
"section": "1.4 Generate counterfactual",
"text": "1.4 Generate counterfactual\n\nγ = 0.50\nμ = 0.10\nMarkdown.parse(\n \"\"\"\n To generate counterfactual explanations we will rely on the most generic approach. As our decision threshold we will use $(γ*100)% here. In other words, the counterfactual is considered as valid, as soon as the classifier is more convinced that it belongs to the target class (blue) than the non-target class (orange). In each round we will implement recourse for $(μ * 100)% of the individuals in the non-target class. \n \"\"\"\n)\n\n\nopt = Flux.Adam(0.01)\ngen = GenericGenerator(;decision_threshold=γ, opt=opt)\n\nFigure 1.3 below shows the recourse outcome, which we denote here as \\(\\mathcal{D}^{\\prime}\\). The obvious observation at this point is that the resulting counterfactuals, while valid, are clearly distinguishable from the factuals that were always in the target class. This is not a new observation and nor is it entirely surprising. In fact, a lot of recent work in this field has tried to address this issue. In this work we wonder what happens when we let these sorts of dynamics play out further in practice. While the outcome in (b) is not surprising, it may be much harder to observe so clearly it in practice (when the data is more complex).\n\ncandidates = findall(ys.==0)\nchosen_individuals = rand(candidates, Int(round(μ*length(candidates))))\nX′ = copy(X)\ny′ = copy(ys)\nfactuals = select_factual(counterfactual_data,chosen_individuals)\noutcome = generate_counterfactual(factuals, 1, counterfactual_data, mod, gen; initialization=:identity)\nX′[:,chosen_individuals] = reduce(hcat, @.(selectdim(counterfactual(outcome), 3, 1)))\ny′[chosen_individuals] = reduce(vcat,@.(selectdim(counterfactual_label(outcome),3,1)))\ncounterfactual_data′ = CounterfactualData(X′,y′')\nplt_single = plot(mod,counterfactual_data′;zoom=0,colorbar=false,title=\"(b)\")\ndisplay(plt_single)\n\n\n\n\nFigure 1.3: The recourse outcome after one round."
},
{
"objectID": "intro.html#retrain",
"href": "intro.html#retrain",
"title": "1 Introduction",
"section": "1.5 Retrain",
"text": "1.5 Retrain\nSuppose the agent in charge of the black-box system has provided recourse to a share of individuals leading to the outcome in Figure 1.3. In practice, models are regularly updated through retraining to account for concept drift, for example. For our experiments, we assume that the agent accepts \\(\\mathcal{D}^{\\prime}\\) as its new ground truth. To isolate the endogenous effects we are interested in here from any other effect, we further assume away any exogenous changes to the data that we might expect to occur in practice. Retraining the model on \\(\\mathcal{D}^{\\prime}\\) leads to a shift of the decision boundary in the direction of the non-target class (Figure 1.4).\n\nmod = Models.train(mod, counterfactual_data′)\nplt_single_retrained = plot(mod,counterfactual_data′;zoom=0,colorbar=false,title=\"(c)\")\ndisplay(plt_single_retrained)\n\n\n\n\nFigure 1.4: The retrained model."
},
{
"objectID": "intro.html#repeat",
"href": "intro.html#repeat",
"title": "1 Introduction",
"section": "1.6 Repeat",
"text": "1.6 Repeat\nWe finally go on to repeat this process of recourse followed by model updates for multiple round. Figure 1.5 below presents the different stages of the experiment side-by-side, where panel (d) represents the outcome after ten rounds.\nAt first glance it seems that costs to individuals seeking recourse are gradually reduced as the decision boundary moves into the direction of the non-target class: they need to exert less effort to move to valid counterfactual states. The problem with this idea is, of course, that there is no free lunch. This reduction inflicts a burden on the agent in charge of the black-box: the group of individuals that is now classified as target class individuals looks entirely different from the original group.\nWhy is this a problem? Let’s, for example, that the two synthetic features accurately describe the credit worthiness of individual seeking loans, where credit-worthiness increases in the South-West direction. Non-target class individuals (orange) are denied credit, while target class individuals (blue) receive a loan. Then the population of borrowers in (d) is much more risky than in (a). Clearly, any lender (bank) aware of such dynamics would avoid them in practice. They might choose not to offer recourse in the first place, generating a cost to all individuals seeking recourse. Alternatively, they may reward first movers, but stop offering recourse after a few rounds.\nThis last point makes it clear that the implementation of recourse by one individual may generate external costs for other individuals. This notion motivates the ideas set out in the paper.\n\ni = 2\nwhile i <= 10\n counterfactual_data′ = CounterfactualData(X′,y′')\n candidates = findall(y′.==0)\n chosen_individuals = rand(candidates, Int(round(μ*length(candidates))))\n Models.train(mod, counterfactual_data′)\n factuals = select_factual(counterfactual_data′,chosen_individuals)\n outcome = generate_counterfactual(factuals, 1, counterfactual_data′, mod, gen; initialization=:identity)\n X′[:,chosen_individuals] = reduce(hcat, @.(selectdim(counterfactual(outcome), 3, 1)))\n y′[chosen_individuals] = reduce(vcat,@.(selectdim(counterfactual_label(outcome),3,1)))\n i += 1\nend\nplt_single_repeat = plot(mod,counterfactual_data′;zoom=0,colorbar=false,title=\"(d)\")\n\n\nplt = plot(plt_original, plt_single, plt_single_retrained, plt_single_repeat, layout=(1,4), legend=false, axis=nothing, size=(600,165))\nsavefig(plt, joinpath(www_path, \"poc.png\"))\ndisplay(plt)\n\n\n\n\nFigure 1.5: The different stages of the experiment."
},
{
"objectID": "sections/data_preprocessing/index.html#generating-synthetic-data",
"href": "sections/data_preprocessing/index.html#generating-synthetic-data",
"title": "2 Data Preprocessing",
"section": "2.1 Generating Synthetic Data",
"text": "2.1 Generating Synthetic Data\n\nn = 1000\np = 2\n\n# Linearly separable:\nX, y = make_blobs(n, p; centers=2, center_box=(-2 => 2), cluster_std=0.1)\ndf = DataFrame(X)\ndf.target .= ifelse.(y.==1,0,1)\nCSV.write(joinpath(data_path, \"linearly_separable.csv\"),df)\n\n# Overlapping:\nX, y = make_blobs(n, p; centers=2, center_box=(-2 => 2), cluster_std=0.5)\ndf = DataFrame(X)\ndf.target .= ifelse.(y.==1,0,1)\nCSV.write(joinpath(data_path, \"overlapping.csv\"),df)\n\n# Circles:\nX, y = make_circles(n; noise=0.15, factor=0.01)\ndf = DataFrame(X)\ndf.target = y\nCSV.write(joinpath(data_path, \"circles.csv\"),df)\n\n# Moon:\nX, y = make_moons(n)\ndf = DataFrame(X)\ndf.target = y\nCSV.write(joinpath(data_path, \"moons.csv\"),df)\n\n\ngenerate_artifacts(data_path)\n\n\ncatalogue = load_synthetic()\nfunction plot_data(data,title)\n plt = plot(title=uppercasefirst(replace(string(title),\"_\" => \" \")))\n scatter!(data)\n return plt\nend\nplts = [plot_data(data,name) for (name, data) in catalogue]\nplt = plot(plts..., layout=(1,4), size=(850,200))\nsavefig(plt, \"paper/www/synthetic_data.png\")\ndisplay(plt)"
},
{
"objectID": "sections/data_preprocessing/index.html#preprocessing-real-world-data",
"href": "sections/data_preprocessing/index.html#preprocessing-real-world-data",
"title": "2 Data Preprocessing",
"section": "2.2 Preprocessing Real-World Data",
"text": "2.2 Preprocessing Real-World Data\n\n2.2.1 California Housing Data\nFetching the data using Python’s sklearn (run this in the Python REPL):\nfrom sklearn.datasets import fetch_california_housing\ndf, y = fetch_california_housing(return_X_y=True, as_frame=True)\ndf[\"target\"] = y.values\ndata_path = \"dev/artifacts/upload/data/real_world\"\nimport os\nif not os.path.isdir(os.path.join(data_path,\"raw\")):\n os.makedirs(os.path.join(data_path,\"raw\"))\ndf.to_csv(os.path.join(data_path,\"raw/cal_housing.csv\"), index=False)\nLoading the data into Julia session.\n\ndf = CSV.read(joinpath(data_path, \"raw/cal_housing.csv\"), DataFrame)\n# Target:\ny = df.target\ny = Float64.(y .>= median(y)); # binary target (positive outcome)\n# Data:\ndf.target = y\n\nRandom undersampling to balance the data:\n\ndf_balanced = getobs(undersample(df, df.target; shuffle=true))[1]\n\nAll features are continuous:\n\nschema(df_balanced)\n\n\n┌────────────┬────────────┬─────────┐\n│ names │ scitypes │ types │\n├────────────┼────────────┼─────────┤\n│ MedInc │ Continuous │ Float64 │\n│ HouseAge │ Continuous │ Float64 │\n│ AveRooms │ Continuous │ Float64 │\n│ AveBedrms │ Continuous │ Float64 │\n│ Population │ Continuous │ Float64 │\n│ AveOccup │ Continuous │ Float64 │\n│ Latitude │ Continuous │ Float64 │\n│ Longitude │ Continuous │ Float64 │\n│ target │ Continuous │ Float64 │\n└────────────┴────────────┴─────────┘\n\n\n\n\nFeature transformation:\n\ntransformer = Standardizer(count=true)\nmach = MLJBase.fit!(machine(transformer, df_balanced[:,Not(:target)]))\nX = MLJBase.transform(mach, df_balanced[:,Not(:target)])\nschema(X)\n\n\n┌────────────┬────────────┬─────────┐\n│ names │ scitypes │ types │\n├────────────┼────────────┼─────────┤\n│ MedInc │ Continuous │ Float64 │\n│ HouseAge │ Continuous │ Float64 │\n│ AveRooms │ Continuous │ Float64 │\n│ AveBedrms │ Continuous │ Float64 │\n│ Population │ Continuous │ Float64 │\n│ AveOccup │ Continuous │ Float64 │\n│ Latitude │ Continuous │ Float64 │\n│ Longitude │ Continuous │ Float64 │\n└────────────┴────────────┴─────────┘\n\n\n\n\nTurning the data into CounterfactualData:\n\nX = Matrix(X)\nX = permutedims(X)\ny = permutedims(df_balanced.target)\ndata = CounterfactualData(X,y)\n\nSaving the data:\n\nCSV.write(joinpath(data_path, \"cal_housing.csv\"), df_balanced) # binary file\nSerialization.serialize(joinpath(data_path,\"cal_housing.jls\"), data) # CounterfactualData\n\n\n\n2.2.2 Give Me Some Credit\nLoading and basic preprocessing:\n\ndf = CSV.read(joinpath(data_path, \"raw/cs-training.csv\"), DataFrame)\nselect!(df, Not([:Column1]))\nrename!(df, :SeriousDlqin2yrs => :target)\nmapcols!(x -> [ifelse(x_==\"NA\", missing, x_) for x_ in x], df)\ndropmissing!(df)\nmapcols!(x -> eltype(x) <: AbstractString ? parse.(Int, x) : x, df)\ndf.target .= map(y -> y == 0 ? 1 : 0, df.target) # postive outcome = no delinquency\n\nBalancing:\n\ndf_balanced = getobs(undersample(df, df.target;shuffle=true))[1]\n\nAll features are continuous:\n\nschema(df_balanced)\n\nFeature transformation:\n\ntransformer = Standardizer(count=true)\nmach = MLJBase.fit!(machine(transformer, df_balanced[:,Not(:target)]))\nX = MLJBase.transform(mach, df_balanced[:,Not(:target)])\nschema(X)\n\nTurning the data into CounterfactualData:\n\nX = Matrix(X)\nX = permutedims(X)\ny = permutedims(df_balanced.target)\ndata = CounterfactualData(X,y)\n\nSaving:\n\nCSV.write(joinpath(data_path, \"gmsc.csv\"), df_balanced) # binary file\nSerialization.serialize(joinpath(data_path,\"gmsc.jls\"), data) # CounterfactualData\n\n\n\n2.2.3 UCI Credit Card Default\nLoading and basic preprocessing:\n\ndf = CSV.read(joinpath(data_path, \"raw/UCI_Credit_Card.csv\"), DataFrame)\nselect!(df, Not([:ID]))\nrename!(df, \"default.payment.next.month\" => :target)\ndropmissing!(df)\ndf.SEX = categorical(df.SEX)\ndf.EDUCATION = categorical(df.EDUCATION)\ndf.MARRIAGE = categorical(df.MARRIAGE)\nmapcols!(x -> eltype(x) <: AbstractString ? parse.(Int, x) : x, df)\ndf.target .= map(y -> y == 0 ? 1 : 0, df.target) # postive outcome = no default\n\nBalancing:\n\ndf_balanced = getobs(undersample(df, df.target;shuffle=true))[1]\n\nNot all features are continuous:\n\nschema(df_balanced)\n\n\n┌───────────┬───────────────┬─────────────────────────────────┐\n│ names │ scitypes │ types │\n├───────────┼───────────────┼─────────────────────────────────┤\n│ LIMIT_BAL │ Continuous │ Float64 │\n│ SEX │ Multiclass{2} │ CategoricalValue{Int64, UInt32} │\n│ EDUCATION │ Multiclass{7} │ CategoricalValue{Int64, UInt32} │\n│ MARRIAGE │ Multiclass{4} │ CategoricalValue{Int64, UInt32} │\n│ AGE │ Count │ Int64 │\n│ PAY_0 │ Count │ Int64 │\n│ PAY_2 │ Count │ Int64 │\n│ PAY_3 │ Count │ Int64 │\n│ PAY_4 │ Count │ Int64 │\n│ PAY_5 │ Count │ Int64 │\n│ PAY_6 │ Count │ Int64 │\n│ BILL_AMT1 │ Continuous │ Float64 │\n│ BILL_AMT2 │ Continuous │ Float64 │\n│ BILL_AMT3 │ Continuous │ Float64 │\n│ BILL_AMT4 │ Continuous │ Float64 │\n│ BILL_AMT5 │ Continuous │ Float64 │\n│ BILL_AMT6 │ Continuous │ Float64 │\n│ PAY_AMT1 │ Continuous │ Float64 │\n│ PAY_AMT2 │ Continuous │ Float64 │\n│ PAY_AMT3 │ Continuous │ Float64 │\n│ PAY_AMT4 │ Continuous │ Float64 │\n│ PAY_AMT5 │ Continuous │ Float64 │\n│ PAY_AMT6 │ Continuous │ Float64 │\n│ target │ Count │ Int64 │\n└───────────┴───────────────┴─────────────────────────────────┘\n\n\n\n\nFeature transformation:\n\ntransformer = Standardizer(count=true) |> ContinuousEncoder()\nmach = MLJBase.fit!(machine(transformer, df_balanced[:,Not(:target)]))\nX = MLJBase.transform(mach, df_balanced[:,Not(:target)])\nschema(X)\n\n\n┌──────────────┬────────────┬─────────┐\n│ names │ scitypes │ types │\n├──────────────┼────────────┼─────────┤\n│ LIMIT_BAL │ Continuous │ Float64 │\n│ SEX__1 │ Continuous │ Float64 │\n│ SEX__2 │ Continuous │ Float64 │\n│ EDUCATION__0 │ Continuous │ Float64 │\n│ EDUCATION__1 │ Continuous │ Float64 │\n│ EDUCATION__2 │ Continuous │ Float64 │\n│ EDUCATION__3 │ Continuous │ Float64 │\n│ EDUCATION__4 │ Continuous │ Float64 │\n│ EDUCATION__5 │ Continuous │ Float64 │\n│ EDUCATION__6 │ Continuous │ Float64 │\n│ MARRIAGE__0 │ Continuous │ Float64 │\n│ MARRIAGE__1 │ Continuous │ Float64 │\n│ MARRIAGE__2 │ Continuous │ Float64 │\n│ MARRIAGE__3 │ Continuous │ Float64 │\n│ AGE │ Continuous │ Float64 │\n│ PAY_0 │ Continuous │ Float64 │\n│ PAY_2 │ Continuous │ Float64 │\n│ PAY_3 │ Continuous │ Float64 │\n│ PAY_4 │ Continuous │ Float64 │\n│ PAY_5 │ Continuous │ Float64 │\n│ PAY_6 │ Continuous │ Float64 │\n│ BILL_AMT1 │ Continuous │ Float64 │\n│ ⋮ │ ⋮ │ ⋮ │\n└──────────────┴────────────┴─────────┘\n 11 rows omitted\n\n\n\n\nCategorical indices:\n\nfeatures_categorical = [\n [2,3],\n collect(4:10),\n collect(11:14)\n]\n\nPreparing for use with CounterfactualExplanations.jl:\n\nX = Matrix(X)\nX = permutedims(X)\ny = permutedims(df_balanced.target)\ndata = CounterfactualData(\n X, y;\n features_categorical = features_categorical\n)\n\nSaving:\n\nCSV.write(joinpath(data_path, \"credit_default.csv\"), df_balanced) # binary file\nSerialization.serialize(joinpath(data_path,\"credit_default.jls\"), data) # CounterfactualData\n\n\ngenerate_artifacts(data_path)"
},
{
"objectID": "sections/experiments/index.html",
"href": "sections/experiments/index.html",
"title": "3 Experimental Results",
"section": "",
"text": "4 Synthetic Data\nThis notebook was used to run the experiments for the synthetic datasets and can be used to reproduce the results in the paper. In the following we first run the experiments and then generate visualizations and tables."
},
{
"objectID": "sections/experiments/index.html#experiment",
"href": "sections/experiments/index.html#experiment",
"title": "3 Experimental Results",
"section": "4.1 Experiment",
"text": "4.1 Experiment\n\nmodels = [\n :LogisticRegression, \n :FluxModel, \n :FluxEnsemble,\n]\nopt = Flux.Descent(0.01) \ngenerators = Dict(\n :Greedy=>GreedyGenerator(), \n :Generic=>GenericGenerator(opt = opt),\n :REVISE=>REVISEGenerator(opt = opt),\n :DICE=>DiCEGenerator(opt = opt),\n)\n\n\nmax_obs = 1000\ncatalogue = load_synthetic(max_obs)\nchoices = [\n :linearly_separable, \n :overlapping, \n :circles, \n :moons,\n]\ndata_sets = filter(p -> p[1] in choices, catalogue)\n\n\nexperiments = set_up_experiments(data_sets,models,generators)\n\n\nplts = []\nfor (exp_name, exp_) in experiments\n for (M_name, M) in exp_.models\n score = round(model_evaluation(M, exp_.test_data),digits=2)\n plt = plot(M, exp_.test_data, title=\"$exp_name;\\n $M_name ($score)\")\n # Errors:\n ids = findall(vec(round.(probs(M, exp_.test_data.X)) .!= exp_.test_data.y))\n x_wrongly_labelled = exp_.test_data.X[:,ids]\n scatter!(plt, x_wrongly_labelled[1,:], x_wrongly_labelled[2,:], ms=7.5, color=:red, label=\"\")\n plts = vcat(plts..., plt)\n end\nend\nplt = plot(plts..., layout=(length(choices),length(models)),size=(length(choices)*300,length(models)*300))\nsavefig(plt, joinpath(www_path,\"models_test_before.png\"))\n\n\nusing AlgorithmicRecourseDynamics.Models: model_evaluation\nplts = []\nfor (exp_name, exp_) in experiments\n for (M_name, M) in exp_.models\n score = round(model_evaluation(M, exp_.train_data),digits=2)\n plt = plot(M, exp_.train_data, title=\"$exp_name;\\n $M_name ($score)\")\n # Errors:\n ids = findall(vec(round.(probs(M, exp_.train_data.X)) .!= exp_.train_data.y))\n x_wrongly_labelled = exp_.train_data.X[:,ids]\n scatter!(plt, x_wrongly_labelled[1,:], x_wrongly_labelled[2,:], ms=7.5, color=:red, label=\"\")\n plts = vcat(plts..., plt)\n end\nend\nplt = plot(plts..., layout=(length(choices),length(models)),size=(length(choices)*300,length(models)*300))\nsavefig(plt, joinpath(www_path,\"models_train_before.png\"))\n\n\nn_evals = 5\nn_rounds = 50\nevaluate_every = Int(round(n_rounds/n_evals))\nn_folds = 5\nT = 100\nresults = run_experiments(\n experiments;\n save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, T=T\n)\nSerialization.serialize(joinpath(output_path,\"results.jls\"),results)\n\n\nplot_dict = Dict(key => Dict() for (key,val) in results)\nfold = 1\nfor (name, res) in results\n exp_ = res.experiment\n plot_dict[name] = Dict(key => [] for (key,val) in exp_.generators)\n rec_sys = exp_.recourse_systems[fold]\n sys_ids = collect(exp_.system_identifiers)\n M = length(rec_sys)\n for m in 1:M\n model_name, generator_name = sys_ids[m]\n M = rec_sys[m].model\n score = round(model_evaluation(M, exp_.test_data),digits=2)\n plt = plot(M, exp_.test_data, title=\"$name;\\n $model_name ($score)\")\n # Errors:\n ids = findall(vec(round.(probs(M, exp_.test_data.X)) .!= exp_.test_data.y))\n x_wrongly_labelled = exp_.test_data.X[:,ids]\n scatter!(plt, x_wrongly_labelled[1,:], x_wrongly_labelled[2,:], ms=7.5, color=:red, label=\"\")\n plot_dict[name][generator_name] = vcat(plot_dict[name][generator_name], plt)\n end\nend\nplot_dict = Dict(key => reduce(vcat, [plots[key] for plots in values(plot_dict)]) for (key, value) in generators)\nfor (name, plts) in plot_dict\n plt = plot(plts..., layout=(length(choices),length(models)),size=(length(choices)*300,length(models)*300))\n savefig(plt, joinpath(www_path,\"models_test_after_$(name).png\"))\nend\n\n\nusing AlgorithmicRecourseDynamics.Models: model_evaluation\nplot_dict = Dict(key => Dict() for (key,val) in results)\nfold = 1\nfor (name, res) in results\n exp_ = res.experiment\n plot_dict[name] = Dict(key => [] for (key,val) in exp_.generators)\n rec_sys = exp_.recourse_systems[fold]\n sys_ids = collect(exp_.system_identifiers)\n M = length(rec_sys)\n for m in 1:M\n model_name, generator_name = sys_ids[m]\n M = rec_sys[m].model\n data = rec_sys[m].data\n score = round(model_evaluation(M, data),digits=2)\n plt = plot(M, data, title=\"$name;\\n $model_name ($score)\")\n # Errors:\n ids = findall(vec(round.(probs(M, data.X)) .!= data.y))\n x_wrongly_labelled = data.X[:,ids]\n scatter!(plt, x_wrongly_labelled[1,:], x_wrongly_labelled[2,:], ms=7.5, color=:red, label=\"\")\n plot_dict[name][generator_name] = vcat(plot_dict[name][generator_name], plt)\n end\nend\nplot_dict = Dict(key => reduce(vcat, [plots[key] for plots in values(plot_dict)]) for (key, value) in generators)\nfor (name, plts) in plot_dict\n plt = plot(plts..., layout=(length(choices),length(models)),size=(length(choices)*300,length(models)*300))\n savefig(plt, joinpath(www_path,\"models_train_after_$(name).png\"))\nend"
},
{
"objectID": "sections/experiments/index.html#plots",
"href": "sections/experiments/index.html#plots",
"title": "3 Experimental Results",
"section": "4.2 Plots",
"text": "4.2 Plots\n\nresults = Serialization.deserialize(joinpath(output_path,\"results.jls\"));\n\n\nusing Images\nline_charts = Dict()\nerrorbar_charts = Dict()\nfor (data_name, res) in results\n plt = plot(res)\n Images.save(joinpath(www_path, \"line_chart_$(data_name).png\"), plt)\n line_charts[data_name] = plt\n plt = plot(res,maximum(res.output.n))\n Images.save(joinpath(www_path, \"errorbar_chart_$(data_name).png\"), plt)\n errorbar_charts[data_name] = plt\nend\n\n\n4.2.1 Line Charts\nFigure 4.1 shows the evolution of the evaluation metrics over the course of the experiment.\n\n\n\n\n\n\n(a) Circles\n\n\n\n\n\n\n\n(b) Linearly Separable\n\n\n\n\n\n\n\n(c) Moons\n\n\n\n\n\n\n\n(d) Overlapping\n\n\n\nFigure 4.1: Line Charts\n\n\n\n\n4.2.2 Error Bar Charts\nFigure 4.2 shows the evaluation metrics at the end of the experiments.\n\n\n\n\n\n\n(a) Circles\n\n\n\n\n\n\n\n(b) Linearly Separable\n\n\n\n\n\n\n\n(c) Moons\n\n\n\n\n\n\n\n(d) Overlapping\n\n\n\nFigure 4.2: Error Bar Charts"
},
{
"objectID": "sections/experiments/index.html#bootstrap",
"href": "sections/experiments/index.html#bootstrap",
"title": "3 Experimental Results",
"section": "4.3 Bootstrap",
"text": "4.3 Bootstrap\n\nn_bootstrap = 100\ndf = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,\"bootstrap.csv\"))\n\n\n\n┌──────────┬─────────┬────────────────────┬────────────────────┬───────────┬──────────────┐\n│ name │ scope │ data │ model │ generator │ p_value_mean │\n│ String31 │ String7 │ String31 │ String31 │ String7 │ Float64 │\n├──────────┼─────────┼────────────────────┼────────────────────┼───────────┼──────────────┤\n│ mmd │ domain │ overlapping │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ domain │ overlapping │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ domain │ overlapping │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ domain │ overlapping │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxModel │ Greedy │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxModel │ REVISE │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxModel │ DICE │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd │ domain │ linearly_separable │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ domain │ linearly_separable │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ domain │ linearly_separable │ LogisticRegression │ REVISE │ 0.238 │\n│ mmd │ domain │ linearly_separable │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxModel │ Greedy │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxModel │ REVISE │ 0.188 │\n│ mmd │ domain │ linearly_separable │ FluxModel │ DICE │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxEnsemble │ REVISE │ 0.158 │\n│ mmd │ domain │ linearly_separable │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd │ domain │ circles │ LogisticRegression │ Generic │ 0.104 │\n│ mmd │ domain │ circles │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ domain │ circles │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ domain │ circles │ LogisticRegression │ DICE │ 0.1275 │\n│ mmd │ domain │ circles │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ circles │ FluxModel │ Greedy │ 0.0 │\n│ mmd │ domain │ circles │ FluxModel │ REVISE │ 0.996 │\n│ mmd │ domain │ circles │ FluxModel │ DICE │ 0.0 │\n│ mmd │ domain │ circles │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ circles │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd │ domain │ circles │ FluxEnsemble │ REVISE │ 1.0 │\n│ mmd │ domain │ circles │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd │ domain │ moons │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ domain │ moons │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ domain │ moons │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ domain │ moons │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ domain │ moons │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ moons │ FluxModel │ Greedy │ 0.0 │\n│ mmd │ domain │ moons │ FluxModel │ REVISE │ 0.0 │\n│ mmd │ domain │ moons │ FluxModel │ DICE │ 0.0 │\n│ mmd │ domain │ moons │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ moons │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd │ domain │ moons │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd │ domain │ moons │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd │ model │ overlapping │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ overlapping │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ model │ overlapping │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ model │ overlapping │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ model │ overlapping │ FluxModel │ Generic │ 0.0 │\n│ mmd │ model │ overlapping │ FluxModel │ Greedy │ 0.0 │\n│ mmd │ model │ overlapping │ FluxModel │ REVISE │ 0.0 │\n│ mmd │ model │ overlapping │ FluxModel │ DICE │ 0.0 │\n│ mmd │ model │ overlapping │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ model │ overlapping │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd │ model │ overlapping │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd │ model │ overlapping │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd │ model │ linearly_separable │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ linearly_separable │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ model │ linearly_separable │ LogisticRegression │ REVISE │ 0.084 │\n│ mmd │ model │ linearly_separable │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ model │ linearly_separable │ FluxModel │ Generic │ 0.87 │\n│ mmd │ model │ linearly_separable │ FluxModel │ Greedy │ 0.616 │\n│ mmd │ model │ linearly_separable │ FluxModel │ REVISE │ 0.65 │\n│ mmd │ model │ linearly_separable │ FluxModel │ DICE │ 0.82 │\n│ mmd │ model │ linearly_separable │ FluxEnsemble │ Generic │ 0.892 │\n│ mmd │ model │ linearly_separable │ FluxEnsemble │ Greedy │ 0.528 │\n│ mmd │ model │ linearly_separable │ FluxEnsemble │ REVISE │ 0.642 │\n│ mmd │ model │ linearly_separable │ FluxEnsemble │ DICE │ 0.878 │\n│ mmd │ model │ circles │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ circles │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ model │ circles │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ model │ circles │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ model │ circles │ FluxModel │ Generic │ 0.392 │\n│ mmd │ model │ circles │ FluxModel │ Greedy │ 0.078 │\n│ mmd │ model │ circles │ FluxModel │ REVISE │ 0.836 │\n│ mmd │ model │ circles │ FluxModel │ DICE │ 0.516 │\n│ mmd │ model │ circles │ FluxEnsemble │ Generic │ 0.456 │\n│ mmd │ model │ circles │ FluxEnsemble │ Greedy │ 0.16 │\n│ mmd │ model │ circles │ FluxEnsemble │ REVISE │ 0.872 │\n│ mmd │ model │ circles │ FluxEnsemble │ DICE │ 0.572 │\n│ mmd │ model │ moons │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ moons │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ model │ moons │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ model │ moons │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ model │ moons │ FluxModel │ Generic │ 0.716 │\n│ mmd │ model │ moons │ FluxModel │ Greedy │ 0.17 │\n│ mmd │ model │ moons │ FluxModel │ REVISE │ 0.67 │\n│ mmd │ model │ moons │ FluxModel │ DICE │ 0.588 │\n│ mmd │ model │ moons │ FluxEnsemble │ Generic │ 0.634 │\n│ mmd │ model │ moons │ FluxEnsemble │ Greedy │ 0.092 │\n│ mmd │ model │ moons │ FluxEnsemble │ REVISE │ 0.75 │\n│ mmd │ model │ moons │ FluxEnsemble │ DICE │ 0.61 │\n│ mmd_grid │ model │ overlapping │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ overlapping │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd_grid │ model │ overlapping │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd_grid │ model │ overlapping │ LogisticRegression │ DICE │ 0.0 │\n│ mmd_grid │ model │ overlapping │ FluxModel │ Generic │ 0.0 │\n│ mmd_grid │ model │ overlapping │ FluxModel │ Greedy │ 0.0 │\n│ mmd_grid │ model │ overlapping │ FluxModel │ REVISE │ 0.004 │\n│ mmd_grid │ model │ overlapping │ FluxModel │ DICE │ 0.0 │\n│ mmd_grid │ model │ overlapping │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ overlapping │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd_grid │ model │ overlapping │ FluxEnsemble │ REVISE │ 0.024 │\n│ mmd_grid │ model │ overlapping │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ LogisticRegression │ DICE │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxModel │ Generic │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxModel │ Greedy │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxModel │ REVISE │ 0.018 │\n│ mmd_grid │ model │ linearly_separable │ FluxModel │ DICE │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxEnsemble │ REVISE │ 0.014 │\n│ mmd_grid │ model │ linearly_separable │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd_grid │ model │ circles │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ circles │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd_grid │ model │ circles │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd_grid │ model │ circles │ LogisticRegression │ DICE │ 0.0 │\n│ mmd_grid │ model │ circles │ FluxModel │ Generic │ 0.0 │\n│ mmd_grid │ model │ circles │ FluxModel │ Greedy │ 0.0 │\n│ mmd_grid │ model │ circles │ FluxModel │ REVISE │ 0.152 │\n│ mmd_grid │ model │ circles │ FluxModel │ DICE │ 0.004 │\n│ mmd_grid │ model │ circles │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ circles │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd_grid │ model │ circles │ FluxEnsemble │ REVISE │ 0.14 │\n│ mmd_grid │ model │ circles │ FluxEnsemble │ DICE │ 0.002 │\n│ mmd_grid │ model │ moons │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ moons │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd_grid │ model │ moons │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd_grid │ model │ moons │ LogisticRegression │ DICE │ 0.0 │\n│ mmd_grid │ model │ moons │ FluxModel │ Generic │ 0.004 │\n│ mmd_grid │ model │ moons │ FluxModel │ Greedy │ 0.0 │\n│ mmd_grid │ model │ moons │ FluxModel │ REVISE │ 0.074 │\n│ mmd_grid │ model │ moons │ FluxModel │ DICE │ 0.016 │\n│ mmd_grid │ model │ moons │ FluxEnsemble │ Generic │ 0.002 │\n│ mmd_grid │ model │ moons │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd_grid │ model │ moons │ FluxEnsemble │ REVISE │ 0.044 │\n│ mmd_grid │ model │ moons │ FluxEnsemble │ DICE │ 0.0 │\n└──────────┴─────────┴────────────────────┴────────────────────┴───────────┴──────────────┘"
},
{
"objectID": "sections/experiments/index.html#sec-app-synthetic-paper",
"href": "sections/experiments/index.html#sec-app-synthetic-paper",
"title": "3 Experimental Results",
"section": "4.4 Chart in paper",
"text": "4.4 Chart in paper\nFigure 4.3 shows the chart that went into the paper.\n\nImages.load(joinpath(www_artifact_path,\"paper_synthetic_results.png\"))\n\n\n\nFigure 4.3: Chart in paper\n\n\n\n\n# echo: false\n\ngenerate_artifacts(output_path)\ngenerate_artifacts(www_path)"
},
{
"objectID": "sections/experiments/index.html#experiment-1",
"href": "sections/experiments/index.html#experiment-1",
"title": "3 Experimental Results",
"section": "5.1 Experiment",
"text": "5.1 Experiment\n\nn_evals = 5\nn_rounds = 50\nevaluate_every = Int(round(n_rounds/n_evals))\nn_folds = 5\nn_samples = 10000\nT = 100\ngenerative_model_params = (epochs=250, latent_dim=8)\nresults = run_experiments(\n experiments;\n save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, T=T, n_samples=n_samples,\n generative_model_params=generative_model_params\n)\nSerialization.serialize(joinpath(output_path,\"results.jls\"),results)\n\n\n5.1.1 Plots\n\nresults = Serialization.deserialize(joinpath(output_path,\"results.jls\"))\n\n\nusing Images\nline_charts = Dict()\nerrorbar_charts = Dict()\nfor (data_name, res) in results\n plt = plot(res)\n Images.save(joinpath(www_path, \"line_chart_$(data_name).png\"), plt)\n line_charts[data_name] = plt\n plt = plot(res,maximum(res.output.n))\n Images.save(joinpath(www_path, \"errorbar_chart_$(data_name).png\"), plt)\n errorbar_charts[data_name] = plt\nend\n\n\n\n5.1.2 Line Charts\nFigure 5.1 shows the evolution of the evaluation metrics over the course of the experiment.\n\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"line_chart\")]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Credit Default\n\n\n\n\n\n\n\n(c) GMSC\n\n\n\nFigure 5.1: Line Charts\n\n\n\n\n\n5.1.3 Error Bar Charts\nFigure 5.2 shows the evaluation metrics at the end of the experiments.\n\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"errorbar_chart\")]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Credit Default\n\n\n\n\n\n\n\n(c) GMSC\n\n\n\nFigure 5.2: Error Bar Charts"
},
{
"objectID": "sections/experiments/index.html#bootstrap-1",
"href": "sections/experiments/index.html#bootstrap-1",
"title": "3 Experimental Results",
"section": "5.2 Bootstrap",
"text": "5.2 Bootstrap\n\nn_bootstrap = 100\ndf = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,\"bootstrap.csv\"))\n\n\n\n┌──────────┬─────────┬────────────────┬────────────────────┬───────────┬──────────────┐\n│ name │ scope │ data │ model │ generator │ p_value_mean │\n│ String31 │ String7 │ String15 │ String31 │ String7 │ Float64 │\n├──────────┼─────────┼────────────────┼────────────────────┼───────────┼──────────────┤\n│ mmd │ domain │ credit_default │ LogisticRegression │ Generic │ 0.594 │\n│ mmd │ domain │ credit_default │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ domain │ credit_default │ LogisticRegression │ DICE │ 0.268 │\n│ mmd │ domain │ credit_default │ LogisticRegression │ Greedy │ 0.388 │\n│ mmd │ domain │ credit_default │ FluxModel │ Generic │ 0.668 │\n│ mmd │ domain │ credit_default │ FluxModel │ REVISE │ 0.0 │\n│ mmd │ domain │ credit_default │ FluxModel │ DICE │ 0.466 │\n│ mmd │ domain │ credit_default │ FluxModel │ Greedy │ 0.998 │\n│ mmd │ domain │ credit_default │ FluxEnsemble │ Generic │ 0.738 │\n│ mmd │ domain │ credit_default │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd │ domain │ credit_default │ FluxEnsemble │ DICE │ 0.634 │\n│ mmd │ domain │ credit_default │ FluxEnsemble │ Greedy │ 1.0 │\n│ mmd │ domain │ cal_housing │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ domain │ cal_housing │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ domain │ cal_housing │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ domain │ cal_housing │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxModel │ REVISE │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxModel │ DICE │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxModel │ Greedy │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd │ domain │ gmsc │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ domain │ gmsc │ LogisticRegression │ REVISE │ 0.112 │\n│ mmd │ domain │ gmsc │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ domain │ gmsc │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxModel │ REVISE │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxModel │ DICE │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxModel │ Greedy │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd │ model │ credit_default │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ credit_default │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ model │ credit_default │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ model │ credit_default │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ model │ credit_default │ FluxModel │ Generic │ 0.0 │\n│ mmd │ model │ credit_default │ FluxModel │ REVISE │ 0.0 │\n│ mmd │ model │ credit_default │ FluxModel │ DICE │ 0.0 │\n│ mmd │ model │ credit_default │ FluxModel │ Greedy │ 0.0 │\n│ mmd │ model │ credit_default │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ model │ credit_default │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd │ model │ credit_default │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd │ model │ credit_default │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd │ model │ cal_housing │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ cal_housing │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ model │ cal_housing │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ model │ cal_housing │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxModel │ Generic │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxModel │ REVISE │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxModel │ DICE │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxModel │ Greedy │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd │ model │ gmsc │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ gmsc │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd │ model │ gmsc │ LogisticRegression │ DICE │ 0.0 │\n│ mmd │ model │ gmsc │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd │ model │ gmsc │ FluxModel │ Generic │ 0.0 │\n│ mmd │ model │ gmsc │ FluxModel │ REVISE │ 0.0 │\n│ mmd │ model │ gmsc │ FluxModel │ DICE │ 0.0 │\n│ mmd │ model │ gmsc │ FluxModel │ Greedy │ 0.0 │\n│ mmd │ model │ gmsc │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ model │ gmsc │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd │ model │ gmsc │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd │ model │ gmsc │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd_grid │ model │ credit_default │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ credit_default │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd_grid │ model │ credit_default │ LogisticRegression │ DICE │ 0.0 │\n│ mmd_grid │ model │ credit_default │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxModel │ Generic │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxModel │ REVISE │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxModel │ DICE │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxModel │ Greedy │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ LogisticRegression │ DICE │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxModel │ Generic │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxModel │ REVISE │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxModel │ DICE │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxModel │ Greedy │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxEnsemble │ DICE │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxEnsemble │ Greedy │ 0.0 │\n│ mmd_grid │ model │ gmsc │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ gmsc │ LogisticRegression │ REVISE │ 0.0 │\n│ mmd_grid │ model │ gmsc │ LogisticRegression │ DICE │ 0.0 │\n│ mmd_grid │ model │ gmsc │ LogisticRegression │ Greedy │ 0.0 │\n│ mmd_grid │ model │ gmsc │ FluxModel │ Generic │ 0.06 │\n│ mmd_grid │ model │ gmsc │ FluxModel │ REVISE │ 0.0 │\n│ mmd_grid │ model │ gmsc │ FluxModel │ DICE │ 0.0 │\n│ mmd_grid │ model │ gmsc │ FluxModel │ Greedy │ 0.0 │\n│ mmd_grid │ model │ gmsc │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ gmsc │ FluxEnsemble │ REVISE │ 0.0 │\n│ mmd_grid │ model │ gmsc │ FluxEnsemble │ DICE │ 0.002 │\n│ mmd_grid │ model │ gmsc │ FluxEnsemble │ Greedy │ 0.0 │\n└──────────┴─────────┴────────────────┴────────────────────┴───────────┴──────────────┘\n\n\n\n5.2.1 Chart in paper\nFigure 5.3 shows the chart that went into the paper.\n\nusing DataFrames, Statistics\nmodel_ = :FluxEnsemble\ndf = DataFrame() \nfor (key, val) in results\n df_ = deepcopy(val.output)\n df_.dataset .= key\n df = vcat(df,df_)\nend\ndf = df[df.n .== maximum(df.n),:]\ndf = df[df.model .== model_,:]\nfilter!(:value => x -> !any(f -> f(x), (ismissing, isnothing, isnan)), df)\ngdf = groupby(df, [:generator, :dataset, :n, :name, :scope])\ndf_plot = combine(gdf, :value => (x -> [(mean(x),mean(x)+std(x),mean(x)-std(x))]) => [:mean, :ymax, :ymin])\ndf_plot = df_plot[[name in [:mmd, :model_performance] for name in df_plot.name],:]\ndf_plot = mapcols(x -> typeof(x) == Vector{Symbol} ? string.(x) : x, df_plot)\ndf_plot.name .= [r[:name] == \"mmd\" ? \"$(r[:name])_$(r[:scope])\" : r[:name] for r in eachrow(df_plot)]\ntransform!(df_plot, :dataset => (X -> [x==\"cal_housing\" ? \"California Housing\" : x for x in X]) => :dataset)\ntransform!(df_plot, :dataset => (X -> [x==\"credit_default\" ? \"Credit Default\" : x for x in X]) => :dataset)\ntransform!(df_plot, :dataset => (X -> [x==\"gmsc\" ? \"GMSC\" : x for x in X]) => :dataset)\ntransform!(df_plot, :name => (X -> [x==\"mmd_domain\" ? \"MMD (domain)\" : x for x in X]) => :name)\ntransform!(df_plot, :name => (X -> [x==\"mmd_model\" ? \"MMD (model)\" : x for x in X]) => :name)\ntransform!(df_plot, :name => (X -> [x==\"model_performance\" ? \"Performance\" : x for x in X]) => :name)\ntransform!(df_plot, :generator => (X -> [x==\"REVISE\" ? \"Latent\" : x for x in X]) => :generator)\n\nncol = length(unique(df_plot.dataset))\nnrow = length(unique(df_plot.name))\n\nusing RCall\nscale_ = 1.75\nR\"\"\"\nlibrary(ggplot2)\nplt <- ggplot($df_plot) +\n geom_bar(aes(x=n, y=mean, fill=generator), stat=\"identity\", alpha=0.5, position=\"dodge\") +\n geom_pointrange( aes(x=n, y=mean, ymin=ymin, ymax=ymax, colour=generator), alpha=0.9, position=position_dodge(width=0.9), size=0.5) +\n facet_grid(\n rows = vars(name),\n cols = vars(dataset), \n scales = \"free_y\"\n ) +\n labs(y = \"Value\") + \n scale_fill_discrete(name=\"Generator:\") +\n scale_colour_discrete(name=\"Generator:\") +\n theme(\n axis.title.x=element_blank(),\n axis.text.x=element_blank(),\n axis.ticks.x=element_blank(),\n legend.position=\"bottom\"\n )\ntemp_path <- file.path(tempdir(), \"plot.png\")\nggsave(temp_path,width=$ncol * $scale_,height=$nrow * $scale_ * 0.8) \n\"\"\"\n\nimg = Images.load(rcopy(R\"temp_path\"))\nImages.save(joinpath(www_path,\"paper_real_world_results.png\"), img)\n\n\nImages.load(joinpath(www_artifact_path,\"paper_real_world_results.png\"))\n\n\n\nFigure 5.3: Chart in paper"
},
{
"objectID": "sections/experiments/index.html#synthetic",
"href": "sections/experiments/index.html#synthetic",
"title": "3 Experimental Results",
"section": "6.1 Synthetic",
"text": "6.1 Synthetic\n\nmax_obs = 1000\ncatalogue = load_synthetic(max_obs)\nchoices = [\n :linearly_separable, \n :overlapping, \n :circles, \n :moons,\n]\ndata_sets = filter(p -> p[1] in choices, catalogue)\n\n\nexperiments = set_up_experiments(data_sets,models,generators)\n\n\nn_evals = 5\nn_rounds = 50\nevaluate_every = Int(round(n_rounds/n_evals))\nn_folds = 5\nT = 100\nusing Serialization\nresults = run_experiments(\n experiments;\n save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, T=T\n)\nSerialization.serialize(joinpath(output_path,\"results_synthetic.jls\"),results)"
},
{
"objectID": "sections/experiments/index.html#plots-2",
"href": "sections/experiments/index.html#plots-2",
"title": "3 Experimental Results",
"section": "6.2 Plots",
"text": "6.2 Plots\n\nusing Serialization\nresults = Serialization.deserialize(joinpath(output_path,\"results_synthetic.jls\"))\n\n\nusing Images\nline_charts = Dict()\nerrorbar_charts = Dict()\nfor (data_name, res) in results\n plt = plot(res)\n Images.save(joinpath(www_path, \"line_chart_$(data_name).png\"), plt)\n line_charts[data_name] = plt\n plt = plot(res,maximum(res.output.n))\n Images.save(joinpath(www_path, \"errorbar_chart_$(data_name).png\"), plt)\n errorbar_charts[data_name] = plt\nend\n\n\n6.2.1 Line Charts\nFigure 6.1 shows the evolution of the evaluation metrics over the course of the experiment.\n\nchoices = [\n :linearly_separable, \n :overlapping, \n :circles, \n :moons,\n]\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"line_chart\") .&& .!contains.(readdir(www_artifact_path),\"latent\")]\nimg_files = img_files[Bool.(reduce(+, map(choice -> contains.(img_files, string(choice)), choices)))]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Circles\n\n\n\n\n\n\n\n(c) Credit Default\n\n\n\n\n\n\n\n(d) GMSC\n\n\n\nFigure 6.1: Line Charts\n\n\n\n\n\n6.2.2 Error Bar Charts\nFigure 6.2 shows the evaluation metrics at the end of the experiments.\n\nchoices = [\n :linearly_separable, \n :overlapping, \n :circles, \n :moons,\n]\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"errorbar_chart\") .&& .!contains.(readdir(www_artifact_path),\"latent\")]\nimg_files = img_files[Bool.(reduce(+, map(choice -> contains.(img_files, string(choice)), choices)))]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Circles\n\n\n\n\n\n\n\n(c) Credit Default\n\n\n\n\n\n\n\n(d) GMSC\n\n\n\nFigure 6.2: Error Bar Charts"
},
{
"objectID": "sections/experiments/index.html#bootstrap-2",
"href": "sections/experiments/index.html#bootstrap-2",
"title": "3 Experimental Results",
"section": "6.3 Bootstrap",
"text": "6.3 Bootstrap\n\nn_bootstrap = 100\ndf = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,\"bootstrap_synthetic.csv\"))\n\n\n\n┌──────────┬─────────┬────────────────────┬────────────────────┬──────────────────────┬──────────────┐\n│ name │ scope │ data │ model │ generator │ p_value_mean │\n│ String31 │ String7 │ String31 │ String31 │ String31 │ Float64 │\n├──────────┼─────────┼────────────────────┼────────────────────┼──────────────────────┼──────────────┤\n│ mmd │ domain │ overlapping │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxModel │ Gravitational │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxModel │ Latent │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ overlapping │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ overlapping │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ domain │ overlapping │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ domain │ overlapping │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ domain │ overlapping │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd │ domain │ overlapping │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxModel │ Gravitational │ 0.184 │\n│ mmd │ domain │ linearly_separable │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxModel │ Latent │ 0.358 │\n│ mmd │ domain │ linearly_separable │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ linearly_separable │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ linearly_separable │ LogisticRegression │ Gravitational │ 0.27 │\n│ mmd │ domain │ linearly_separable │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ domain │ linearly_separable │ LogisticRegression │ Latent │ 0.586 │\n│ mmd │ domain │ linearly_separable │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxEnsemble │ Gravitational │ 0.152 │\n│ mmd │ domain │ linearly_separable │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd │ domain │ linearly_separable │ FluxEnsemble │ Latent │ 0.38 │\n│ mmd │ domain │ linearly_separable │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ circles │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ circles │ FluxModel │ Gravitational │ 0.0 │\n│ mmd │ domain │ circles │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd │ domain │ circles │ FluxModel │ Latent │ 1.0 │\n│ mmd │ domain │ circles │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ circles │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ circles │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ domain │ circles │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ domain │ circles │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ domain │ circles │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ domain │ circles │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ circles │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd │ domain │ circles │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd │ domain │ circles │ FluxEnsemble │ Latent │ 1.0 │\n│ mmd │ domain │ circles │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ moons │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ moons │ FluxModel │ Gravitational │ 0.0 │\n│ mmd │ domain │ moons │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd │ domain │ moons │ FluxModel │ Latent │ 0.0 │\n│ mmd │ domain │ moons │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ moons │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ moons │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ domain │ moons │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ domain │ moons │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ domain │ moons │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ domain │ moons │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ moons │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd │ domain │ moons │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd │ domain │ moons │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd │ domain │ moons │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ model │ overlapping │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd │ model │ overlapping │ FluxModel │ Gravitational │ 0.024 │\n│ mmd │ model │ overlapping │ FluxModel │ ClapROAR │ 0.012 │\n│ mmd │ model │ overlapping │ FluxModel │ Latent │ 0.0 │\n│ mmd │ model │ overlapping │ FluxModel │ Generic │ 0.0 │\n│ mmd │ model │ overlapping │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ model │ overlapping │ LogisticRegression │ Gravitational │ 0.006 │\n│ mmd │ model │ overlapping │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ model │ overlapping │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ model │ overlapping │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ overlapping │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd │ model │ overlapping │ FluxEnsemble │ Gravitational │ 0.018 │\n│ mmd │ model │ overlapping │ FluxEnsemble │ ClapROAR │ 0.004 │\n│ mmd │ model │ overlapping │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd │ model │ overlapping │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ model │ linearly_separable │ FluxModel │ Generic_conservative │ 0.852 │\n│ mmd │ model │ linearly_separable │ FluxModel │ Gravitational │ 0.89 │\n│ mmd │ model │ linearly_separable │ FluxModel │ ClapROAR │ 0.84 │\n│ mmd │ model │ linearly_separable │ FluxModel │ Latent │ 0.722 │\n│ mmd │ model │ linearly_separable │ FluxModel │ Generic │ 0.83 │\n│ mmd │ model │ linearly_separable │ LogisticRegression │ Generic_conservative │ 0.176 │\n│ mmd │ model │ linearly_separable │ LogisticRegression │ Gravitational │ 0.898 │\n│ mmd │ model │ linearly_separable │ LogisticRegression │ ClapROAR │ 0.866 │\n│ mmd │ model │ linearly_separable │ LogisticRegression │ Latent │ 0.338 │\n│ mmd │ model │ linearly_separable │ LogisticRegression │ Generic │ 0.002 │\n│ mmd │ model │ linearly_separable │ FluxEnsemble │ Generic_conservative │ 0.868 │\n│ mmd │ model │ linearly_separable │ FluxEnsemble │ Gravitational │ 0.912 │\n│ mmd │ model │ linearly_separable │ FluxEnsemble │ ClapROAR │ 0.916 │\n│ mmd │ model │ linearly_separable │ FluxEnsemble │ Latent │ 0.822 │\n│ mmd │ model │ linearly_separable │ FluxEnsemble │ Generic │ 0.818 │\n│ mmd │ model │ circles │ FluxModel │ Generic_conservative │ 0.356 │\n│ mmd │ model │ circles │ FluxModel │ Gravitational │ 0.48 │\n│ mmd │ model │ circles │ FluxModel │ ClapROAR │ 0.484 │\n│ mmd │ model │ circles │ FluxModel │ Latent │ 0.802 │\n│ mmd │ model │ circles │ FluxModel │ Generic │ 0.482 │\n│ mmd │ model │ circles │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ model │ circles │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ model │ circles │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ model │ circles │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ model │ circles │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ circles │ FluxEnsemble │ Generic_conservative │ 0.462 │\n│ mmd │ model │ circles │ FluxEnsemble │ Gravitational │ 0.282 │\n│ mmd │ model │ circles │ FluxEnsemble │ ClapROAR │ 0.534 │\n│ mmd │ model │ circles │ FluxEnsemble │ Latent │ 0.91 │\n│ mmd │ model │ circles │ FluxEnsemble │ Generic │ 0.408 │\n│ mmd │ model │ moons │ FluxModel │ Generic_conservative │ 0.568 │\n│ mmd │ model │ moons │ FluxModel │ Gravitational │ 0.868 │\n│ mmd │ model │ moons │ FluxModel │ ClapROAR │ 0.594 │\n│ mmd │ model │ moons │ FluxModel │ Latent │ 0.598 │\n│ mmd │ model │ moons │ FluxModel │ Generic │ 0.604 │\n│ mmd │ model │ moons │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ model │ moons │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ model │ moons │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ model │ moons │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ model │ moons │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ moons │ FluxEnsemble │ Generic_conservative │ 0.48 │\n│ mmd │ model │ moons │ FluxEnsemble │ Gravitational │ 0.922 │\n│ mmd │ model │ moons │ FluxEnsemble │ ClapROAR │ 0.584 │\n│ mmd │ model │ moons │ FluxEnsemble │ Latent │ 0.654 │\n│ mmd │ model │ moons │ FluxEnsemble │ Generic │ 0.412 │\n│ mmd_grid │ model │ overlapping │ FluxModel │ Generic_conservative │ 0.002 │\n│ mmd_grid │ model │ overlapping │ FluxModel │ Gravitational │ 0.034 │\n│ mmd_grid │ model │ overlapping │ FluxModel │ ClapROAR │ 0.022 │\n│ mmd_grid │ model │ overlapping │ FluxModel │ Latent │ 0.008 │\n│ mmd_grid │ model │ overlapping │ FluxModel │ Generic │ 0.0 │\n│ mmd_grid │ model │ overlapping │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ overlapping │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ overlapping │ LogisticRegression │ ClapROAR │ 0.002 │\n│ mmd_grid │ model │ overlapping │ LogisticRegression │ Latent │ 0.0 │\n│ mmd_grid │ model │ overlapping │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ overlapping │ FluxEnsemble │ Generic_conservative │ 0.006 │\n│ mmd_grid │ model │ overlapping │ FluxEnsemble │ Gravitational │ 0.034 │\n│ mmd_grid │ model │ overlapping │ FluxEnsemble │ ClapROAR │ 0.058 │\n│ mmd_grid │ model │ overlapping │ FluxEnsemble │ Latent │ 0.016 │\n│ mmd_grid │ model │ overlapping │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxModel │ Gravitational │ 0.062 │\n│ mmd_grid │ model │ linearly_separable │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxModel │ Latent │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxModel │ Generic │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ LogisticRegression │ Gravitational │ 0.004 │\n│ mmd_grid │ model │ linearly_separable │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ LogisticRegression │ Latent │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxEnsemble │ Gravitational │ 0.128 │\n│ mmd_grid │ model │ linearly_separable │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd_grid │ model │ linearly_separable │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ circles │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ circles │ FluxModel │ Gravitational │ 0.002 │\n│ mmd_grid │ model │ circles │ FluxModel │ ClapROAR │ 0.004 │\n│ mmd_grid │ model │ circles │ FluxModel │ Latent │ 0.176 │\n│ mmd_grid │ model │ circles │ FluxModel │ Generic │ 0.008 │\n│ mmd_grid │ model │ circles │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ circles │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ circles │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ circles │ LogisticRegression │ Latent │ 0.0 │\n│ mmd_grid │ model │ circles │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ circles │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ circles │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ circles │ FluxEnsemble │ ClapROAR │ 0.002 │\n│ mmd_grid │ model │ circles │ FluxEnsemble │ Latent │ 0.24 │\n│ mmd_grid │ model │ circles │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ moons │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ moons │ FluxModel │ Gravitational │ 0.052 │\n│ mmd_grid │ model │ moons │ FluxModel │ ClapROAR │ 0.002 │\n│ mmd_grid │ model │ moons │ FluxModel │ Latent │ 0.024 │\n│ mmd_grid │ model │ moons │ FluxModel │ Generic │ 0.0 │\n│ mmd_grid │ model │ moons │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ moons │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ moons │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ moons │ LogisticRegression │ Latent │ 0.0 │\n│ mmd_grid │ model │ moons │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ moons │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ moons │ FluxEnsemble │ Gravitational │ 0.164 │\n│ mmd_grid │ model │ moons │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ moons │ FluxEnsemble │ Latent │ 0.058 │\n│ mmd_grid │ model │ moons │ FluxEnsemble │ Generic │ 0.0 │\n└──────────┴─────────┴────────────────────┴────────────────────┴──────────────────────┴──────────────┘"
},
{
"objectID": "sections/experiments/index.html#chart-in-paper-1",
"href": "sections/experiments/index.html#chart-in-paper-1",
"title": "3 Experimental Results",
"section": "6.4 Chart in paper",
"text": "6.4 Chart in paper\nFigure 6.3 shows the chart that went into the paper.\n\nImages.load(joinpath(www_artifact_path,\"paper_synthetic_results.png\"))\n\n\n\nFigure 6.3: Chart in paper"
},
{
"objectID": "sections/experiments/index.html#latent-space-search",
"href": "sections/experiments/index.html#latent-space-search",
"title": "3 Experimental Results",
"section": "6.5 Latent Space Search",
"text": "6.5 Latent Space Search\n\ngenerators = Dict(\n :Latent=>GenericGenerator(opt = opt, decision_threshold=0.5),\n :Latent_conservative=>GenericGenerator(opt = opt, decision_threshold=0.9),\n :Gravitational=>GravitationalGenerator(opt = opt),\n :ClapROAR=>ClapROARGenerator(opt = opt)\n)\n\n\nexperiments = set_up_experiments(data_sets,models,generators)\n\n\nn_evals = 5\nn_rounds = 50\nevaluate_every = Int(round(n_rounds/n_evals))\nn_folds = 5\nT = 100\nusing Serialization\nresults = run_experiments(\n experiments;\n save_path=output_path, evaluate_every=evaluate_every, n_rounds=n_rounds, n_folds=n_folds, T=T, \n latent_space = true\n)\nSerialization.serialize(joinpath(output_path,\"results_synthetic_latent.jls\"),results)\n\n\nusing Serialization\nresults = Serialization.deserialize(joinpath(output_path,\"results_synthetic_latent.jls\"))\n\n\nusing Images\nline_charts = Dict()\nerrorbar_charts = Dict()\nfor (data_name, res) in results\n plt = plot(res)\n Images.save(joinpath(www_path, \"line_chart_latent_$(data_name).png\"), plt)\n line_charts[data_name] = plt\n plt = plot(res,maximum(res.output.n))\n Images.save(joinpath(www_path, \"errorbar_chart_latent_$(data_name).png\"), plt)\n errorbar_charts[data_name] = plt\nend"
},
{
"objectID": "sections/experiments/index.html#plots-3",
"href": "sections/experiments/index.html#plots-3",
"title": "3 Experimental Results",
"section": "6.6 Plots",
"text": "6.6 Plots\n\n6.6.1 Line Charts\nFigure 6.4 shows the evolution of the evaluation metrics over the course of the experiment.\n\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"line_chart\") .&& contains.(readdir(www_artifact_path),\"latent\")]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n display(load(img))\nend\n\n\n\n\n\n\n(a) Circles\n\n\n\n\n\n\n\n(b) Linearly Separable\n\n\n\n\n\n\n\n(c) Moons\n\n\n\n\n\n\n\n(d) Overlapping\n\n\n\nFigure 6.4: Line Charts\n\n\n\n\n\n6.6.2 Error Bar Charts\nFigure 6.5 shows the evaluation metrics at the end of the experiments.\n\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"errorbar_chart\") .&& contains.(readdir(www_artifact_path),\"latent\")]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n display(load(img))\nend\n\n\n\n\n\n\n(a) Circles\n\n\n\n\n\n\n\n(b) Linearly Separable\n\n\n\n\n\n\n\n(c) Moons\n\n\n\n\n\n\n\n\n(d) Overlapping\n\n\n\nFigure 6.5: Error Bar Charts"
},
{
"objectID": "sections/experiments/index.html#bootstrap-3",
"href": "sections/experiments/index.html#bootstrap-3",
"title": "3 Experimental Results",
"section": "6.7 Bootstrap",
"text": "6.7 Bootstrap\n\nn_bootstrap = 100\ndf = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,\"bootstrap_latent.csv\"))"
},
{
"objectID": "sections/experiments/index.html#chart-in-paper-2",
"href": "sections/experiments/index.html#chart-in-paper-2",
"title": "3 Experimental Results",
"section": "6.8 Chart in paper",
"text": "6.8 Chart in paper\nFigure 6.6 shows the chart that went into the paper.\n\nImages.load(joinpath(www_artifact_path,\"paper_synthetic_latent_results.png\"))\n\n\n\nFigure 6.6: Chart in paper"
},
{
"objectID": "sections/experiments/index.html#real-world",
"href": "sections/experiments/index.html#real-world",
"title": "3 Experimental Results",
"section": "6.9 Real World",
"text": "6.9 Real World\n\nmodels = [\n :LogisticRegression, \n :FluxModel, \n :FluxEnsemble,\n]\nopt = Flux.Descent(0.01) \ngenerators = Dict(\n :Generic=>GenericGenerator(opt = opt, decision_threshold=0.5),\n :Latent=>REVISEGenerator(opt = opt),\n :Generic_conservative=>GenericGenerator(opt = opt, decision_threshold=0.9),\n :Gravitational=>GravitationalGenerator(opt = opt),\n :ClapROAR=>ClapROARGenerator(opt = opt)\n)\n\n\nmax_obs = 2500\ndata_path = data_dir(\"real_world\")\ndata_sets = load_real_world(max_obs)\nchoices = [\n :cal_housing, \n :credit_default, \n :gmsc, \n]\ndata_sets = filter(p -> p[1] in choices, data_sets)\n\n\nusing CounterfactualExplanations.DataPreprocessing: unpack\nbs = 500\nfunction data_loader(data::CounterfactualData)\n X, y = unpack(data)\n data = Flux.DataLoader((X,y),batchsize=bs)\n return data\nend\nmodel_params = (batch_norm=false,n_hidden=64,n_layers=3,dropout=true,p_dropout=0.1)\n\n\nexperiments = set_up_experiments(\n data_sets,models,generators; \n pre_train_models=100, model_params=model_params, \n data_loader=data_loader\n)\n\n\nn_evals = 5\nn_rounds = 50\nevaluate_every = Int(round(n_rounds/n_evals))\nn_folds = 5\nn_samples = 10000\nT = 100\ngenerative_model_params = (epochs=250, latent_dim=8)\nresults = run_experiments(\n experiments;\n save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, T=T, n_samples=n_samples,\n generative_model_params=generative_model_params\n)\nSerialization.serialize(joinpath(output_path,\"results_real_world.jls\"),results)\n\n\nusing Serialization\nresults = Serialization.deserialize(joinpath(output_path,\"results_real_world.jls\"))\n\n\nusing Images\nline_charts = Dict()\nerrorbar_charts = Dict()\nfor (data_name, res) in results\n plt = plot(res)\n Images.save(joinpath(www_path, \"line_chart_$(data_name).png\"), plt)\n line_charts[data_name] = plt\n plt = plot(res,maximum(res.output.n))\n Images.save(joinpath(www_path, \"errorbar_chart_$(data_name).png\"), plt)\n errorbar_charts[data_name] = plt\nend\n\n\n6.9.1 Line Charts\nFigure 5.1 shows the evolution of the evaluation metrics over the course of the experiment.\n\nchoices = [\n :cal_housing, \n :credit_default, \n :gmsc, \n]\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"line_chart\")]\nimg_files = img_files[Bool.(reduce(+, map(choice -> contains.(img_files, string(choice)), choices)))]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Credit Default\n\n\n\n\n\n\n\n(c) GMSC\n\n\n\nFigure 6.7: Line Charts\n\n\n\n\n\n6.9.2 Error Bar Charts\nFigure 5.2 shows the evaluation metrics at the end of the experiments.\n\nchoices = [\n :cal_housing, \n :credit_default, \n :gmsc, \n]\nimg_files = readdir(www_artifact_path)[contains.(readdir(www_artifact_path),\"errorbar_chart\")]\nimg_files = img_files[Bool.(reduce(+, map(choice -> contains.(img_files, string(choice)), choices)))]\nimg_files = joinpath.(www_artifact_path,img_files)\nfor img in img_files\n display(load(img))\nend\n\n\n\n\n\n\n(a) California Housing\n\n\n\n\n\n\n\n(b) Credit Default\n\n\n\n\n\n\n\n\n(c) GMSC\n\n\n\nFigure 6.8: Error Bar Charts\n\n\n\n\n\n6.9.3 Bootstrap\n\nn_bootstrap = 100\ndf = run_bootstrap(results, n_bootstrap; filename=joinpath(output_path,\"bootstrap_real_world.csv\"))\n\n\n\n┌──────────┬─────────┬────────────────┬────────────────────┬──────────────────────┬──────────────┐\n│ name │ scope │ data │ model │ generator │ p_value_mean │\n│ String31 │ String7 │ String15 │ String31 │ String31 │ Float64 │\n├──────────┼─────────┼────────────────┼────────────────────┼──────────────────────┼──────────────┤\n│ mmd │ domain │ credit_default │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd │ domain │ credit_default │ FluxEnsemble │ Generic_conservative │ 1.0 │\n│ mmd │ domain │ credit_default │ FluxEnsemble │ ClapROAR │ 1.0 │\n│ mmd │ domain │ credit_default │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd │ domain │ credit_default │ FluxEnsemble │ Generic │ 1.0 │\n│ mmd │ domain │ credit_default │ FluxModel │ Latent │ 0.0 │\n│ mmd │ domain │ credit_default │ FluxModel │ Generic_conservative │ 1.0 │\n│ mmd │ domain │ credit_default │ FluxModel │ ClapROAR │ 1.0 │\n│ mmd │ domain │ credit_default │ FluxModel │ Gravitational │ 0.0 │\n│ mmd │ domain │ credit_default │ FluxModel │ Generic │ 1.0 │\n│ mmd │ domain │ credit_default │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ domain │ credit_default │ LogisticRegression │ Generic_conservative │ 0.256 │\n│ mmd │ domain │ credit_default │ LogisticRegression │ ClapROAR │ 1.0 │\n│ mmd │ domain │ credit_default │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ domain │ credit_default │ LogisticRegression │ Generic │ 1.0 │\n│ mmd │ domain │ cal_housing │ FluxEnsemble │ Latent │ 0.398 │\n│ mmd │ domain │ cal_housing │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxModel │ Latent │ 0.204 │\n│ mmd │ domain │ cal_housing │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxModel │ Gravitational │ 0.0 │\n│ mmd │ domain │ cal_housing │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ cal_housing │ LogisticRegression │ Latent │ 0.2 │\n│ mmd │ domain │ cal_housing │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ cal_housing │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ domain │ cal_housing │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ domain │ cal_housing │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxModel │ Latent │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxModel │ Gravitational │ 0.0 │\n│ mmd │ domain │ gmsc │ FluxModel │ Generic │ 0.0 │\n│ mmd │ domain │ gmsc │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ domain │ gmsc │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ domain │ gmsc │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ domain │ gmsc │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ domain │ gmsc │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ credit_default │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd │ model │ credit_default │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd │ model │ credit_default │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd │ model │ credit_default │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd │ model │ credit_default │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ model │ credit_default │ FluxModel │ Latent │ 0.0 │\n│ mmd │ model │ credit_default │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd │ model │ credit_default │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd │ model │ credit_default │ FluxModel │ Gravitational │ 0.0 │\n│ mmd │ model │ credit_default │ FluxModel │ Generic │ 0.0 │\n│ mmd │ model │ credit_default │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ model │ credit_default │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ model │ credit_default │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ model │ credit_default │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ model │ credit_default │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxModel │ Latent │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxModel │ Gravitational │ 0.0 │\n│ mmd │ model │ cal_housing │ FluxModel │ Generic │ 0.0 │\n│ mmd │ model │ cal_housing │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ model │ cal_housing │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ model │ cal_housing │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ model │ cal_housing │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ model │ cal_housing │ LogisticRegression │ Generic │ 0.0 │\n│ mmd │ model │ gmsc │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd │ model │ gmsc │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd │ model │ gmsc │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd │ model │ gmsc │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd │ model │ gmsc │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd │ model │ gmsc │ FluxModel │ Latent │ 0.0 │\n│ mmd │ model │ gmsc │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd │ model │ gmsc │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd │ model │ gmsc │ FluxModel │ Gravitational │ 0.0 │\n│ mmd │ model │ gmsc │ FluxModel │ Generic │ 0.0 │\n│ mmd │ model │ gmsc │ LogisticRegression │ Latent │ 0.0 │\n│ mmd │ model │ gmsc │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd │ model │ gmsc │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd │ model │ gmsc │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd │ model │ gmsc │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxModel │ Latent │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxModel │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ credit_default │ FluxModel │ Generic │ 0.0 │\n│ mmd_grid │ model │ credit_default │ LogisticRegression │ Latent │ 0.0 │\n│ mmd_grid │ model │ credit_default │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ credit_default │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ credit_default │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ credit_default │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxEnsemble │ Latent │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxEnsemble │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxModel │ Latent │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxModel │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxModel │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxModel │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ FluxModel │ Generic │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ LogisticRegression │ Latent │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ cal_housing │ LogisticRegression │ Generic │ 0.0 │\n│ mmd_grid │ model │ gmsc │ FluxEnsemble │ Latent │ 0.016 │\n│ mmd_grid │ model │ gmsc │ FluxEnsemble │ Generic_conservative │ 0.008 │\n│ mmd_grid │ model │ gmsc │ FluxEnsemble │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ gmsc │ FluxEnsemble │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ gmsc │ FluxEnsemble │ Generic │ 0.0 │\n│ mmd_grid │ model │ gmsc │ FluxModel │ Latent │ 0.072 │\n│ mmd_grid │ model │ gmsc │ FluxModel │ Generic_conservative │ 0.2 │\n│ mmd_grid │ model │ gmsc │ FluxModel │ ClapROAR │ 0.244 │\n│ mmd_grid │ model │ gmsc │ FluxModel │ Gravitational │ 0.056 │\n│ mmd_grid │ model │ gmsc │ FluxModel │ Generic │ 0.23 │\n│ mmd_grid │ model │ gmsc │ LogisticRegression │ Latent │ 0.0 │\n│ mmd_grid │ model │ gmsc │ LogisticRegression │ Generic_conservative │ 0.0 │\n│ mmd_grid │ model │ gmsc │ LogisticRegression │ ClapROAR │ 0.0 │\n│ mmd_grid │ model │ gmsc │ LogisticRegression │ Gravitational │ 0.0 │\n│ mmd_grid │ model │ gmsc │ LogisticRegression │ Generic │ 0.0 │\n└──────────┴─────────┴────────────────┴────────────────────┴──────────────────────┴──────────────┘\n\n\n\n\n6.9.4 Chart in paper\nFigure 6.9 shows the chart that went into the paper.\n\nImages.load(joinpath(www_artifact_path,\"paper_real_world_results.png\"))\n\n\n\nFigure 6.9: Chart in paper"
},
{
"objectID": "sections/generators/index.html",
"href": "sections/generators/index.html",
"title": "4 Generators",
"section": "",
"text": "The generators have been moved into CounterfactualExplanations.jl. For more information so the package documentation."
}
]