Overview
Pironen and Vehtari (Statistical Computing 27:711-735; 2017) present a very comprehensive overview of methods for model selection in a Bayesian context. I encourage you to read the whole thing, but this notebook is going to introduce only the best performing of the methods they review: projection predictive feature selection. Let’s unpack what that means, starting with “feature selection.”
Feature selection is just another name for “variable selection.” What’s different here from what we saw with horseshoe priors is that rather than simply looking at the posterior estimates and saying to ourselves “These variables (features) seem important and thoee don’t”, projection predictive feature selection provides a statistical criterion to help us identify the important variables (features).
What about the “projection prediction” part? Well, that’s a bit more complicated. You’ll need to read Pironen and Vehtari to get all of the details, but here it is in a nutshell.
We start with a full model that (we think) includes all of the covariates that could be relevant in predicting the response. Now imagine that we examine reduced models that don’t include all of the covariates, but instead of fitting those reduced models to the data and comparing their performance in predicting the data we fit them to data predicted from the full model, i.e., its posterior predictions, and find the reduced model that does the “best job” of matching the posterior predictions. By “best job” we mean that we pick the model with the smallest number of covariates we can while keeping the predictive performance of the reduced model about the same as the full model.
While it may seem strange to pick covariates based on how well models perform on predicted rather than observed data, Pironen and Vehtari show that this approach works very well when compared with other alternatives that have been suggested. And they’ve put together a very nice package, projpred that makes it easy to use the approach.
Setting up the data
As usual, we have to start by generating the data.
library(tidyverse)
library(reshape2)
library(ggplot2)
library(cowplot)
library(mvtnorm)
library(corrplot)
rm(list = ls())
## intetcept
##
beta0 <- 1.0
## regression coefficients
##
beta <- c(1.0, -1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0)
## pattern of correlation matrix, all non-zero entries are set to saem
## correlation, covariance matrix caldulated from individual variances and a
## single association parameter governing the non-zero correlation coefficients
##
## Note: Not just any pattern will work here. The correlation matrix and
## covariance matrix generated from this pattern must be positive definite.
## If you change this pattern, you may get an error when you try to generate
## data with a non-zero association parameter.
##
Rho <- matrix(nrow = 9, ncol = , byrow = TRUE,
data = c(1,0,1,0,1,0,1,0,1,
0,1,0,1,0,1,0,1,0,
1,0,1,0,1,0,1,0,1,
0,1,0,1,0,1,0,1,0,
1,0,1,0,1,0,1,0,1,
0,1,0,1,0,1,0,1,0,
1,0,1,0,1,0,1,0,1,
0,1,0,1,0,1,0,1,0,
1,0,1,0,1,0,1,0,1
))
## vector of standard deviations for covariates
##
sigma <- rep(1, 9)
## construct a covariance matrix from the pattern, standard deviations, and
## one parameter in [-1,1] that governs the magnitude of non-zero correlation
## coefficients
##
## Rho - the pattern of associations
## sigma - the vector of standard deviations
## rho - the association parameter
##
construct_Sigma <- function(Rho, sigma, rho) {
## get the correlation matris
##
Rho <- Rho*rho
for (i in 1:ncol(Rho)) {
Rho[i,i] <- 1.0
}
## notice the use of matrix multiplication
##
Sigma <- diag(sigma) %*% Rho %*% diag(sigma)
return(Sigma)
}
## set the random number seed manually so that every run of the code will
## produce the same numbers
##
set.seed(1234)
n_samp <- 100
cov_str <- rmvnorm(n_samp,
mean = rep(0, nrow(Rho)),
sigma = construct_Sigma(Rho, sigma, 0.8))
resid <- rep(2.0, n_samp)
y_str <- rnorm(nrow(cov_str), mean = beta0 + cov_str %*% beta, sd = resid)
dat_1 <- data.frame(y_str, cov_str, rep("Strong", length(y_str)))
cov_str <- rmvnorm(n_samp,
mean = rep(0, nrow(Rho)),
sigma = construct_Sigma(Rho, sigma, 0.8))
y_str <- rnorm(nrow(cov_str), mean = beta0 + cov_str %*% beta, sd = resid)
dat_2 <- data.frame(y_str, cov_str, rep("Strong", length(y_str)))
column_names <- c("y", paste("x", seq(1, length(beta)), sep = ""), "Scenario")
colnames(dat_1) <- column_names
colnames(dat_2) <- column_names
## saving results in scale allows me to use them later for prediction with
## new data
##
scale_1 <- lapply(dat_1[, 1:10], scale)
scale_2 <- lapply(dat_2[, 1:10], scale)
## when assigning the same scaling to a data frame, the scaling attributes
## are lost
##
dat_1[, 1:10] <- lapply(dat_1[, 1:10], scale)
dat_2[, 1:10] <- lapply(dat_2[, 1:10], scale)
Trying projection
Since we’ll be running the projection predictive analysis on both data sets, I’ve written a small function, projection_prediction() that will run everything for us and collect the results. Notice that we start by fitting the full model with horseshoe priors.
library(rstanarm)
library(projpred)
library(bayesplot)
options(mc.cores = parallel::detectCores())
projection_prediction <- function(dat) {
n <- nrow(dat)
D <- ncol(dat[,2:10])
p0 <- 3
tau0 <- p0/(D - p0) * 1/sqrt(n)
prior_coeff <- hs(global_scale = tau0, slab_scale = 1)
fit <- stan_glm(y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9, data = dat,
prior = prior_coeff, refresh = 0)
vs <- varsel(fit, method = "forward")
cvs <- cv_varsel(fit, method = "forward", verbose = FALSE)
proj_size <- suggest_size(cvs)
proj <- project(vs, nv = proj_size, ns = 2000)
mcmc_intervals(as.matrix(proj),
pars = c("(Intercept)", names(vs$vind[1:proj_size]), "sigma"))
pred <- proj_linpred(vs, xnew = dat[, 2:10], ynew = dat$y,
nv = proj_size, integrated = TRUE)
for.plot <- data.frame(Observed = dat$y,
Predicted = pred$pred)
p <- ggplot(for.plot, aes(x = Observed, y = Predicted)) +
geom_point() +
geom_smooth(method = "lm") +
geom_abline(slope = 1, intercept = 0) +
labs(x = "Observed", y = "Predicted")
print(p)
return(list(proj = proj, vs = vs, cvs = cvs, proj_size = proj_size))
}
summarize_posterior <- function(x, credible = 0.95, digits = 3) {
lo_p <- (1.0 - credible)/2.0
hi_p <- credible + lo_p
ci <- quantile(x, c(lo_p, hi_p))
results <- sprintf("% 5.3f (% 5.3f, % 5.3f)\n", mean(x), ci[1], ci[2])
cat(results, sep = "")
}
summarize_results <- function(x, credible = 0.95, digits = 3) {
vars <- colnames(as.matrix(x))
for (i in 1:length(vars)) {
label <- sprintf("%11s: ", vars[i])
cat(label, sep = "")
summarize_posterior(as.matrix(x)[,i])
}
}
proj_1 <- projection_prediction(dat_1)
proj_2 <- projection_prediction(dat_2)
summarize_results(proj_1$proj)
(Intercept): 0.003 (-0.157, 0.154)
x3: 0.346 ( 0.075, 0.595)
x1: 0.252 ( 0.010, 0.521)
sigma: 0.833 ( 0.727, 0.973)
summarize_results(proj_2$proj)
(Intercept): -0.001 (-0.148, 0.149)
x3: 0.578 ( 0.421, 0.726)
x6: -0.401 (-0.547, -0.256)
sigma: 0.759 ( 0.655, 0.884)
Here we have results that are quite different from one another. Not only is the coefficient on x3 quite different in the two data sets, but the variables identified as important are different: x3 and x1 in analysis of the first data set and x3 and x6 in the second data set. That isn’t quite as disturbing as it might first appear. Look back at the results from our regression with horseshoe priors on all of the data. You’ll see that x1 and x3 had a greater magnitude in analysis of the first data set than any other covariates. Similarly, x3 and x6 were the most important in analysis of the second data set.
Picking the number of covariates
In the code I ran above I used suggest_size() to pick the number of covariates to use. If you read the documentation for suggest_size() you’ll notice this sentence in the Description:
It is recommended that the user studies the results via varsel_plot and/or varsel_stats and makes the final decision based on what is most appropriate for the given problem.
Let’s take a look at the results from varsel_plot() first.
print(varsel_plot(proj_1$cvs, stats = c("elpd", "rmse"), deltas = TRUE))
print(varsel_plot(proj_2$cvs, stats = c("elpd", "rmse"), deltas = TRUE))
elpd abd rmse are measures of how well a model fits the data. The complete model in our example includes 9 parameters. You can see that in the first data set a model with only 2 covariates works almost as well as the full model, although the model with 3 covariates looks as if it might work a little better. In the second data set, there is clearly no advantage to including more than 2 covariates.
The pictures are pretty clear, but let’s look at some the differences numerically using varsel_stats(). We’ll focus on elpd.
stats_1 <- varsel_stats(proj_1$cvs) %>%
mutate(diff = elpd - last(elpd), percdiff = diff/last(elpd))
stats_2 <- varsel_stats(proj_2$cvs) %>%
mutate(diff = elpd - last(elpd), percdiff = diff/last(elpd))
print(round(stats_1, 3))
print(round(stats_2, 3))
By default suggest_size() identifies the smallest model where elpd is within one standard deviation (elpd.se) of the full model. In both data sets that happens when two covariates are included. You will notice, however, that just as a model with 3 parameters looked as if it might be a little better than one with only 2 for the first data set, here the model with 3 parameters improves the model with 2 parameters by 0.028, which is more than twice the improvement made by any incuding any additional parameters. This suggests to me that it probably worth looking at the results from the first data set where we include 3 covariates instead of 2.
proj_1_3 <- project(proj_1$vs, nv = 3, ns = 2000)
summarize_results(proj_1_3)
(Intercept): 0.003 (-0.157, 0.154)
x3: 0.354 ( 0.084, 0.603)
x1: 0.270 ( 0.019, 0.540)
x2: -0.135 (-0.288, 0.007)
sigma: 0.819 ( 0.713, 0.957)
That’s rather interesting. It picked out the “right” coefficients, in the sense that there the ones we set as non-zero in generating the data, and it also got the “right” signs for them. Notice also that the coefficient on x2 is substantially larger than it is in the analysis of the full model with horseshoe priors (again ignoring the very broad overlap in credible intervals).
We don’t have a good reason to look at a model with more than 2 covariates in the second data set, but let’s try it with 3 and see what happens.
proj_2_3 <- project(proj_2$vs, nv = 3, ns = 2000)
summarize_results(proj_2_3)
(Intercept): -0.001 (-0.148, 0.149)
x3: 0.484 ( 0.233, 0.698)
x6: -0.399 (-0.545, -0.254)
x1: 0.126 (-0.045, 0.384)
sigma: 0.751 ( 0.645, 0.873)
That did bring in x1, which we know ought to be there. But again, the only reason we have to prefer this model given these data is our prior knowledge that x3 ought to be in the model.
Conclusions
If you can run a regression model in stan_glm() or stan_glmer(), you can easily use horseshoe priors. And if you can do that, you can easily use projection predictive variable selection to identtify the “most important” set of covariates. The simple example here suggests a couple of things:
Be sure to examine varsel_plot(), varsel_stats(), or both rather than simply relying on suggest_size() to tell you how many covariates to include.
It’s probably worth your time to take a look at models that include one or two more parameters than what your examination of varsel_plot() and varsel_stats() suggest. You’ll need to bring in your subject matter knowledge here, but there might be a covariate with an association worth considering based on subtantive grounds, even if it’s not “significant” in a conventional sense.
That last point raises a much broader issue that I’ll return to in the last blog post in this series, where I try to summarize the lessons we’ve learned and provide some general guidelines. The difference between “significant” and “not Significant” is not itself statistically significant.
LS0tCnRpdGxlOiAiUHJvamVjdGlvbiBwcmVkaWN0aXZlIGZlYXR1cmUgc2VsZWN0aW9uIgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKLS0tCiMjIE92ZXJ2aWV3CgpQaXJvbmVuIGFuZCBWZWh0YXJpIChbX1N0YXRpc3RpY2FsIENvbXB1dGluZ18gMjc6NzExLTczNTsgMjAxN10oaHR0cDovL2R4LmRvaS5vcmcvMTAuMTAwNy9zMTEyMjItMDE2LTk2NDkteSkpIHByZXNlbnQgYSB2ZXJ5IGNvbXByZWhlbnNpdmUgb3ZlcnZpZXcgb2YgbWV0aG9kcyBmb3IgbW9kZWwgc2VsZWN0aW9uIGluIGEgQmF5ZXNpYW4gY29udGV4dC4gSSBlbmNvdXJhZ2UgeW91IHRvIHJlYWQgdGhlIHdob2xlIHRoaW5nLCBidXQgdGhpcyBub3RlYm9vayBpcyBnb2luZyB0byBpbnRyb2R1Y2Ugb25seSB0aGUgYmVzdCBwZXJmb3JtaW5nIG9mIHRoZSBtZXRob2RzIHRoZXkgcmV2aWV3OiBwcm9qZWN0aW9uIHByZWRpY3RpdmUgZmVhdHVyZSBzZWxlY3Rpb24uIExldCdzIHVucGFjayB3aGF0IHRoYXQgbWVhbnMsIHN0YXJ0aW5nIHdpdGggImZlYXR1cmUgc2VsZWN0aW9uLiIKCkZlYXR1cmUgc2VsZWN0aW9uIGlzIGp1c3QgYW5vdGhlciBuYW1lIGZvciAidmFyaWFibGUgc2VsZWN0aW9uLiIgV2hhdCdzIGRpZmZlcmVudCBoZXJlIGZyb20gd2hhdCB3ZSBzYXcgd2l0aCBob3JzZXNob2UgcHJpb3JzIGlzIHRoYXQgcmF0aGVyIHRoYW4gc2ltcGx5IGxvb2tpbmcgYXQgdGhlIHBvc3RlcmlvciBlc3RpbWF0ZXMgYW5kIHNheWluZyB0byBvdXJzZWx2ZXMgIlRoZXNlIHZhcmlhYmxlcyAoZmVhdHVyZXMpIHNlZW0gaW1wb3J0YW50IGFuZCB0aG9lZSBkb24ndCIsIHByb2plY3Rpb24gcHJlZGljdGl2ZSBmZWF0dXJlIHNlbGVjdGlvbiBwcm92aWRlcyBhIHN0YXRpc3RpY2FsIGNyaXRlcmlvbiB0byBoZWxwIHVzIGlkZW50aWZ5IHRoZSBpbXBvcnRhbnQgdmFyaWFibGVzIChmZWF0dXJlcykuCgpXaGF0IGFib3V0IHRoZSAicHJvamVjdGlvbiBwcmVkaWN0aW9uIiBwYXJ0PyBXZWxsLCB0aGF0J3MgYSBiaXQgbW9yZSBjb21wbGljYXRlZC4gWW91J2xsIG5lZWQgdG8gcmVhZCBQaXJvbmVuIGFuZCBWZWh0YXJpIHRvIGdldCBhbGwgb2YgdGhlIGRldGFpbHMsIGJ1dCBoZXJlIGl0IGlzIGluIGEgbnV0c2hlbGwuCgpXZSBzdGFydCB3aXRoIGEgZnVsbCBtb2RlbCB0aGF0ICh3ZSB0aGluaykgaW5jbHVkZXMgYWxsIG9mIHRoZSBjb3ZhcmlhdGVzIHRoYXQgY291bGQgYmUgcmVsZXZhbnQgaW4gcHJlZGljdGluZyB0aGUgcmVzcG9uc2UuIE5vdyBpbWFnaW5lIHRoYXQgd2UgZXhhbWluZSByZWR1Y2VkIG1vZGVscyB0aGF0IGRvbid0IGluY2x1ZGUgYWxsIG9mIHRoZSBjb3ZhcmlhdGVzLCBfKmJ1dCpfIGluc3RlYWQgb2YgZml0dGluZyB0aG9zZSByZWR1Y2VkIG1vZGVscyB0byB0aGUgZGF0YSBhbmQgY29tcGFyaW5nIHRoZWlyIHBlcmZvcm1hbmNlIGluIHByZWRpY3RpbmcgdGhlIGRhdGEgd2UgZml0IHRoZW0gdG8gZGF0YSBfKnByZWRpY3RlZCpfIGZyb20gdGhlIGZ1bGwgbW9kZWwsIGkuZS4sIGl0cyBwb3N0ZXJpb3IgcHJlZGljdGlvbnMsIGFuZCBmaW5kIHRoZSByZWR1Y2VkIG1vZGVsIHRoYXQgZG9lcyB0aGUgImJlc3Qgam9iIiBvZiBtYXRjaGluZyB0aGUgcG9zdGVyaW9yIHByZWRpY3Rpb25zLiBCeSAiYmVzdCBqb2IiIHdlIG1lYW4gdGhhdCB3ZSBwaWNrIHRoZSBtb2RlbCB3aXRoIHRoZSBzbWFsbGVzdCBudW1iZXIgb2YgY292YXJpYXRlcyB3ZSBjYW4gd2hpbGUga2VlcGluZyB0aGUgcHJlZGljdGl2ZSBwZXJmb3JtYW5jZSBvZiB0aGUgcmVkdWNlZCBtb2RlbCBhYm91dCB0aGUgc2FtZSBhcyB0aGUgZnVsbCBtb2RlbC4KCldoaWxlIGl0IG1heSBzZWVtIHN0cmFuZ2UgdG8gcGljayBjb3ZhcmlhdGVzIGJhc2VkIG9uIGhvdyB3ZWxsIG1vZGVscyBwZXJmb3JtIG9uIF8qcHJlZGljdGVkKl8gcmF0aGVyIHRoYW4gXypvYnNlcnZlZCpfIGRhdGEsIFBpcm9uZW4gYW5kIFZlaHRhcmkgc2hvdyB0aGF0IHRoaXMgYXBwcm9hY2ggd29ya3MgdmVyeSB3ZWxsIHdoZW4gY29tcGFyZWQgd2l0aCBvdGhlciBhbHRlcm5hdGl2ZXMgdGhhdCBoYXZlIGJlZW4gc3VnZ2VzdGVkLiBBbmQgdGhleSd2ZSBwdXQgdG9nZXRoZXIgYSB2ZXJ5IG5pY2UgcGFja2FnZSwgYHByb2pwcmVkYCB0aGF0IG1ha2VzIGl0IGVhc3kgdG8gdXNlIHRoZSBhcHByb2FjaC4KCiMjIFNldHRpbmcgdXAgdGhlIGRhdGEKCkFzIHVzdWFsLCB3ZSBoYXZlIHRvIHN0YXJ0IGJ5IGdlbmVyYXRpbmcgdGhlIGRhdGEuCmBgYHtyIHNldHVwLCB3YXJuaW5nID0gRkFMU0UsIG1lc3NhZ2UgPSBGQUxTRX0KbGlicmFyeSh0aWR5dmVyc2UpCmxpYnJhcnkocmVzaGFwZTIpCmxpYnJhcnkoZ2dwbG90MikKbGlicmFyeShjb3dwbG90KQpsaWJyYXJ5KG12dG5vcm0pCmxpYnJhcnkoY29ycnBsb3QpCgpybShsaXN0ID0gbHMoKSkKYGBgCgpgYGB7cn0KIyMgaW50ZXRjZXB0CiMjCmJldGEwIDwtIDEuMAojIyByZWdyZXNzaW9uIGNvZWZmaWNpZW50cwojIwpiZXRhIDwtIGMoMS4wLCAtMS4wLCAxLjAsIDAuMCwgMC4wLCAwLjAsIDAuMCwgMC4wLCAwLjApCiMjIHBhdHRlcm4gb2YgY29ycmVsYXRpb24gbWF0cml4LCBhbGwgbm9uLXplcm8gZW50cmllcyBhcmUgc2V0IHRvIHNhZW0KIyMgY29ycmVsYXRpb24sIGNvdmFyaWFuY2UgbWF0cml4IGNhbGR1bGF0ZWQgZnJvbSBpbmRpdmlkdWFsIHZhcmlhbmNlcyBhbmQgYSAKIyMgc2luZ2xlIGFzc29jaWF0aW9uIHBhcmFtZXRlciBnb3Zlcm5pbmcgdGhlIG5vbi16ZXJvIGNvcnJlbGF0aW9uIGNvZWZmaWNpZW50cwojIwojIyBOb3RlOiBOb3QganVzdCBhbnkgcGF0dGVybiB3aWxsIHdvcmsgaGVyZS4gVGhlIGNvcnJlbGF0aW9uIG1hdHJpeCBhbmQKIyMgY292YXJpYW5jZSBtYXRyaXggZ2VuZXJhdGVkIGZyb20gdGhpcyBwYXR0ZXJuIG11c3QgYmUgcG9zaXRpdmUgZGVmaW5pdGUuCiMjIElmIHlvdSBjaGFuZ2UgdGhpcyBwYXR0ZXJuLCB5b3UgbWF5IGdldCBhbiBlcnJvciB3aGVuIHlvdSB0cnkgdG8gZ2VuZXJhdGUKIyMgZGF0YSB3aXRoIGEgbm9uLXplcm8gYXNzb2NpYXRpb24gcGFyYW1ldGVyLgojIwpSaG8gPC0gbWF0cml4KG5yb3cgPSA5LCBuY29sID0gLCBieXJvdyA9IFRSVUUsIAogICAgICAgICAgICAgIGRhdGEgPSBjKDEsMCwxLDAsMSwwLDEsMCwxLAogICAgICAgICAgICAgICAgICAgICAgIDAsMSwwLDEsMCwxLDAsMSwwLAogICAgICAgICAgICAgICAgICAgICAgIDEsMCwxLDAsMSwwLDEsMCwxLAogICAgICAgICAgICAgICAgICAgICAgIDAsMSwwLDEsMCwxLDAsMSwwLAogICAgICAgICAgICAgICAgICAgICAgIDEsMCwxLDAsMSwwLDEsMCwxLAogICAgICAgICAgICAgICAgICAgICAgIDAsMSwwLDEsMCwxLDAsMSwwLAogICAgICAgICAgICAgICAgICAgICAgIDEsMCwxLDAsMSwwLDEsMCwxLAogICAgICAgICAgICAgICAgICAgICAgIDAsMSwwLDEsMCwxLDAsMSwwLAogICAgICAgICAgICAgICAgICAgICAgIDEsMCwxLDAsMSwwLDEsMCwxCiAgICAgICAgICAgICAgICAgICAgICAgKSkKIyMgdmVjdG9yIG9mIHN0YW5kYXJkIGRldmlhdGlvbnMgZm9yIGNvdmFyaWF0ZXMKIyMKc2lnbWEgPC0gcmVwKDEsIDkpCgojIyBjb25zdHJ1Y3QgYSBjb3ZhcmlhbmNlIG1hdHJpeCBmcm9tIHRoZSBwYXR0ZXJuLCBzdGFuZGFyZCBkZXZpYXRpb25zLCBhbmQKIyMgb25lIHBhcmFtZXRlciBpbiBbLTEsMV0gdGhhdCBnb3Zlcm5zIHRoZSBtYWduaXR1ZGUgb2Ygbm9uLXplcm8gY29ycmVsYXRpb24KIyMgY29lZmZpY2llbnRzCiMjCiMjIFJobyAtIHRoZSBwYXR0ZXJuIG9mIGFzc29jaWF0aW9ucwojIyBzaWdtYSAtIHRoZSB2ZWN0b3Igb2Ygc3RhbmRhcmQgZGV2aWF0aW9ucwojIyByaG8gLSB0aGUgYXNzb2NpYXRpb24gcGFyYW1ldGVyCiMjCmNvbnN0cnVjdF9TaWdtYSA8LSBmdW5jdGlvbihSaG8sIHNpZ21hLCByaG8pIHsKICAjIyBnZXQgdGhlIGNvcnJlbGF0aW9uIG1hdHJpcwogICMjCiAgUmhvIDwtIFJobypyaG8KICBmb3IgKGkgaW4gMTpuY29sKFJobykpIHsKICAgIFJob1tpLGldIDwtIDEuMAogIH0KICAjIyBub3RpY2UgdGhlIHVzZSBvZiBtYXRyaXggbXVsdGlwbGljYXRpb24KICAjIwogIFNpZ21hIDwtIGRpYWcoc2lnbWEpICUqJSBSaG8gJSolIGRpYWcoc2lnbWEpCiAgcmV0dXJuKFNpZ21hKQp9CgojIyBzZXQgdGhlIHJhbmRvbSBudW1iZXIgc2VlZCBtYW51YWxseSBzbyB0aGF0IGV2ZXJ5IHJ1biBvZiB0aGUgY29kZSB3aWxsIAojIyBwcm9kdWNlIHRoZSBzYW1lIG51bWJlcnMKIyMKc2V0LnNlZWQoMTIzNCkKCm5fc2FtcCA8LSAxMDAKY292X3N0ciA8LSBybXZub3JtKG5fc2FtcCwKICAgICAgICAgICAgICAgICAgIG1lYW4gPSByZXAoMCwgbnJvdyhSaG8pKSwKICAgICAgICAgICAgICAgICAgIHNpZ21hID0gY29uc3RydWN0X1NpZ21hKFJobywgc2lnbWEsIDAuOCkpCgpyZXNpZCA8LSByZXAoMi4wLCBuX3NhbXApCgp5X3N0ciA8LSBybm9ybShucm93KGNvdl9zdHIpLCBtZWFuID0gYmV0YTAgKyBjb3Zfc3RyICUqJSBiZXRhLCBzZCA9IHJlc2lkKQpkYXRfMSA8LSBkYXRhLmZyYW1lKHlfc3RyLCBjb3Zfc3RyLCByZXAoIlN0cm9uZyIsIGxlbmd0aCh5X3N0cikpKQoKY292X3N0ciA8LSBybXZub3JtKG5fc2FtcCwKICAgICAgICAgICAgICAgICAgIG1lYW4gPSByZXAoMCwgbnJvdyhSaG8pKSwKICAgICAgICAgICAgICAgICAgIHNpZ21hID0gY29uc3RydWN0X1NpZ21hKFJobywgc2lnbWEsIDAuOCkpCnlfc3RyIDwtIHJub3JtKG5yb3coY292X3N0ciksIG1lYW4gPSBiZXRhMCArIGNvdl9zdHIgJSolIGJldGEsIHNkID0gcmVzaWQpCmRhdF8yIDwtIGRhdGEuZnJhbWUoeV9zdHIsIGNvdl9zdHIsIHJlcCgiU3Ryb25nIiwgbGVuZ3RoKHlfc3RyKSkpCgpjb2x1bW5fbmFtZXMgPC0gYygieSIsIHBhc3RlKCJ4Iiwgc2VxKDEsIGxlbmd0aChiZXRhKSksIHNlcCA9ICIiKSwgIlNjZW5hcmlvIikKY29sbmFtZXMoZGF0XzEpIDwtIGNvbHVtbl9uYW1lcwpjb2xuYW1lcyhkYXRfMikgPC0gY29sdW1uX25hbWVzCgojIyBzYXZpbmcgcmVzdWx0cyBpbiBzY2FsZSBhbGxvd3MgbWUgdG8gdXNlIHRoZW0gbGF0ZXIgZm9yIHByZWRpY3Rpb24gd2l0aAojIyBuZXcgZGF0YQojIwpzY2FsZV8xIDwtIGxhcHBseShkYXRfMVssIDE6MTBdLCBzY2FsZSkKc2NhbGVfMiA8LSBsYXBwbHkoZGF0XzJbLCAxOjEwXSwgc2NhbGUpCgojIyB3aGVuIGFzc2lnbmluZyB0aGUgc2FtZSBzY2FsaW5nIHRvIGEgZGF0YSBmcmFtZSwgdGhlIHNjYWxpbmcgYXR0cmlidXRlcwojIyBhcmUgbG9zdAojIwpkYXRfMVssIDE6MTBdIDwtIGxhcHBseShkYXRfMVssIDE6MTBdLCBzY2FsZSkKZGF0XzJbLCAxOjEwXSA8LSBsYXBwbHkoZGF0XzJbLCAxOjEwXSwgc2NhbGUpCmBgYAoKIyMgVHJ5aW5nIHByb2plY3Rpb24gCgpTaW5jZSB3ZSdsbCBiZSBydW5uaW5nIHRoZSBwcm9qZWN0aW9uIHByZWRpY3RpdmUgYW5hbHlzaXMgb24gYm90aCBkYXRhIHNldHMsIEkndmUgd3JpdHRlbiBhIHNtYWxsIGZ1bmN0aW9uLCBgcHJvamVjdGlvbl9wcmVkaWN0aW9uKClgIHRoYXQgd2lsbCBydW4gZXZlcnl0aGluZyBmb3IgdXMgYW5kIGNvbGxlY3QgdGhlIHJlc3VsdHMuIE5vdGljZSB0aGF0IHdlIHN0YXJ0IGJ5IGZpdHRpbmcgdGhlIGZ1bGwgbW9kZWwgd2l0aCBbaG9yc2VzaG9lIHByaW9yc10oaHR0cDovL2Rhcndpbi5lZWIudWNvbm4uZWR1L3VuY29tbW9uLWdyb3VuZC9ibG9nLzIwMTkvMDkvMTYvYS1iYXllc2lhbi1hcHByb2FjaC10by12YXJpYWJsZS1zZWxlY3Rpb24tdXNpbmctaG9yc2VzaG9lLXByaW9ycy8pLgpgYGB7ciwgd2FybmluZyA9IEZBTFNFLCBtZXNzYWdlID0gRkFMU0V9CmxpYnJhcnkocnN0YW5hcm0pCmxpYnJhcnkocHJvanByZWQpCmxpYnJhcnkoYmF5ZXNwbG90KQoKb3B0aW9ucyhtYy5jb3JlcyA9IHBhcmFsbGVsOjpkZXRlY3RDb3JlcygpKQoKcHJvamVjdGlvbl9wcmVkaWN0aW9uIDwtIGZ1bmN0aW9uKGRhdCkgewogIG4gPC0gbnJvdyhkYXQpCiAgRCA8LSBuY29sKGRhdFssMjoxMF0pCiAgcDAgPC0gMwogIHRhdTAgPC0gcDAvKEQgLSBwMCkgKiAxL3NxcnQobikKICBwcmlvcl9jb2VmZiA8LSBocyhnbG9iYWxfc2NhbGUgPSB0YXUwLCBzbGFiX3NjYWxlID0gMSkKICBmaXQgPC0gc3Rhbl9nbG0oeSB+IHgxICsgeDIgKyB4MyArIHg0ICsgeDUgKyB4NiArIHg3ICsgeDggKyB4OSwgZGF0YSA9IGRhdCwgCiAgICAgICAgICAgICAgICAgIHByaW9yID0gcHJpb3JfY29lZmYsIHJlZnJlc2ggPSAwKQoKICB2cyA8LSB2YXJzZWwoZml0LCBtZXRob2QgPSAiZm9yd2FyZCIpCiAgY3ZzIDwtIGN2X3ZhcnNlbChmaXQsIG1ldGhvZCA9ICJmb3J3YXJkIiwgdmVyYm9zZSA9IEZBTFNFKQoKICBwcm9qX3NpemUgPC0gc3VnZ2VzdF9zaXplKGN2cykKICBwcm9qIDwtIHByb2plY3QodnMsIG52ID0gcHJval9zaXplLCBucyA9IDIwMDApCiAgbWNtY19pbnRlcnZhbHMoYXMubWF0cml4KHByb2opLAogICAgICAgICAgICAgICAgIHBhcnMgPSBjKCIoSW50ZXJjZXB0KSIsIG5hbWVzKHZzJHZpbmRbMTpwcm9qX3NpemVdKSwgInNpZ21hIikpCgogIHByZWQgPC0gcHJval9saW5wcmVkKHZzLCB4bmV3ID0gZGF0WywgMjoxMF0sIHluZXcgPSBkYXQkeSwgCiAgICAgICAgICAgICAgICAgICAgICAgbnYgPSBwcm9qX3NpemUsIGludGVncmF0ZWQgPSBUUlVFKQogIGZvci5wbG90IDwtIGRhdGEuZnJhbWUoT2JzZXJ2ZWQgPSBkYXQkeSwKICAgICAgICAgICAgICAgICAgICAgICAgIFByZWRpY3RlZCA9IHByZWQkcHJlZCkKICBwIDwtIGdncGxvdChmb3IucGxvdCwgYWVzKHggPSBPYnNlcnZlZCwgeSA9IFByZWRpY3RlZCkpICsgCiAgICBnZW9tX3BvaW50KCkgKyAKICAgIGdlb21fc21vb3RoKG1ldGhvZCA9ICJsbSIpICsKICAgIGdlb21fYWJsaW5lKHNsb3BlID0gMSwgaW50ZXJjZXB0ID0gMCkgKwogICAgbGFicyh4ID0gIk9ic2VydmVkIiwgeSA9ICJQcmVkaWN0ZWQiKQogIHByaW50KHApCgogIHJldHVybihsaXN0KHByb2ogPSBwcm9qLCB2cyA9IHZzLCBjdnMgPSBjdnMsIHByb2pfc2l6ZSA9IHByb2pfc2l6ZSkpCn0KCnN1bW1hcml6ZV9wb3N0ZXJpb3IgPC0gZnVuY3Rpb24oeCwgY3JlZGlibGUgPSAwLjk1LCBkaWdpdHMgPSAzKSB7CiAgbG9fcCA8LSAoMS4wIC0gY3JlZGlibGUpLzIuMAogIGhpX3AgPC0gY3JlZGlibGUgKyBsb19wCiAgY2kgPC0gcXVhbnRpbGUoeCwgYyhsb19wLCBoaV9wKSkKICByZXN1bHRzIDwtIHNwcmludGYoIiUgNS4zZiAoJSA1LjNmLCAlIDUuM2YpXG4iLCBtZWFuKHgpLCBjaVsxXSwgY2lbMl0pCiAgY2F0KHJlc3VsdHMsIHNlcCA9ICIiKQp9CgpzdW1tYXJpemVfcmVzdWx0cyA8LSBmdW5jdGlvbih4LCBjcmVkaWJsZSA9IDAuOTUsIGRpZ2l0cyA9IDMpIHsKICB2YXJzIDwtIGNvbG5hbWVzKGFzLm1hdHJpeCh4KSkKICBmb3IgKGkgaW4gMTpsZW5ndGgodmFycykpIHsKICAgIGxhYmVsIDwtIHNwcmludGYoIiUxMXM6ICIsIHZhcnNbaV0pCiAgICBjYXQobGFiZWwsIHNlcCA9ICIiKQogICAgc3VtbWFyaXplX3Bvc3Rlcmlvcihhcy5tYXRyaXgoeClbLGldKQogIH0KfQoKcHJval8xIDwtIHByb2plY3Rpb25fcHJlZGljdGlvbihkYXRfMSkKcHJval8yIDwtIHByb2plY3Rpb25fcHJlZGljdGlvbihkYXRfMikKc3VtbWFyaXplX3Jlc3VsdHMocHJval8xJHByb2opCnN1bW1hcml6ZV9yZXN1bHRzKHByb2pfMiRwcm9qKQpgYGAKSGVyZSB3ZSBoYXZlIHJlc3VsdHMgdGhhdCBhcmUgcXVpdGUgZGlmZmVyZW50IGZyb20gb25lIGFub3RoZXIuIE5vdCBvbmx5IGlzIHRoZSBjb2VmZmljaWVudCBvbiBgeDNgIHF1aXRlIGRpZmZlcmVudCBpbiB0aGUgdHdvIGRhdGEgc2V0cyxbXjFdIGJ1dCB0aGUgdmFyaWFibGVzIGlkZW50aWZpZWQgYXMgaW1wb3J0YW50IGFyZSBkaWZmZXJlbnQ6IGB4M2AgYW5kIGB4MWAgaW4gYW5hbHlzaXMgb2YgdGhlIGZpcnN0IGRhdGEgc2V0IGFuZCBgeDNgIGFuZCBgeDZgIGluIHRoZSBzZWNvbmQgZGF0YSBzZXQuIFRoYXQgaXNuJ3QgcXVpdGUgYXMgZGlzdHVyYmluZyBhcyBpdCBtaWdodCBmaXJzdCBhcHBlYXIuIExvb2sgYmFjayBhdCB0aGUgcmVzdWx0cyBmcm9tIG91ciByZWdyZXNzaW9uIHdpdGggW2hvcnNlc2hvZSBwcmlvcnNdKGh0dHA6Ly9kYXJ3aW4uZWViLnVjb25uLmVkdS9wYWdlcy92YXJpYWJsZS1zZWxlY3Rpb24vaG9yc2VzaG9lLXByaW9ycy5uYi5odG1sKSBvbiBhbGwgb2YgdGhlIGRhdGEuIFlvdSdsbCBzZWUgdGhhdCBgeDFgIGFuZCBgeDNgIGhhZCBhIGdyZWF0ZXIgbWFnbml0dWRlIGluIGFuYWx5c2lzIG9mIHRoZSBmaXJzdCBkYXRhIHNldCB0aGFuIGFueSBvdGhlciBjb3ZhcmlhdGVzLiBTaW1pbGFybHksIGB4M2AgYW5kIGB4NmAgd2VyZSB0aGUgbW9zdCBpbXBvcnRhbnQgaW4gYW5hbHlzaXMgb2YgdGhlIHNlY29uZCBkYXRhIHNldC4KCiMjIFBpY2tpbmcgdGhlIG51bWJlciBvZiBjb3ZhcmlhdGVzCgpJbiB0aGUgY29kZSBJIHJhbiBhYm92ZSBJIHVzZWQgYHN1Z2dlc3Rfc2l6ZSgpYCB0byBwaWNrIHRoZSBudW1iZXIgb2YgY292YXJpYXRlcyB0byB1c2UuIElmIHlvdSByZWFkIHRoZSBkb2N1bWVudGF0aW9uIGZvciBgc3VnZ2VzdF9zaXplKClgW14yXSB5b3UnbGwgbm90aWNlIHRoaXMgc2VudGVuY2UgaW4gdGhlIERlc2NyaXB0aW9uOgoKPiBJdCBpcyByZWNvbW1lbmRlZCB0aGF0IHRoZSB1c2VyIHN0dWRpZXMgdGhlIHJlc3VsdHMgdmlhIGB2YXJzZWxfcGxvdGAgYW5kL29yIGB2YXJzZWxfc3RhdHNgIGFuZCBtYWtlcyB0aGUgZmluYWwgZGVjaXNpb24gYmFzZWQgb24gd2hhdCBpcyBtb3N0IGFwcHJvcHJpYXRlIGZvciB0aGUgZ2l2ZW4gcHJvYmxlbS4KCkxldCdzIHRha2UgYSBsb29rIGF0IHRoZSByZXN1bHRzIGZyb20gYHZhcnNlbF9wbG90KClgIGZpcnN0LgoKYGBge3J9CnByaW50KHZhcnNlbF9wbG90KHByb2pfMSRjdnMsIHN0YXRzID0gYygiZWxwZCIsICJybXNlIiksIGRlbHRhcyA9IFRSVUUpKQpwcmludCh2YXJzZWxfcGxvdChwcm9qXzIkY3ZzLCBzdGF0cyA9IGMoImVscGQiLCAicm1zZSIpLCBkZWx0YXMgPSBUUlVFKSkKYGBgCmBlbHBkYCBhYmQgYHJtc2VgIGFyZSBtZWFzdXJlcyBvZiBob3cgd2VsbCBhIG1vZGVsIGZpdHMgdGhlIGRhdGEuW14zXSBUaGUgY29tcGxldGUgbW9kZWwgaW4gb3VyIGV4YW1wbGUgaW5jbHVkZXMgOSBwYXJhbWV0ZXJzLiBZb3UgY2FuIHNlZSB0aGF0IGluIHRoZSBmaXJzdCBkYXRhIHNldCBhIG1vZGVsIHdpdGggb25seSAyIGNvdmFyaWF0ZXMgd29ya3MgYWxtb3N0IGFzIHdlbGwgYXMgdGhlIGZ1bGwgbW9kZWwsIGFsdGhvdWdoIHRoZSBtb2RlbCB3aXRoIDMgY292YXJpYXRlcyBsb29rcyBhcyBpZiBpdCBtaWdodCB3b3JrIGEgbGl0dGxlIGJldHRlci4gSW4gdGhlIHNlY29uZCBkYXRhIHNldCwgdGhlcmUgaXMgY2xlYXJseSBubyBhZHZhbnRhZ2UgdG8gaW5jbHVkaW5nIG1vcmUgdGhhbiAyIGNvdmFyaWF0ZXMuCgpUaGUgcGljdHVyZXMgYXJlIHByZXR0eSBjbGVhciwgYnV0IGxldCdzIGxvb2sgYXQgc29tZSB0aGUgZGlmZmVyZW5jZXMgbnVtZXJpY2FsbHkgdXNpbmcgYHZhcnNlbF9zdGF0cygpYC4gV2UnbGwgZm9jdXMgb24gYGVscGRgLgoKYGBge3J9CnN0YXRzXzEgPC0gdmFyc2VsX3N0YXRzKHByb2pfMSRjdnMpICU+JSAKICBtdXRhdGUoZGlmZiA9IGVscGQgLSBsYXN0KGVscGQpLCBwZXJjZGlmZiA9IGRpZmYvbGFzdChlbHBkKSkKc3RhdHNfMiA8LSB2YXJzZWxfc3RhdHMocHJval8yJGN2cykgJT4lIAogIG11dGF0ZShkaWZmID0gZWxwZCAtIGxhc3QoZWxwZCksIHBlcmNkaWZmID0gZGlmZi9sYXN0KGVscGQpKQoKcHJpbnQocm91bmQoc3RhdHNfMSwgMykpCnByaW50KHJvdW5kKHN0YXRzXzIsIDMpKQpgYGAKQnkgZGVmYXVsdCBgc3VnZ2VzdF9zaXplKClgIGlkZW50aWZpZXMgdGhlIHNtYWxsZXN0IG1vZGVsIHdoZXJlIGBlbHBkYCBpcyB3aXRoaW4gb25lIHN0YW5kYXJkIGRldmlhdGlvbiAoYGVscGQuc2VgKSBvZiB0aGUgZnVsbCBtb2RlbC4gSW4gYm90aCBkYXRhIHNldHMgdGhhdCBoYXBwZW5zIHdoZW4gdHdvIGNvdmFyaWF0ZXMgYXJlIGluY2x1ZGVkLiBZb3Ugd2lsbCBub3RpY2UsIGhvd2V2ZXIsIHRoYXQganVzdCBhcyBhIG1vZGVsIHdpdGggMyBwYXJhbWV0ZXJzIGxvb2tlZCBhcyBpZiBpdCBtaWdodCBiZSBhIGxpdHRsZSBiZXR0ZXIgdGhhbiBvbmUgd2l0aCBvbmx5IDIgZm9yIHRoZSBmaXJzdCBkYXRhIHNldCxbXjRdIGhlcmUgdGhlIG1vZGVsIHdpdGggMyBwYXJhbWV0ZXJzIGltcHJvdmVzIHRoZSBtb2RlbCB3aXRoIDIgcGFyYW1ldGVycyBieSAwLjAyOCwgd2hpY2ggaXMgbW9yZSB0aGFuIHR3aWNlIHRoZSBpbXByb3ZlbWVudCBtYWRlIGJ5IGFueSBpbmN1ZGluZyBhbnkgYWRkaXRpb25hbCBwYXJhbWV0ZXJzLiBUaGlzIHN1Z2dlc3RzIHRvIG1lIHRoYXQgaXQgcHJvYmFibHkgd29ydGggbG9va2luZyBhdCB0aGUgcmVzdWx0cyBmcm9tIHRoZSBmaXJzdCBkYXRhIHNldCB3aGVyZSB3ZSBpbmNsdWRlIDMgY292YXJpYXRlcyBpbnN0ZWFkIG9mIDIuCgpgYGB7cn0KcHJval8xXzMgPC0gcHJvamVjdChwcm9qXzEkdnMsIG52ID0gMywgbnMgPSAyMDAwKQpzdW1tYXJpemVfcmVzdWx0cyhwcm9qXzFfMykKYGBgClRoYXQncyByYXRoZXIgaW50ZXJlc3RpbmcuIEl0IHBpY2tlZCBvdXQgdGhlICJyaWdodCIgY29lZmZpY2llbnRzLCBpbiB0aGUgc2Vuc2UgdGhhdCB0aGVyZSB0aGUgb25lcyB3ZSBzZXQgYXMgbm9uLXplcm8gaW4gZ2VuZXJhdGluZyB0aGUgZGF0YSwgYW5kIGl0IGFsc28gZ290IHRoZSAicmlnaHQiIHNpZ25zIGZvciB0aGVtLiBOb3RpY2UgYWxzbyB0aGF0IHRoZSBjb2VmZmljaWVudCBvbiBgeDJgIGlzIHN1YnN0YW50aWFsbHkgbGFyZ2VyIHRoYW4gaXQgaXMgaW4gdGhlIGFuYWx5c2lzIG9mIHRoZSBmdWxsIG1vZGVsIHdpdGggaG9yc2VzaG9lIHByaW9ycyAoYWdhaW4gaWdub3JpbmcgdGhlIHZlcnkgYnJvYWQgb3ZlcmxhcCBpbiBjcmVkaWJsZSBpbnRlcnZhbHMpLgoKV2UgZG9uJ3QgaGF2ZSBhIGdvb2QgcmVhc29uIHRvIGxvb2sgYXQgYSBtb2RlbCB3aXRoIG1vcmUgdGhhbiAyIGNvdmFyaWF0ZXMgaW4gdGhlIHNlY29uZCBkYXRhIHNldCwgYnV0IGxldCdzIHRyeSBpdCB3aXRoIDMgYW5kIHNlZSB3aGF0IGhhcHBlbnMuCgpgYGB7cn0KcHJval8yXzMgPC0gcHJvamVjdChwcm9qXzIkdnMsIG52ID0gMywgbnMgPSAyMDAwKQpzdW1tYXJpemVfcmVzdWx0cyhwcm9qXzJfMykKYGBgClRoYXQgZGlkIGJyaW5nIGluIGB4MWAsIHdoaWNoIHdlIGtub3cgb3VnaHQgdG8gYmUgdGhlcmUuIEJ1dCBhZ2FpbiwgdGhlIG9ubHkgcmVhc29uIHdlIGhhdmUgdG8gcHJlZmVyIHRoaXMgbW9kZWwgZ2l2ZW4gdGhlc2UgZGF0YSBpcyBvdXIgcHJpb3Iga25vd2xlZGdlIHRoYXQgYHgzYCBvdWdodCB0byBiZSBpbiB0aGUgbW9kZWwuIAoKIyMgQ29uY2x1c2lvbnMKCklmIHlvdSBjYW4gcnVuIGEgcmVncmVzc2lvbiBtb2RlbCBpbiBgc3Rhbl9nbG0oKWAgb3IgYHN0YW5fZ2xtZXIoKWAsIHlvdSBjYW4gZWFzaWx5IHVzZSBob3JzZXNob2UgcHJpb3JzLiBBbmQgaWYgeW91IGNhbiBkbyB0aGF0LCB5b3UgY2FuIGVhc2lseSB1c2UgcHJvamVjdGlvbiBwcmVkaWN0aXZlIHZhcmlhYmxlIHNlbGVjdGlvbiB0byBpZGVudHRpZnkgdGhlICJtb3N0IGltcG9ydGFudCIgc2V0IG9mIGNvdmFyaWF0ZXMuIFRoZSBzaW1wbGUgZXhhbXBsZSBoZXJlIHN1Z2dlc3RzIGEgY291cGxlIG9mIHRoaW5nczoKCjEuIEJlIHN1cmUgdG8gZXhhbWluZSBgdmFyc2VsX3Bsb3QoKWAsIGB2YXJzZWxfc3RhdHMoKWAsIG9yIGJvdGggcmF0aGVyIHRoYW4gc2ltcGx5IHJlbHlpbmcgb24gYHN1Z2dlc3Rfc2l6ZSgpYCB0byB0ZWxsIHlvdSBob3cgbWFueSBjb3ZhcmlhdGVzIHRvIGluY2x1ZGUuCgoyLiBJdCdzIHByb2JhYmx5IHdvcnRoIHlvdXIgdGltZSB0byB0YWtlIGEgbG9vayBhdCBtb2RlbHMgdGhhdCBpbmNsdWRlIG9uZSBvciB0d28gbW9yZSBwYXJhbWV0ZXJzIHRoYW4gd2hhdCB5b3VyIGV4YW1pbmF0aW9uIG9mIGB2YXJzZWxfcGxvdCgpYCBhbmQgYHZhcnNlbF9zdGF0cygpYCBzdWdnZXN0LiBZb3UnbGwgbmVlZCB0byBicmluZyBpbiB5b3VyIHN1YmplY3QgbWF0dGVyIGtub3dsZWRnZSBoZXJlLFteNV0gYnV0IHRoZXJlIG1pZ2h0IGJlIGEgY292YXJpYXRlIHdpdGggYW4gYXNzb2NpYXRpb24gd29ydGggY29uc2lkZXJpbmcgYmFzZWQgb24gc3VidGFudGl2ZSBncm91bmRzLCBldmVuIGlmIGl0J3Mgbm90ICJzaWduaWZpY2FudCIgaW4gYSBjb252ZW50aW9uYWwgc2Vuc2UuCgpUaGF0IGxhc3QgcG9pbnQgcmFpc2VzIGEgbXVjaCBicm9hZGVyIGlzc3VlIHRoYXQgSSdsbCByZXR1cm4gdG8gaW4gdGhlIGxhc3QgYmxvZyBwb3N0IGluIHRoaXMgc2VyaWVzLCB3aGVyZSBJIHRyeSB0byBzdW1tYXJpemUgdGhlIGxlc3NvbnMgd2UndmUgbGVhcm5lZCBhbmQgcHJvdmlkZSBzb21lIGdlbmVyYWwgZ3VpZGVsaW5lcy4gVGhlIGRpZmZlcmVuY2UgYmV0d2VlbiAic2lnbmlmaWNhbnQiIGFuZCAibm90IFNpZ25pZmljYW50IiBpcyBub3QgaXRzZWxmIHN0YXRpc3RpY2FsbHkgc2lnbmlmaWNhbnQuW142XQoKW14xXTogSSdtIGJlaW5nIHNsb3BweSBoZXJlLiBJIHNob3VsZCBoYXZlIHNhaWQgInRoZSBfcG9zdGVyaW9yIG1lYW5zXyBmb3IgYHgzYCBhcmUgcXVpdGUgZGlmZmVyZW50LCBidXQgdGhlIHBvc3RlcmlvciBkaXN0cmlidXRpb24gb3ZlcmxhcHMgcXVpdGUgYnJvYWRseS4iCgpbXjJdOiBZb3Ugc2hvdWxkIGFsd2F5cywgX2Fsd2F5c18sIF8qKmFsd2F5cyoqXyByZWFkIHRoZSBkb2N1bWVudGF0aW9uLgoKW14zXTogYHJtc2VgIGlzIHRoZSBlYXNpZXN0IHRvIHVuZGVyc3RhbmQuIEl0J3MgdGhlICJyb290IG1lYW4gc3F1YXJlZCBlcnJvciIsIGkuZS4sIHRoZSBzcXVhcmUgcm9vdCBvZiB0aGUgYXZlcmFnZSBzcXVhcmVkIGVycm9yIChzcXVhcmVkIHJlc2lkdWFsKSBhY3Jvc3MgYWxsIG9ic2VydmF0aW9ucy4gSWYgJHhfaSQgaXMgdGhlICRpJHRoIG9ic2VydmF0aW9uIGFuZCAkXG11X2kkIGlzIHRoZSBwcmVkaWN0aW9uIGZvciB0aGF0IG9ic2VydmF0aW9uLCB0aGVuIAokJApybXNlID0gXHNxcnR7XHN1bV9pICh4X2kgLSBcbXVfaSleMn0gXHF1YWQgLgokJApgcm1zZWAgaXMgYW4gYXBwcm9wcmlhdGUgbWVhc3VyZSBvbmx5IGZvciBtb2RlbHMgd2hlcmUgdGhlIHJlc2lkdWFsIHZhcmlhbmNlIGlzIG5vcm1hbGx5IGRpc3RyaWJ1dGVkLiBgZWxwZGAgaXMgYSBtb3JlIGdlbmVyYWwgbWVhc3VyZSB0aGF0IGNhbiBiZSB1c2VkIHdpdGggYW55IG1vZGVsIHRoYXQgYHJzdGFuYXJtYCBmaXRzLgoKW140XTogU28gbG9uZyBhcyB5b3UgcGF5IGF0dGVudGlvbiBvbmx5IHRvIHRoZSBwb3N0ZXJpb3IgbWVhbnMgYW5kIG5vdCB0aGVpciBzdGFuZGFyZCBlcnJvcnMuCgpbXjVdOiBTaW5jZSB5b3Ugd29uJ3Qga25vdyB0aGUgdHJ1dGggYXMgd2UgZGlkIGhlcmUuCgpbXjZdOiBUaGF0J3MgdGhlIHRpdGxlIG9mIGEgcGFwZXIgYnkgQW5kcmV3IEdlbG1hbiBhbmQgSGFsIFN0ZXJuIGluIF9UaGUgQW1lcmljYW4gU3RhdGlzdGljaWFuXy4gZG9pOiBbMTAuMTE5OC8wMDAzMTMwMDZYMTUyNjQ5XShodHRwczovL2RvaS5vcmcvMTAuMTE5OC8wMDAzMTMwMDZYMTUyNjQ5KQ==