Using the Lasso to select covariates

The problem of identifying covariates that should be included in a regression has been studied for a long time. More than two decades ago, Rob Tibshirani introduced the Lasso to let the data tell us which covariates to include. If you want the details, including how the Lasso has been generalized, refer to the Wikipedia page. For our purposes, it’s enough for you to know that for linear regression fit by least squares, the Lasso simply includes a constraint on the regression coefficients.

What is the Lasso?

To make that constraint clear, let’s write out explicitly what the least squares criterion is:

\[ \mbox{min}_{\beta}\left\{\frac{1}{N}\sum_i\left(y_i - (\beta_0 + \sum_j\beta_jx_{ij})\right)^2\right\} \]

In words this simply means that we find the set of \(\beta\)s that minimize the difference between our predictions and the observed data. The Lasso does that too, but it adds a constraint on the set of \(\beta\)s, namely

\[ \sum_j |\beta_j| < \lambda \quad , \]

where \(\lambda\) is a parameter that determines the maximum influence of all covariates on the response variable when all of the individual influences are combined. To incorporate the constraint into the algorithm we use this criterion:

\[ \mbox{min}_{\beta}\left\{\frac{1}{N}\sum_i\left(y_i - (\beta_0 + \sum_j\beta_jx_{ij})\right)^2 + \lambda\sum_j|\beta_j|\right\} \]

In a simple least squares regression we don’t care how big any of the \(\beta\)s are individually and we don’t care about how big their overall magnitude is. In using the Lasso we are making the assumption that not only are we unwilling to accept the idea that any individual \(\beta\) is large, but we are also unwilling for any combination of them to be large either. The effect of this, as we’ll see is not only to limit the magnitude of individual regression coefficients, but also to keep some of them close to zero.

Trying out the Lasso

As usual, our first step is to regenerate the data we’ve been playing with.

library(tidyverse)
library(reshape2)
library(ggplot2)
library(cowplot)
library(mvtnorm)
library(corrplot)

rm(list = ls())

## intetcept
##
beta0 <- 1.0
## regression coefficients
##
beta <- c(1.0, -1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0)
## pattern of correlation matrix, all non-zero entries are set to saem
## correlation, covariance matrix caldulated from individual variances and a 
## single association parameter governing the non-zero correlation coefficients
##
## Note: Not just any pattern will work here. The correlation matrix and
## covariance matrix generated from this pattern must be positive definite.
## If you change this pattern, you may get an error when you try to generate
## data with a non-zero association parameter.
##
Rho <- matrix(nrow = 9, ncol = , byrow = TRUE, 
              data = c(1,0,1,0,1,0,1,0,1,
                       0,1,0,1,0,1,0,1,0,
                       1,0,1,0,1,0,1,0,1,
                       0,1,0,1,0,1,0,1,0,
                       1,0,1,0,1,0,1,0,1,
                       0,1,0,1,0,1,0,1,0,
                       1,0,1,0,1,0,1,0,1,
                       0,1,0,1,0,1,0,1,0,
                       1,0,1,0,1,0,1,0,1
                       ))
## vector of standard deviations for covariates
##
sigma <- rep(1, 9)

## construct a covariance matrix from the pattern, standard deviations, and
## one parameter in [-1,1] that governs the magnitude of non-zero correlation
## coefficients
##
## Rho - the pattern of associations
## sigma - the vector of standard deviations
## rho - the association parameter
##
construct_Sigma <- function(Rho, sigma, rho) {
  ## get the correlation matris
  ##
  Rho <- Rho*rho
  for (i in 1:ncol(Rho)) {
    Rho[i,i] <- 1.0
  }
  ## notice the use of matrix multiplication
  ##
  Sigma <- diag(sigma) %*% Rho %*% diag(sigma)
  return(Sigma)
}

## set the random number seed manually so that every run of the code will 
## produce the same numbers
##
set.seed(1234)

n_samp <- 100
cov_str <- rmvnorm(n_samp,
                   mean = rep(0, nrow(Rho)),
                   sigma = construct_Sigma(Rho, sigma, 0.8))

resid <- rep(2.0, n_samp)

y_str <- rnorm(nrow(cov_str), mean = beta0 + cov_str %*% beta, sd = resid)
dat_1 <- data.frame(y_str, cov_str, rep("Strong", length(y_str)))

cov_str <- rmvnorm(n_samp,
                   mean = rep(0, nrow(Rho)),
                   sigma = construct_Sigma(Rho, sigma, 0.8))
y_str <- rnorm(nrow(cov_str), mean = beta0 + cov_str %*% beta, sd = resid)
dat_2 <- data.frame(y_str, cov_str, rep("Strong", length(y_str)))

column_names <- c("y", paste("x", seq(1, length(beta)), sep = ""), "Scenario")
colnames(dat_1) <- column_names
colnames(dat_2) <- column_names

## saving results in scale allows me to use them later for prediction with
## new data
##
scale_1 <- lapply(dat_1[, 1:10], scale)
scale_2 <- lapply(dat_2[, 1:10], scale)

## when assigning the same scaling to a data frame, the scaling attributes
## are lost
##
dat_1[, 1:10] <- lapply(dat_1[, 1:10], scale)
dat_2[, 1:10] <- lapply(dat_2[, 1:10], scale)

OK. Now that we have the data, let’s try the lasso and see what we get.¹

Fitting the lasso

The first thing we have to do is to find a value for \(\lambda\). A good approach is to cv.glmnet(), which will split the data set into test and training data set and use cross-validation to identify the best value.² Since we’re going to do this for both data sets, we start by writing a function that will do it all for us. In this function, I holdout some of the data, by default 20 percent, to illustrate how well the model performs on within sample predictions.

library(glmnet)

lasso <- function(dat, title, holdout = 0.2) {
  x_vars <- model.matrix(y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9, 
                         data = dat)[, -1]
  y_var <- dat$y
  
  ## select the training set (a random sample from 50% of the rows)
  ##
  train <- sample(1:nrow(x_vars), nrow(x_vars)*(1.0 - holdout))
  ## construct the text set
  ##
  test <- setdiff(1:nrow(x_vars), train)
  
  ## use a series of different lambda's that cv.glmnet() picks to identify
  ## the best one
  ##
  cv_output <- cv.glmnet(x_vars[train, ], y_var[train])
  
  ## now predict values for the test set and the training set so that we can
  ## compare observations with predictions for both of them
  ## 
  predict_train <- predict(cv_output, x_vars[train, ], s = "lambda.min")
  predict_train_mat <- cbind(y_var[train], predict_train, "Training")
  if (holdout > 0) {
    predict_test <- predict(cv_output, x_vars[test, ], s = "lambda.min")
    predict_test_mat <- cbind(y_var[test], predict_test, "Test")
    predict <- rbind(predict_train_mat, predict_test_mat)
  } else {
    predict <- predict_train_mat
  }
  
  colnames(predict) <- c("Observed", "Predicted", "Data set")
  predict <- as.data.frame(predict)
  predict$Observed <- as.numeric(as.character(predict$Observed))
  predict$Predicted <- as.numeric(as.character(predict$Predicted))
  
  if (holdout > 0) {
    p <- ggplot(predict, aes(x = Observed, y = Predicted, 
                             color = `Data set`, fill = `Data set`))
  } else {
    p <- ggplot(predict, aes(x = Observed, y = Predicted))
  }
  p <- p + geom_point() +
    geom_abline(slope = 1.0, intercept = 0.0) + 
    geom_smooth(aes(group = 1), method = "lm")
    ggtitle(title)
  print(p)

  return(cv_output)
}

## set seed to allow results to be reproduced
##
set.seed(1234)

lasso_1 <- lasso(dat_1, "Data set 1")

lasso_2 <- lasso(dat_2, "Data set 2")

You should notice a couple of things so far:

The training set does a good job of predicting the test data.³
The predicted values are, on average, less than the observed values.⁴

Comparing coefficient estimates

Since we’ve fit a linear regression as part of fitting the lasso, we can also get a report on the regression coefficients identified as part of the regression, but first, let’s refit the lasso to the entire data set.

lasso_1 <- lasso(dat_1, "Data set 1", holdout = 0.0)

lasso_2 <- lasso(dat_2, "Data set 2", holdout = 0.0)


round(coef(lasso_1, s = "lambda.min"), 3)

10 x 1 sparse Matrix of class "dgCMatrix"
                 1
(Intercept)  0.000
x1           0.239
x2          -0.083
x3           0.306
x4          -0.043
x5           .    
x6          -0.052
x7           .    
x8           .    
x9           0.107

round(coef(lasso_2, s = "lambda.min"), 3)

10 x 1 sparse Matrix of class "dgCMatrix"
                 1
(Intercept)  0.000
x1           0.182
x2          -0.186
x3           0.417
x4           .    
x5           .    
x6          -0.235
x7           .    
x8           .    
x9           .

Comparing predictions

As we’ve done in the past, let’s compare out of sample predictions from the two data sets.⁵

new_data <- data.frame(x1 = 4.0, x2 = 4.0, x3 = 4.0,
                       x4 = 4.0, x5 = 4.0, x6 = 4.0,
                       x7 = 4.0, x8 = 4.0, x9 = 4.0)

## re-scale the new data by subtracting mean and dividing by standard deviation
##
new_data_1 <- (new_data - lapply(scale_1[2:10], attr, "scaled:center")) /
               lapply(scale_1[2:10], attr, "scaled:scale")
new_data_2 <- (new_data - lapply(scale_2[2:10], attr, "scaled:center")) /
               lapply(scale_2[2:10], attr, "scaled:scale")
  
predict_1 <- predict(lasso_1, as.matrix(new_data_1), s = "lambda.min")
predict_2 <- predict(lasso_2, as.matrix(new_data_2), s = "lambda.min")

cat("Prediction from data set 1: ", round(predict_1, 3), "\n",
    "  True answer:              ", beta0 + as.matrix(new_data_1) %*% beta, "\n",
    "Prediction from data set 2: ", round(predict_2, 3), "\n",
    "  True answer:              ", beta0 + as.matrix(new_data_2) %*% beta, "\n", 
    sep = "")

Prediction from data set 1: 2.05
  True answer:              5.583683
Prediction from data set 2: 0.927
  True answer:              5.711435

Those predictions don’t seem quite as different as they did in the past, but they’re both a long way from the true answers. (The true answers differ because of the different scaling of the two data sets.) So here we have a case where the out of sample predictions seem fairly stable, which is good, but they’re a long way off, which is bad.

Closing thoughts

So the Lasso is promising in some ways, but beyond the usual challenges associated with out of sample extrapolation, there is another problem that limits its usefulness. You’ll notice that the results from coef() don’t include a standard error. That’s because Lasso estimates can be very biased, meaning that the standard error may not be a good estimate of how reliable a regression coefficient is.⁶

The Lasso is the simplest example of what’s known as regularized regression. glmnet() can fit logistic, Poisson, and Cox proportional hazard models as well as simple linear models, and by varying the alpha parameter in the call to glmnet() you can get a result anywhere from ridge regresssion (with alpha = 0) to the Lasso (with alpha = 1). Values of alpha between 0 and 1 correspond to an “elastic net.” See the notes Trevor Hastie and Junyiang Qian developed for more details.⁷

If you’re paying close attention, you’ll notice one small difference between this version of the data generating code and what we’ve used before. I used scale() to center each of the data columns on 0 and set the standard deviation in each column to 1. I haven’t done that in previous notebooks, since I knew that all of the covariates were generated on the same scale. You’ll want to do it with real data most of the time to make sure that the regression coefficients are on a comparable scale. As an exercise, you might want to go back to some of the earlier notebooks and try the examples with scaled data.↩
See https://www.rstatisticsblog.com/data-science-in-action/lasso-regression/ for details. The code here follows what you’ll find there very closely.↩
You could check that more formally by calculating the \(R^2\) for the test data set and comparing it to the training set. I haven’t done that, but eyeballing it suggests that they’d be pretty close.↩
The black line is a 1:1 line. The slope of the regression of Predicted on Observed is less than 1. In fact, if you run lm() on Predicted vs. Observed, you’ll find that the slope is 0.415 for data set 1 and 0.514, which are precisely the multiple R-squared from the corresponding multiple regressions.↩
The predict() method for our lasso model has a slightly different interface than we’ve seen before.↩
See p. 18 of the documentation for penalized(). Here’s a link to the documentation on CRAN: https://cran.r-project.org/web/packages/penalized/vignettes/penalized.pdf ↩
https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html ↩

LS0tCnRpdGxlOiAiVXNpbmcgdGhlIExhc3NvIHRvIHNlbGVjdCBjb3ZhcmlhdGVzIgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKLS0tCgpUaGUgcHJvYmxlbSBvZiBpZGVudGlmeWluZyBjb3ZhcmlhdGVzIHRoYXQgc2hvdWxkIGJlIGluY2x1ZGVkIGluIGEgcmVncmVzc2lvbiBoYXMgYmVlbiBzdHVkaWVkIGZvciBhIGxvbmcgdGltZS4gTW9yZSB0aGFuIHR3byBkZWNhZGVzIGFnbywgUm9iIFRpYnNoaXJhbmkgaW50cm9kdWNlZCB0aGUgW0xhc3NvXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9MYXNzb18oc3RhdGlzdGljcykpIHRvIGxldCB0aGUgZGF0YSB0ZWxsIHVzIHdoaWNoIGNvdmFyaWF0ZXMgdG8gaW5jbHVkZS4gSWYgeW91IHdhbnQgdGhlIGRldGFpbHMsIGluY2x1ZGluZyBob3cgdGhlIExhc3NvIGhhcyBiZWVuIGdlbmVyYWxpemVkLCByZWZlciB0byB0aGUgV2lraXBlZGlhIHBhZ2UuIEZvciBvdXIgcHVycG9zZXMsIGl0J3MgZW5vdWdoIGZvciB5b3UgdG8ga25vdyB0aGF0IGZvciBsaW5lYXIgcmVncmVzc2lvbiBmaXQgYnkgbGVhc3Qgc3F1YXJlcywgdGhlIExhc3NvIHNpbXBseSBpbmNsdWRlcyBhIGNvbnN0cmFpbnQgb24gdGhlIHJlZ3Jlc3Npb24gY29lZmZpY2llbnRzLiAKCiMjIFdoYXQgaXMgdGhlIExhc3NvPwoKVG8gbWFrZSB0aGF0IGNvbnN0cmFpbnQgY2xlYXIsIGxldCdzIHdyaXRlIG91dCBleHBsaWNpdGx5IHdoYXQgdGhlIGxlYXN0IHNxdWFyZXMgY3JpdGVyaW9uIGlzOgoKJCQKXG1ib3h7bWlufV97XGJldGF9XGxlZnRce1xmcmFjezF9e059XHN1bV9pXGxlZnQoeV9pIC0gKFxiZXRhXzAgKyBcc3VtX2pcYmV0YV9qeF97aWp9KVxyaWdodCleMlxyaWdodFx9CiQkCgpJbiB3b3JkcyB0aGlzIHNpbXBseSBtZWFucyB0aGF0IHdlIGZpbmQgdGhlIHNldCBvZiAkXGJldGEkcyB0aGF0IG1pbmltaXplIHRoZSBkaWZmZXJlbmNlIGJldHdlZW4gb3VyIHByZWRpY3Rpb25zIGFuZCB0aGUgb2JzZXJ2ZWQgZGF0YS4gVGhlIExhc3NvIGRvZXMgdGhhdCB0b28sIGJ1dCBpdCBhZGRzIGEgY29uc3RyYWludCBvbiB0aGUgc2V0IG9mICRcYmV0YSRzLCBuYW1lbHkKCiQkClxzdW1faiB8XGJldGFfanwgPCBcbGFtYmRhIFxxdWFkICwKJCQKCndoZXJlICRcbGFtYmRhJCBpcyBhIHBhcmFtZXRlciB0aGF0IGRldGVybWluZXMgdGhlIG1heGltdW0gaW5mbHVlbmNlIG9mIGFsbCBjb3ZhcmlhdGVzIG9uIHRoZSByZXNwb25zZSB2YXJpYWJsZSB3aGVuIGFsbCBvZiB0aGUgaW5kaXZpZHVhbCBpbmZsdWVuY2VzIGFyZSBjb21iaW5lZC4gVG8gaW5jb3Jwb3JhdGUgdGhlIGNvbnN0cmFpbnQgaW50byB0aGUgYWxnb3JpdGhtIHdlIHVzZSB0aGlzIGNyaXRlcmlvbjoKCiQkClxtYm94e21pbn1fe1xiZXRhfVxsZWZ0XHtcZnJhY3sxfXtOfVxzdW1faVxsZWZ0KHlfaSAtIChcYmV0YV8wICsgXHN1bV9qXGJldGFfanhfe2lqfSlccmlnaHQpXjIgKyBcbGFtYmRhXHN1bV9qfFxiZXRhX2p8XHJpZ2h0XH0KJCQKCgpJbiBhIHNpbXBsZSBsZWFzdCBzcXVhcmVzIHJlZ3Jlc3Npb24gd2UgZG9uJ3QgY2FyZSBob3cgYmlnIGFueSBvZiB0aGUgJFxiZXRhJHMgYXJlIGluZGl2aWR1YWxseSBhbmQgd2UgZG9uJ3QgY2FyZSBhYm91dCBob3cgYmlnIHRoZWlyIG92ZXJhbGwgbWFnbml0dWRlIGlzLiBJbiB1c2luZyB0aGUgTGFzc28gd2UgYXJlIG1ha2luZyB0aGUgYXNzdW1wdGlvbiB0aGF0IG5vdCBvbmx5IGFyZSB3ZSB1bndpbGxpbmcgdG8gYWNjZXB0IHRoZSBpZGVhIHRoYXQgYW55IGluZGl2aWR1YWwgJFxiZXRhJCBpcyBsYXJnZSwgYnV0IHdlIGFyZSBhbHNvIHVud2lsbGluZyBmb3IgYW55IGNvbWJpbmF0aW9uIG9mIHRoZW0gdG8gYmUgbGFyZ2UgZWl0aGVyLiBUaGUgZWZmZWN0IG9mIHRoaXMsIGFzIHdlJ2xsIHNlZSBpcyBub3Qgb25seSB0byBsaW1pdCB0aGUgbWFnbml0dWRlIG9mIGluZGl2aWR1YWwgcmVncmVzc2lvbiBjb2VmZmljaWVudHMsIGJ1dCBhbHNvIHRvIGtlZXAgc29tZSBvZiB0aGVtIGNsb3NlIHRvIHplcm8uCgojIyBUcnlpbmcgb3V0IHRoZSBMYXNzbwoKQXMgdXN1YWwsIG91ciBmaXJzdCBzdGVwIGlzIHRvIHJlZ2VuZXJhdGUgdGhlIGRhdGEgd2UndmUgYmVlbiBwbGF5aW5nIHdpdGguCmBgYHtyIHNldHVwLCB3YXJuaW5nID0gRkFMU0UsIG1lc3NhZ2UgPSBGQUxTRX0KbGlicmFyeSh0aWR5dmVyc2UpCmxpYnJhcnkocmVzaGFwZTIpCmxpYnJhcnkoZ2dwbG90MikKbGlicmFyeShjb3dwbG90KQpsaWJyYXJ5KG12dG5vcm0pCmxpYnJhcnkoY29ycnBsb3QpCgpybShsaXN0ID0gbHMoKSkKYGBgCgpgYGB7cn0KIyMgaW50ZXRjZXB0CiMjCmJldGEwIDwtIDEuMAojIyByZWdyZXNzaW9uIGNvZWZmaWNpZW50cwojIwpiZXRhIDwtIGMoMS4wLCAtMS4wLCAxLjAsIDAuMCwgMC4wLCAwLjAsIDAuMCwgMC4wLCAwLjApCiMjIHBhdHRlcm4gb2YgY29ycmVsYXRpb24gbWF0cml4LCBhbGwgbm9uLXplcm8gZW50cmllcyBhcmUgc2V0IHRvIHNhZW0KIyMgY29ycmVsYXRpb24sIGNvdmFyaWFuY2UgbWF0cml4IGNhbGR1bGF0ZWQgZnJvbSBpbmRpdmlkdWFsIHZhcmlhbmNlcyBhbmQgYSAKIyMgc2luZ2xlIGFzc29jaWF0aW9uIHBhcmFtZXRlciBnb3Zlcm5pbmcgdGhlIG5vbi16ZXJvIGNvcnJlbGF0aW9uIGNvZWZmaWNpZW50cwojIwojIyBOb3RlOiBOb3QganVzdCBhbnkgcGF0dGVybiB3aWxsIHdvcmsgaGVyZS4gVGhlIGNvcnJlbGF0aW9uIG1hdHJpeCBhbmQKIyMgY292YXJpYW5jZSBtYXRyaXggZ2VuZXJhdGVkIGZyb20gdGhpcyBwYXR0ZXJuIG11c3QgYmUgcG9zaXRpdmUgZGVmaW5pdGUuCiMjIElmIHlvdSBjaGFuZ2UgdGhpcyBwYXR0ZXJuLCB5b3UgbWF5IGdldCBhbiBlcnJvciB3aGVuIHlvdSB0cnkgdG8gZ2VuZXJhdGUKIyMgZGF0YSB3aXRoIGEgbm9uLXplcm8gYXNzb2NpYXRpb24gcGFyYW1ldGVyLgojIwpSaG8gPC0gbWF0cml4KG5yb3cgPSA5LCBuY29sID0gLCBieXJvdyA9IFRSVUUsIAogICAgICAgICAgICAgIGRhdGEgPSBjKDEsMCwxLDAsMSwwLDEsMCwxLAogICAgICAgICAgICAgICAgICAgICAgIDAsMSwwLDEsMCwxLDAsMSwwLAogICAgICAgICAgICAgICAgICAgICAgIDEsMCwxLDAsMSwwLDEsMCwxLAogICAgICAgICAgICAgICAgICAgICAgIDAsMSwwLDEsMCwxLDAsMSwwLAogICAgICAgICAgICAgICAgICAgICAgIDEsMCwxLDAsMSwwLDEsMCwxLAogICAgICAgICAgICAgICAgICAgICAgIDAsMSwwLDEsMCwxLDAsMSwwLAogICAgICAgICAgICAgICAgICAgICAgIDEsMCwxLDAsMSwwLDEsMCwxLAogICAgICAgICAgICAgICAgICAgICAgIDAsMSwwLDEsMCwxLDAsMSwwLAogICAgICAgICAgICAgICAgICAgICAgIDEsMCwxLDAsMSwwLDEsMCwxCiAgICAgICAgICAgICAgICAgICAgICAgKSkKIyMgdmVjdG9yIG9mIHN0YW5kYXJkIGRldmlhdGlvbnMgZm9yIGNvdmFyaWF0ZXMKIyMKc2lnbWEgPC0gcmVwKDEsIDkpCgojIyBjb25zdHJ1Y3QgYSBjb3ZhcmlhbmNlIG1hdHJpeCBmcm9tIHRoZSBwYXR0ZXJuLCBzdGFuZGFyZCBkZXZpYXRpb25zLCBhbmQKIyMgb25lIHBhcmFtZXRlciBpbiBbLTEsMV0gdGhhdCBnb3Zlcm5zIHRoZSBtYWduaXR1ZGUgb2Ygbm9uLXplcm8gY29ycmVsYXRpb24KIyMgY29lZmZpY2llbnRzCiMjCiMjIFJobyAtIHRoZSBwYXR0ZXJuIG9mIGFzc29jaWF0aW9ucwojIyBzaWdtYSAtIHRoZSB2ZWN0b3Igb2Ygc3RhbmRhcmQgZGV2aWF0aW9ucwojIyByaG8gLSB0aGUgYXNzb2NpYXRpb24gcGFyYW1ldGVyCiMjCmNvbnN0cnVjdF9TaWdtYSA8LSBmdW5jdGlvbihSaG8sIHNpZ21hLCByaG8pIHsKICAjIyBnZXQgdGhlIGNvcnJlbGF0aW9uIG1hdHJpcwogICMjCiAgUmhvIDwtIFJobypyaG8KICBmb3IgKGkgaW4gMTpuY29sKFJobykpIHsKICAgIFJob1tpLGldIDwtIDEuMAogIH0KICAjIyBub3RpY2UgdGhlIHVzZSBvZiBtYXRyaXggbXVsdGlwbGljYXRpb24KICAjIwogIFNpZ21hIDwtIGRpYWcoc2lnbWEpICUqJSBSaG8gJSolIGRpYWcoc2lnbWEpCiAgcmV0dXJuKFNpZ21hKQp9CgojIyBzZXQgdGhlIHJhbmRvbSBudW1iZXIgc2VlZCBtYW51YWxseSBzbyB0aGF0IGV2ZXJ5IHJ1biBvZiB0aGUgY29kZSB3aWxsIAojIyBwcm9kdWNlIHRoZSBzYW1lIG51bWJlcnMKIyMKc2V0LnNlZWQoMTIzNCkKCm5fc2FtcCA8LSAxMDAKY292X3N0ciA8LSBybXZub3JtKG5fc2FtcCwKICAgICAgICAgICAgICAgICAgIG1lYW4gPSByZXAoMCwgbnJvdyhSaG8pKSwKICAgICAgICAgICAgICAgICAgIHNpZ21hID0gY29uc3RydWN0X1NpZ21hKFJobywgc2lnbWEsIDAuOCkpCgpyZXNpZCA8LSByZXAoMi4wLCBuX3NhbXApCgp5X3N0ciA8LSBybm9ybShucm93KGNvdl9zdHIpLCBtZWFuID0gYmV0YTAgKyBjb3Zfc3RyICUqJSBiZXRhLCBzZCA9IHJlc2lkKQpkYXRfMSA8LSBkYXRhLmZyYW1lKHlfc3RyLCBjb3Zfc3RyLCByZXAoIlN0cm9uZyIsIGxlbmd0aCh5X3N0cikpKQoKY292X3N0ciA8LSBybXZub3JtKG5fc2FtcCwKICAgICAgICAgICAgICAgICAgIG1lYW4gPSByZXAoMCwgbnJvdyhSaG8pKSwKICAgICAgICAgICAgICAgICAgIHNpZ21hID0gY29uc3RydWN0X1NpZ21hKFJobywgc2lnbWEsIDAuOCkpCnlfc3RyIDwtIHJub3JtKG5yb3coY292X3N0ciksIG1lYW4gPSBiZXRhMCArIGNvdl9zdHIgJSolIGJldGEsIHNkID0gcmVzaWQpCmRhdF8yIDwtIGRhdGEuZnJhbWUoeV9zdHIsIGNvdl9zdHIsIHJlcCgiU3Ryb25nIiwgbGVuZ3RoKHlfc3RyKSkpCgpjb2x1bW5fbmFtZXMgPC0gYygieSIsIHBhc3RlKCJ4Iiwgc2VxKDEsIGxlbmd0aChiZXRhKSksIHNlcCA9ICIiKSwgIlNjZW5hcmlvIikKY29sbmFtZXMoZGF0XzEpIDwtIGNvbHVtbl9uYW1lcwpjb2xuYW1lcyhkYXRfMikgPC0gY29sdW1uX25hbWVzCgojIyBzYXZpbmcgcmVzdWx0cyBpbiBzY2FsZSBhbGxvd3MgbWUgdG8gdXNlIHRoZW0gbGF0ZXIgZm9yIHByZWRpY3Rpb24gd2l0aAojIyBuZXcgZGF0YQojIwpzY2FsZV8xIDwtIGxhcHBseShkYXRfMVssIDE6MTBdLCBzY2FsZSkKc2NhbGVfMiA8LSBsYXBwbHkoZGF0XzJbLCAxOjEwXSwgc2NhbGUpCgojIyB3aGVuIGFzc2lnbmluZyB0aGUgc2FtZSBzY2FsaW5nIHRvIGEgZGF0YSBmcmFtZSwgdGhlIHNjYWxpbmcgYXR0cmlidXRlcwojIyBhcmUgbG9zdAojIwpkYXRfMVssIDE6MTBdIDwtIGxhcHBseShkYXRfMVssIDE6MTBdLCBzY2FsZSkKZGF0XzJbLCAxOjEwXSA8LSBsYXBwbHkoZGF0XzJbLCAxOjEwXSwgc2NhbGUpCmBgYAoKT0suIE5vdyB0aGF0IHdlIGhhdmUgdGhlIGRhdGEsIGxldCdzIHRyeSB0aGUgbGFzc28gYW5kIHNlZSB3aGF0IHdlIGdldC5bXjFdIAoKIyMjIEZpdHRpbmcgdGhlIGxhc3NvCgpUaGUgZmlyc3QgdGhpbmcgd2UgaGF2ZSB0byBkbyBpcyB0byBmaW5kIGEgdmFsdWUgZm9yICRcbGFtYmRhJC4gQSBnb29kIGFwcHJvYWNoIGlzIHRvIGBjdi5nbG1uZXQoKWAsIHdoaWNoIHdpbGwgc3BsaXQgdGhlIGRhdGEgc2V0IGludG8gdGVzdCBhbmQgdHJhaW5pbmcgZGF0YSBzZXQgYW5kIHVzZSBjcm9zcy12YWxpZGF0aW9uIHRvIGlkZW50aWZ5IHRoZSBiZXN0IHZhbHVlLlteMl0gU2luY2Ugd2UncmUgZ29pbmcgdG8gZG8gdGhpcyBmb3IgYm90aCBkYXRhIHNldHMsIHdlIHN0YXJ0IGJ5IHdyaXRpbmcgYSBmdW5jdGlvbiB0aGF0IHdpbGwgZG8gaXQgYWxsIGZvciB1cy4gSW4gdGhpcyBmdW5jdGlvbiwgSSBob2xkb3V0IHNvbWUgb2YgdGhlIGRhdGEsIGJ5IGRlZmF1bHQgMjAgcGVyY2VudCwgdG8gaWxsdXN0cmF0ZSBob3cgd2VsbCB0aGUgbW9kZWwgcGVyZm9ybXMgb24gd2l0aGluIHNhbXBsZSBwcmVkaWN0aW9ucy4KCmBgYHtyLCB3YXJuaW5nID0gRkFMU0UsIG1lc3NhZ2UgPSBGQUxTRX0KbGlicmFyeShnbG1uZXQpCgpsYXNzbyA8LSBmdW5jdGlvbihkYXQsIHRpdGxlLCBob2xkb3V0ID0gMC4yKSB7CiAgeF92YXJzIDwtIG1vZGVsLm1hdHJpeCh5IH4geDEgKyB4MiArIHgzICsgeDQgKyB4NSArIHg2ICsgeDcgKyB4OCArIHg5LCAKICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGEgPSBkYXQpWywgLTFdCiAgeV92YXIgPC0gZGF0JHkKICAKICAjIyBzZWxlY3QgdGhlIHRyYWluaW5nIHNldCAoYSByYW5kb20gc2FtcGxlIGZyb20gNTAlIG9mIHRoZSByb3dzKQogICMjCiAgdHJhaW4gPC0gc2FtcGxlKDE6bnJvdyh4X3ZhcnMpLCBucm93KHhfdmFycykqKDEuMCAtIGhvbGRvdXQpKQogICMjIGNvbnN0cnVjdCB0aGUgdGV4dCBzZXQKICAjIwogIHRlc3QgPC0gc2V0ZGlmZigxOm5yb3coeF92YXJzKSwgdHJhaW4pCiAgCiAgIyMgdXNlIGEgc2VyaWVzIG9mIGRpZmZlcmVudCBsYW1iZGEncyB0aGF0IGN2LmdsbW5ldCgpIHBpY2tzIHRvIGlkZW50aWZ5CiAgIyMgdGhlIGJlc3Qgb25lCiAgIyMKICBjdl9vdXRwdXQgPC0gY3YuZ2xtbmV0KHhfdmFyc1t0cmFpbiwgXSwgeV92YXJbdHJhaW5dKQogIAogICMjIG5vdyBwcmVkaWN0IHZhbHVlcyBmb3IgdGhlIHRlc3Qgc2V0IGFuZCB0aGUgdHJhaW5pbmcgc2V0IHNvIHRoYXQgd2UgY2FuCiAgIyMgY29tcGFyZSBvYnNlcnZhdGlvbnMgd2l0aCBwcmVkaWN0aW9ucyBmb3IgYm90aCBvZiB0aGVtCiAgIyMgCiAgcHJlZGljdF90cmFpbiA8LSBwcmVkaWN0KGN2X291dHB1dCwgeF92YXJzW3RyYWluLCBdLCBzID0gImxhbWJkYS5taW4iKQogIHByZWRpY3RfdHJhaW5fbWF0IDwtIGNiaW5kKHlfdmFyW3RyYWluXSwgcHJlZGljdF90cmFpbiwgIlRyYWluaW5nIikKICBpZiAoaG9sZG91dCA+IDApIHsKICAgIHByZWRpY3RfdGVzdCA8LSBwcmVkaWN0KGN2X291dHB1dCwgeF92YXJzW3Rlc3QsIF0sIHMgPSAibGFtYmRhLm1pbiIpCiAgICBwcmVkaWN0X3Rlc3RfbWF0IDwtIGNiaW5kKHlfdmFyW3Rlc3RdLCBwcmVkaWN0X3Rlc3QsICJUZXN0IikKICAgIHByZWRpY3QgPC0gcmJpbmQocHJlZGljdF90cmFpbl9tYXQsIHByZWRpY3RfdGVzdF9tYXQpCiAgfSBlbHNlIHsKICAgIHByZWRpY3QgPC0gcHJlZGljdF90cmFpbl9tYXQKICB9CiAgCiAgY29sbmFtZXMocHJlZGljdCkgPC0gYygiT2JzZXJ2ZWQiLCAiUHJlZGljdGVkIiwgIkRhdGEgc2V0IikKICBwcmVkaWN0IDwtIGFzLmRhdGEuZnJhbWUocHJlZGljdCkKICBwcmVkaWN0JE9ic2VydmVkIDwtIGFzLm51bWVyaWMoYXMuY2hhcmFjdGVyKHByZWRpY3QkT2JzZXJ2ZWQpKQogIHByZWRpY3QkUHJlZGljdGVkIDwtIGFzLm51bWVyaWMoYXMuY2hhcmFjdGVyKHByZWRpY3QkUHJlZGljdGVkKSkKICAKICBpZiAoaG9sZG91dCA+IDApIHsKICAgIHAgPC0gZ2dwbG90KHByZWRpY3QsIGFlcyh4ID0gT2JzZXJ2ZWQsIHkgPSBQcmVkaWN0ZWQsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbG9yID0gYERhdGEgc2V0YCwgZmlsbCA9IGBEYXRhIHNldGApKQogIH0gZWxzZSB7CiAgICBwIDwtIGdncGxvdChwcmVkaWN0LCBhZXMoeCA9IE9ic2VydmVkLCB5ID0gUHJlZGljdGVkKSkKICB9CiAgcCA8LSBwICsgZ2VvbV9wb2ludCgpICsKICAgIGdlb21fYWJsaW5lKHNsb3BlID0gMS4wLCBpbnRlcmNlcHQgPSAwLjApICsgCiAgICBnZW9tX3Ntb290aChhZXMoZ3JvdXAgPSAxKSwgbWV0aG9kID0gImxtIikKICAgIGdndGl0bGUodGl0bGUpCiAgcHJpbnQocCkKCiAgcmV0dXJuKGN2X291dHB1dCkKfQoKIyMgc2V0IHNlZWQgdG8gYWxsb3cgcmVzdWx0cyB0byBiZSByZXByb2R1Y2VkCiMjCnNldC5zZWVkKDEyMzQpCgpsYXNzb18xIDwtIGxhc3NvKGRhdF8xLCAiRGF0YSBzZXQgMSIpCmxhc3NvXzIgPC0gbGFzc28oZGF0XzIsICJEYXRhIHNldCAyIikKYGBgCgpZb3Ugc2hvdWxkIG5vdGljZSBhIGNvdXBsZSBvZiB0aGluZ3Mgc28gZmFyOgoKMS4gVGhlIHRyYWluaW5nIHNldCBkb2VzIGEgZ29vZCBqb2Igb2YgcHJlZGljdGluZyB0aGUgdGVzdCBkYXRhLlteM10KMi4gVGhlIHByZWRpY3RlZCB2YWx1ZXMgYXJlLCBvbiBhdmVyYWdlLCBsZXNzIHRoYW4gdGhlIG9ic2VydmVkIHZhbHVlcy5bXjRdCgojIyBDb21wYXJpbmcgY29lZmZpY2llbnQgZXN0aW1hdGVzCgpTaW5jZSB3ZSd2ZSBmaXQgYSBsaW5lYXIgcmVncmVzc2lvbiBhcyBwYXJ0IG9mIGZpdHRpbmcgdGhlIGxhc3NvLCB3ZSBjYW4gYWxzbyBnZXQgYSByZXBvcnQgb24gdGhlIHJlZ3Jlc3Npb24gY29lZmZpY2llbnRzIGlkZW50aWZpZWQgYXMgcGFydCBvZiB0aGUgcmVncmVzc2lvbiwgYnV0IGZpcnN0LCBsZXQncyByZWZpdCB0aGUgbGFzc28gdG8gdGhlIGVudGlyZSBkYXRhIHNldC4KCmBgYHtyfQpsYXNzb18xIDwtIGxhc3NvKGRhdF8xLCAiRGF0YSBzZXQgMSIsIGhvbGRvdXQgPSAwLjApCmxhc3NvXzIgPC0gbGFzc28oZGF0XzIsICJEYXRhIHNldCAyIiwgaG9sZG91dCA9IDAuMCkKCnJvdW5kKGNvZWYobGFzc29fMSwgcyA9ICJsYW1iZGEubWluIiksIDMpCnJvdW5kKGNvZWYobGFzc29fMiwgcyA9ICJsYW1iZGEubWluIiksIDMpCmBgYAoKIyMgQ29tcGFyaW5nIHByZWRpY3Rpb25zCgpBcyB3ZSd2ZSBkb25lIGluIHRoZSBwYXN0LCBsZXQncyBjb21wYXJlIG91dCBvZiBzYW1wbGUgcHJlZGljdGlvbnMgZnJvbSB0aGUgdHdvIGRhdGEgc2V0cy5bXjVdCgpgYGB7cn0KbmV3X2RhdGEgPC0gZGF0YS5mcmFtZSh4MSA9IDQuMCwgeDIgPSA0LjAsIHgzID0gNC4wLAogICAgICAgICAgICAgICAgICAgICAgIHg0ID0gNC4wLCB4NSA9IDQuMCwgeDYgPSA0LjAsCiAgICAgICAgICAgICAgICAgICAgICAgeDcgPSA0LjAsIHg4ID0gNC4wLCB4OSA9IDQuMCkKCiMjIHJlLXNjYWxlIHRoZSBuZXcgZGF0YSBieSBzdWJ0cmFjdGluZyBtZWFuIGFuZCBkaXZpZGluZyBieSBzdGFuZGFyZCBkZXZpYXRpb24KIyMKbmV3X2RhdGFfMSA8LSAobmV3X2RhdGEgLSBsYXBwbHkoc2NhbGVfMVsyOjEwXSwgYXR0ciwgInNjYWxlZDpjZW50ZXIiKSkgLwogICAgICAgICAgICAgICBsYXBwbHkoc2NhbGVfMVsyOjEwXSwgYXR0ciwgInNjYWxlZDpzY2FsZSIpCm5ld19kYXRhXzIgPC0gKG5ld19kYXRhIC0gbGFwcGx5KHNjYWxlXzJbMjoxMF0sIGF0dHIsICJzY2FsZWQ6Y2VudGVyIikpIC8KICAgICAgICAgICAgICAgbGFwcGx5KHNjYWxlXzJbMjoxMF0sIGF0dHIsICJzY2FsZWQ6c2NhbGUiKQogIApwcmVkaWN0XzEgPC0gcHJlZGljdChsYXNzb18xLCBhcy5tYXRyaXgobmV3X2RhdGFfMSksIHMgPSAibGFtYmRhLm1pbiIpCnByZWRpY3RfMiA8LSBwcmVkaWN0KGxhc3NvXzIsIGFzLm1hdHJpeChuZXdfZGF0YV8yKSwgcyA9ICJsYW1iZGEubWluIikKCmNhdCgiUHJlZGljdGlvbiBmcm9tIGRhdGEgc2V0IDE6ICIsIHJvdW5kKHByZWRpY3RfMSwgMyksICJcbiIsCiAgICAiICBUcnVlIGFuc3dlcjogICAgICAgICAgICAgICIsIGJldGEwICsgYXMubWF0cml4KG5ld19kYXRhXzEpICUqJSBiZXRhLCAiXG4iLAogICAgIlByZWRpY3Rpb24gZnJvbSBkYXRhIHNldCAyOiAiLCByb3VuZChwcmVkaWN0XzIsIDMpLCAiXG4iLAogICAgIiAgVHJ1ZSBhbnN3ZXI6ICAgICAgICAgICAgICAiLCBiZXRhMCArIGFzLm1hdHJpeChuZXdfZGF0YV8yKSAlKiUgYmV0YSwgIlxuIiwgCiAgICBzZXAgPSAiIikKYGBgCgpUaG9zZSBwcmVkaWN0aW9ucyBkb24ndCBzZWVtIHF1aXRlIGFzIGRpZmZlcmVudCBhcyB0aGV5IGRpZCBpbiB0aGUgcGFzdCwgYnV0IHRoZXkncmUgYm90aCBhIGxvbmcgd2F5IGZyb20gdGhlIHRydWUgYW5zd2Vycy4gKFRoZSB0cnVlIGFuc3dlcnMgZGlmZmVyIGJlY2F1c2Ugb2YgdGhlIGRpZmZlcmVudCBzY2FsaW5nIG9mIHRoZSB0d28gZGF0YSBzZXRzLikgU28gaGVyZSB3ZSBoYXZlIGEgY2FzZSB3aGVyZSB0aGUgb3V0IG9mIHNhbXBsZSBwcmVkaWN0aW9ucyBzZWVtIGZhaXJseSBzdGFibGUsIHdoaWNoIGlzIGdvb2QsIGJ1dCB0aGV5J3JlIGEgbG9uZyB3YXkgb2ZmLCB3aGljaCBpcyBiYWQuCgojIyBDbG9zaW5nIHRob3VnaHRzCgpTbyB0aGUgTGFzc28gaXMgcHJvbWlzaW5nIGluIHNvbWUgd2F5cywgYnV0IGJleW9uZCB0aGUgdXN1YWwgY2hhbGxlbmdlcyBhc3NvY2lhdGVkIHdpdGggb3V0IG9mIHNhbXBsZSBleHRyYXBvbGF0aW9uLCB0aGVyZSBpcyBhbm90aGVyIHByb2JsZW0gdGhhdCBsaW1pdHMgaXRzIHVzZWZ1bG5lc3MuIFlvdSdsbCBub3RpY2UgdGhhdCB0aGUgcmVzdWx0cyBmcm9tIGBjb2VmKClgIGRvbid0IGluY2x1ZGUgYSBzdGFuZGFyZCBlcnJvci4gVGhhdCdzIGJlY2F1c2UgTGFzc28gZXN0aW1hdGVzIGNhbiBiZSB2ZXJ5IGJpYXNlZCwgbWVhbmluZyB0aGF0IHRoZSBzdGFuZGFyZCBlcnJvciBtYXkgbm90IGJlIGEgZ29vZCBlc3RpbWF0ZSBvZiBob3cgcmVsaWFibGUgYSByZWdyZXNzaW9uIGNvZWZmaWNpZW50IGlzLlteNl0gCgpUaGUgTGFzc28gaXMgdGhlIHNpbXBsZXN0IGV4YW1wbGUgb2Ygd2hhdCdzIGtub3duIGFzIF9yZWd1bGFyaXplZCByZWdyZXNzaW9uXy4gYGdsbW5ldCgpYCBjYW4gZml0IGxvZ2lzdGljLCBQb2lzc29uLCBhbmQgQ294IHByb3BvcnRpb25hbCBoYXphcmQgbW9kZWxzIGFzIHdlbGwgYXMgc2ltcGxlIGxpbmVhciBtb2RlbHMsIGFuZCBieSB2YXJ5aW5nIHRoZSBgYWxwaGFgIHBhcmFtZXRlciBpbiB0aGUgY2FsbCB0byBgZ2xtbmV0KClgIHlvdSBjYW4gZ2V0IGEgcmVzdWx0IGFueXdoZXJlIGZyb20gcmlkZ2UgcmVncmVzc3Npb24gKHdpdGggYGFscGhhYCA9IDApIHRvIHRoZSBMYXNzbyAod2l0aCBgYWxwaGFgID0gMSkuIFZhbHVlcyBvZiBgYWxwaGFgIGJldHdlZW4gMCBhbmQgMSBjb3JyZXNwb25kIHRvIGFuICJlbGFzdGljIG5ldC4iIFNlZSB0aGUgbm90ZXMgVHJldm9yIEhhc3RpZSBhbmQgSnVueWlhbmcgUWlhbiBkZXZlbG9wZWQgZm9yIG1vcmUgZGV0YWlscy5bXjddCgoKW14xXTogSWYgeW91J3JlIHBheWluZyBjbG9zZSBhdHRlbnRpb24sIHlvdSdsbCBub3RpY2Ugb25lIHNtYWxsIGRpZmZlcmVuY2UgYmV0d2VlbiB0aGlzIHZlcnNpb24gb2YgdGhlIGRhdGEgZ2VuZXJhdGluZyBjb2RlIGFuZCB3aGF0IHdlJ3ZlIHVzZWQgYmVmb3JlLiBJIHVzZWQgYHNjYWxlKClgIHRvIGNlbnRlciBlYWNoIG9mIHRoZSBkYXRhIGNvbHVtbnMgb24gMCBhbmQgc2V0IHRoZSBzdGFuZGFyZCBkZXZpYXRpb24gaW4gZWFjaCBjb2x1bW4gdG8gMS4gSSBoYXZlbid0IGRvbmUgdGhhdCBpbiBwcmV2aW91cyBub3RlYm9va3MsIHNpbmNlIEkga25ldyB0aGF0IGFsbCBvZiB0aGUgY292YXJpYXRlcyB3ZXJlIGdlbmVyYXRlZCBvbiB0aGUgc2FtZSBzY2FsZS4gWW91J2xsIHdhbnQgdG8gZG8gaXQgd2l0aCByZWFsIGRhdGEgbW9zdCBvZiB0aGUgdGltZSB0byBtYWtlIHN1cmUgdGhhdCB0aGUgcmVncmVzc2lvbiBjb2VmZmljaWVudHMgYXJlIG9uIGEgY29tcGFyYWJsZSBzY2FsZS4gQXMgYW4gZXhlcmNpc2UsIHlvdSBtaWdodCB3YW50IHRvIGdvIGJhY2sgdG8gc29tZSBvZiB0aGUgZWFybGllciBub3RlYm9va3MgYW5kIHRyeSB0aGUgZXhhbXBsZXMgd2l0aCBzY2FsZWQgZGF0YS4KClteMl06IFNlZSBbaHR0cHM6Ly93d3cucnN0YXRpc3RpY3NibG9nLmNvbS9kYXRhLXNjaWVuY2UtaW4tYWN0aW9uL2xhc3NvLXJlZ3Jlc3Npb24vXShodHRwczovL3d3dy5yc3RhdGlzdGljc2Jsb2cuY29tL2RhdGEtc2NpZW5jZS1pbi1hY3Rpb24vbGFzc28tcmVncmVzc2lvbi8pIGZvciBkZXRhaWxzLiBUaGUgY29kZSBoZXJlIGZvbGxvd3Mgd2hhdCB5b3UnbGwgZmluZCB0aGVyZSB2ZXJ5IGNsb3NlbHkuCgpbXjNdOiBZb3UgY291bGQgY2hlY2sgdGhhdCBtb3JlIGZvcm1hbGx5IGJ5IGNhbGN1bGF0aW5nIHRoZSAkUl4yJCBmb3IgdGhlIHRlc3QgZGF0YSBzZXQgYW5kIGNvbXBhcmluZyBpdCB0byB0aGUgdHJhaW5pbmcgc2V0LiBJIGhhdmVuJ3QgZG9uZSB0aGF0LCBidXQgZXllYmFsbGluZyBpdCBzdWdnZXN0cyB0aGF0IHRoZXknZCBiZSBwcmV0dHkgY2xvc2UuCgpbXjRdOiBUaGUgYmxhY2sgbGluZSBpcyBhIDE6MSBsaW5lLiBUaGUgc2xvcGUgb2YgdGhlIHJlZ3Jlc3Npb24gb2YgUHJlZGljdGVkIG9uIE9ic2VydmVkIGlzIGxlc3MgdGhhbiAxLiBJbiBmYWN0LCBpZiB5b3UgcnVuIGBsbSgpYCBvbiBgUHJlZGljdGVkYCB2cy4gYE9ic2VydmVkYCwgeW91J2xsIGZpbmQgdGhhdCB0aGUgc2xvcGUgaXMgMC40MTUgZm9yIGRhdGEgc2V0IDEgYW5kIDAuNTE0LCB3aGljaCBhcmUgcHJlY2lzZWx5IHRoZSBtdWx0aXBsZSBSLXNxdWFyZWQgZnJvbSB0aGUgY29ycmVzcG9uZGluZyBtdWx0aXBsZSByZWdyZXNzaW9ucy4KClteNV06IFRoZSBgcHJlZGljdCgpYCBtZXRob2QgZm9yIG91ciBsYXNzbyBtb2RlbCBoYXMgYSBzbGlnaHRseSBkaWZmZXJlbnQgaW50ZXJmYWNlIHRoYW4gd2UndmUgc2VlbiBiZWZvcmUuCgpbXjZdOiBTZWUgcC4gMTggb2YgdGhlIGRvY3VtZW50YXRpb24gZm9yIGBwZW5hbGl6ZWQoKWAuIEhlcmUncyBhIGxpbmsgdG8gdGhlIGRvY3VtZW50YXRpb24gb24gQ1JBTjogW2h0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy9wZW5hbGl6ZWQvdmlnbmV0dGVzL3BlbmFsaXplZC5wZGZdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy9wZW5hbGl6ZWQvdmlnbmV0dGVzL3BlbmFsaXplZC5wZGYpCgpbXjddOiBbaHR0cHM6Ly93ZWIuc3RhbmZvcmQuZWR1L35oYXN0aWUvZ2xtbmV0L2dsbW5ldF9hbHBoYS5odG1sXShodHRwczovL3dlYi5zdGFuZm9yZC5lZHUvfmhhc3RpZS9nbG1uZXQvZ2xtbmV0X2FscGhhLmh0bWwp