Genomic prediction with brms

Recall that our basic approach to genomic prediction is to do a multiple regression of a trait on a genotypes at a set of loci that we’ve identified. In lecture we used stan_lm() and stan_glm() from rstanarm, because we simulated the data without any population structure.

For this week’s lab, we’re going to reuse the gypsy moth (Lymantria dispar) data that we used last week. Since it has population structure, we need to take account of it. Rather than writing another script in Stan, we’re going to use brms which makes the whole process a lot easier.1

Fitting the model

Here’s an example of an analysis of mass using the five loci that had the largest magnitude of effects in our analysis last week. You’ll notice that I’m using a different relatedness file than last week. It’s calculated in the same way (as you’ll see below), but brms expects to see it in a different format from the one used in the Stan program I wrote.

library(tidyverse)
Registered S3 methods overwritten by 'dbplyr':
  method         from
  print.tbl_lazy     
  print.tbl_sql      
── Attaching packages ───────────────────────────── tidyverse 1.3.1 ──
✓ ggplot2 3.3.5     ✓ purrr   0.3.4
✓ tibble  3.1.6     ✓ dplyr   1.0.7
✓ tidyr   1.1.4     ✓ stringr 1.4.0
✓ readr   2.1.0     ✓ forcats 0.5.1
── Conflicts ──────────────────────────────── tidyverse_conflicts() ──
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()
library(brms)
Loading required package: Rcpp
Registered S3 methods overwritten by 'htmltools':
  method               from         
  print.html           tools:rstudio
  print.shiny.tag      tools:rstudio
  print.shiny.tag.list tools:rstudio
Registered S3 method overwritten by 'htmlwidgets':
  method           from         
  print.htmlwidget tools:rstudio
Loading 'brms' package (version 2.16.1). Useful instructions
can be found by typing help('brms'). A more detailed introduction
to the package is available through vignette('brms_overview').

Attaching package: ‘brms’

The following object is masked from ‘package:stats’:

    ar
rm(list = ls())

options(mc.cores = parallel::detectCores())

dat <- read_csv("gypsymoth.csv")
Rows: 141 Columns: 223
── Column specification ──────────────────────────────────────────────
Delimiter: ","
chr   (2): pops, sample
dbl (221): Mass, PD, TDT, X2517, X2840, X2841, X3915, X4293, X4612...

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
A <- read_tsv("gypsymoth_relatedness_for_brms.txt",
              show_col_types = FALSE)

fit_Mass <- brm(Mass ~ X34862 + X29507 + X44522 + X40856 + X89362 + 
                  (1|gr(sample, cov = A)),
                data = dat,
                family = gaussian(),
                set_prior(horseshoe(df = 3, par_ratio = 0.5)),
                data2 = list(A = A),
                iter = 5000,
                refresh = 0)  
Compiling Stan program...
Start sampling
There were 39 divergent transitions after warmup. See
http://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup
to find out why this is a problem and how to eliminate them.Examine the pairs() plot to diagnose sampling problems
summary(fit_Mass)
There were 39 divergent transitions after warmup. Increasing adapt_delta above 0.8 may help. See http://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup
 Family: gaussian 
  Links: mu = identity; sigma = identity 
Formula: Mass ~ X34862 + X29507 + X44522 + X40856 + X89362 + (1 | gr(sample, cov = A)) 
   Data: dat (Number of observations: 141) 
  Draws: 4 chains, each with iter = 5000; warmup = 2500; thin = 1;
         total post-warmup draws = 10000

Group-Level Effects: 
~sample (Number of levels: 141) 
              Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
sd(Intercept)     0.03      0.01     0.00     0.05 1.00     2168
              Tail_ESS
sd(Intercept)     3131

Population-Level Effects: 
          Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept     0.27      0.01     0.24     0.30 1.00     6179     7830
X34862        0.01      0.01    -0.00     0.03 1.00     4261     6014
X29507       -0.00      0.01    -0.02     0.00 1.00     8320     8516
X44522       -0.01      0.01    -0.02     0.00 1.00     5094     7892
X40856        0.01      0.01    -0.00     0.03 1.00     3565     7232
X89362        0.01      0.01    -0.00     0.02 1.00     4938     8532

Family Specific Parameters: 
      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sigma     0.05      0.00     0.05     0.06 1.00     5583     6213

Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).

The (1|gr(sample, cov = A)) is where the relatedness comes in. sample identifies the particular individual, and A is the genetic relatedness matrix. set_prior(horseshoe(df = 3, par_ratio = 0.5)) sets the horseshoe prior as described in the lecture notes, where par_ratio is our prior guess at what fraction of the loci might be important. You’ll notice that if we compare the estimates from this analysis with those from the locus-by-locus analysis last week, they seem to be smaller in magnitude but to have the same sign.

Assessing model fit

Now that we’ve estimated allelic effects at these five loci we can ask how well they account for variation in mass. We’ll do so by estimating \(R^2\), the proportion of variation in mass that is accounted for by the model. The \(R^2\) we’ll calculate is a bit different from those you may have encountered with other multiple regression models, because it’s a Bayesian \(R^2\). We have an estimate both of the mean \(R^2\) and credible intervals for that estimate.

bayes_R2(fit_Mass)
    Estimate Est.Error       Q2.5     Q97.5
R2 0.1986823 0.0832772 0.04699403 0.3664717

So with only five loci we’ve accounted for about 20% of the variation, although there is quite a bit of uncertainty around that estimate.

We can also visualize the fit by plotting the observed values against those predicted from the model.

predicted_Mass <- predict(fit_Mass)

library(ggplot2)

for_plot <- tibble(Predicted = predicted_Mass[, "Estimate"],
                   Observed = dat$Mass)
p <- ggplot(for_plot, aes(x = Predicted, y = Observed)) +
  geom_point() +
  geom_abline(slope = 1, intercept = 0, color = "red", 
              linetype = "dashed") +
  theme_bw()
p

You can see that there’s one outlier on mass that we probably should have eliminated before analyzing the data, but we won’t worry about that for our purposes.

How much of the prediction is due to background genetic variation

When we predicted the trait in the last bit of code, we included all of the information about how individuals are related to one another. It’s quite easy to leave that information out to see how well a pure genomic prediction would work with these data.

predict_reduced <- predict(fit_Mass, 
                           re_formula = NA)

With that prediction in hand, it’s easy to compare the predictions to one another. “Full” in the following plot refers to the model that includes relatedness information. “Reduced” refers to the model that doesn’t include it.

for_plot <- tibble(Predicted = c(predicted_Mass[, "Estimate"],
                                 predict_reduced[, "Estimate"]),
                   Observed = c(dat$Mass,
                                dat$Mass),
                   Source = c(rep("Full", nrow(dat)), 
                               rep("Reduced", nrow(dat)))
                   )
p <- ggplot(for_plot, aes(x = Predicted, y = Observed)) +
  geom_point() +
  geom_abline(slope = 1, intercept = 0, color = "red", 
              linetype = "dashed") +
  theme_bw() +
  facet_wrap(~ Source)
p

It looks as if points in the reduced model are not quite as tightly grouped around the 1:1 line as they are in the full model, but let’s compare the \(R^2\) of the two models to see if that impression is right.

cat("Full model...\n")
Full model...
bayes_R2(fit_Mass)
    Estimate Est.Error       Q2.5     Q97.5
R2 0.1986823 0.0832772 0.04699403 0.3664717
cat("Reduced model...\n")
Reduced model...
bayes_R2(fit_Mass, 
         re_formula = NA)
     Estimate  Est.Error         Q2.5     Q97.5
R2 0.08954445 0.06011212 0.0004490187 0.2186064

Our visual impression was right the full model accounts for about 20% of the variation, while the reduced model accounts for only about 9%. In other words, about half of the predictive power from the model is coming from relatedness among individuals rather than the allelic effects themselves.

Problems and questions

Data files

  1. Perform a genomic prediction analysis for mass, PD, and TDT using the 10 loci having the largest estimated allelic effects on each trait.

  2. Assess how well the model accounts for variation in each trait.

  3. Assess how much additional information the allelic effects provide in predicting each trait beyond the information already provided in the relationship matrix.

  4. BONUS QUESTION (UNGRADED): How might including the relationship matrix in this kind of analysis limit your ability to use the allelic effects estimated in these populations to predict traits in populations that weren’t included in the analysis.

How I prepared the data for analysis

See last week’s lab for the data sources. Here’s the code, which is nearly identical to what I used last week.

library(tidyverse)
library(popkin)

rm(list = ls())

genos <- read_tsv("gm_genotypes_012_final_5122017.txt",
                  col_names = FALSE,
                  na = "-1",
                  show_col_types = FALSE)
phenos <- read_csv("gm_phenotypes.csv",
                   show_col_types = FALSE)

## keep only loci scored in more than 100 individuals
##
n_scored <- apply(genos, 2, sum, na.rm = TRUE)
genos <- genos[, n_scored > 100]

## keep only individuals scored at more than 4000 loci
##
n_scored <- apply(genos, 1, sum, na.rm = TRUE)
genos <- genos[n_scored > 4000, ]
phenos <- phenos[n_scored > 4000, ]
## and exclude any remaining loci where one or more individuals isn't scored
##
genos <- genos[, !is.na(apply(genos, 2, sum))]

## strip off the individual ids to identify populations
##
pops <- gsub("_.*", "", phenos$sample)

## bind the poopulation, phenotype, and genotype information together
##
dat <- cbind(pops, phenos, genos)

## estimate the kinship among individuals
##
A <- popkin(t(as.matrix(genos)), 
              subpops = pops)
colnames(A) <- dat$sample
rownames(A) <- dat$sample
write_tsv(as.data.frame(A),
          file = "gypsymoth_relatedness_for_brms.txt")

  1. It is probably also a lot more reliable than anything I’d write, since the author is a real expert on Bayesian inference, and I am merely a reasonably competent user.↩︎

LS0tCnRpdGxlOiAiR2Vub21pYyBwcmVkaWN0aW9uIGluIGd5cHN5IG1vdGhzIgpvdXRwdXQ6IAogIGh0bWxfbm90ZWJvb2s6CiAgICB0b2M6IHllcwogICAgdG9jX2Zsb2F0OiB0cnVlCiAgICBjc3M6ICJSLW5vdGVib29rLmNzcyIKLS0tCgojIEdlbm9taWMgcHJlZGljdGlvbiB3aXRoIGBicm1zYAoKUmVjYWxsIHRoYXQgb3VyIGJhc2ljIGFwcHJvYWNoIHRvIGdlbm9taWMgcHJlZGljdGlvbiBpcyB0byBkbyBhIG11bHRpcGxlIHJlZ3Jlc3Npb24gb2YgYSB0cmFpdCBvbiBhIGdlbm90eXBlcyBhdCBhIHNldCBvZiBsb2NpIHRoYXQgd2UndmUgaWRlbnRpZmllZC4gSW4gbGVjdHVyZSB3ZSB1c2VkIGBzdGFuX2xtKClgIGFuZCBgc3Rhbl9nbG0oKWAgZnJvbSBgcnN0YW5hcm1gLCBiZWNhdXNlIHdlIHNpbXVsYXRlZCB0aGUgZGF0YSB3aXRob3V0IGFueSBwb3B1bGF0aW9uIHN0cnVjdHVyZS4KCkZvciB0aGlzIHdlZWsncyBsYWIsIHdlJ3JlIGdvaW5nIHRvIHJldXNlIHRoZSBneXBzeSBtb3RoIChfTHltYW50cmlhIGRpc3Bhcl8pIGRhdGEgdGhhdCB3ZSB1c2VkIFtsYXN0IHdlZWtdKGh0dHA6Ly9kYXJ3aW4uZWViLnVjb25uLmVkdS9lZWIzNDgtcmVzb3VyY2VzL0xhYjEzLm5iLmh0bWwpLiBTaW5jZSBpdCBoYXMgcG9wdWxhdGlvbiBzdHJ1Y3R1cmUsIHdlIG5lZWQgdG8gdGFrZSBhY2NvdW50IG9mIGl0LiBSYXRoZXIgdGhhbiB3cml0aW5nIGFub3RoZXIgc2NyaXB0IGluIGBTdGFuYCwgd2UncmUgZ29pbmcgdG8gdXNlIGBicm1zYCB3aGljaCBtYWtlcyB0aGUgd2hvbGUgcHJvY2VzcyBhIGxvdCBlYXNpZXIuXltJdCBpcyBwcm9iYWJseSBhbHNvIGEgKioqbG90KioqIG1vcmUgcmVsaWFibGUgdGhhbiBhbnl0aGluZyBJJ2Qgd3JpdGUsIHNpbmNlIHRoZSBhdXRob3IgaXMgYSByZWFsIGV4cGVydCBvbiBCYXllc2lhbiBpbmZlcmVuY2UsIGFuZCBJIGFtIG1lcmVseSBhIHJlYXNvbmFibHkgY29tcGV0ZW50IHVzZXIuXQoKIyMgRml0dGluZyB0aGUgbW9kZWwKCkhlcmUncyBhbiBleGFtcGxlIG9mIGFuIGFuYWx5c2lzIG9mIG1hc3MgdXNpbmcgdGhlIGZpdmUgbG9jaSB0aGF0IGhhZCB0aGUgbGFyZ2VzdCBtYWduaXR1ZGUgb2YgZWZmZWN0cyBpbiBvdXIgYW5hbHlzaXMgbGFzdCB3ZWVrLiBZb3UnbGwgbm90aWNlIHRoYXQgSSdtIHVzaW5nIGEgZGlmZmVyZW50IHJlbGF0ZWRuZXNzIGZpbGUgdGhhbiBsYXN0IHdlZWsuIEl0J3MgY2FsY3VsYXRlZCBpbiB0aGUgc2FtZSB3YXkgKGFzIHlvdSdsbCBzZWUgW2JlbG93XSgjaG93LWktcHJlcGFyZWQtdGhlLWRhdGEtZm9yLWFuYWx5c2lzKSksIGJ1dCBgYnJtc2AgZXhwZWN0cyB0byBzZWUgaXQgaW4gYSBkaWZmZXJlbnQgZm9ybWF0IGZyb20gdGhlIG9uZSB1c2VkIGluIHRoZSBgU3RhbmAgcHJvZ3JhbSBJIHdyb3RlLgoKYGBge3J9CmxpYnJhcnkodGlkeXZlcnNlKQpsaWJyYXJ5KGJybXMpCgpybShsaXN0ID0gbHMoKSkKCm9wdGlvbnMobWMuY29yZXMgPSBwYXJhbGxlbDo6ZGV0ZWN0Q29yZXMoKSkKCmRhdCA8LSByZWFkX2NzdigiZ3lwc3ltb3RoLmNzdiIpCkEgPC0gcmVhZF90c3YoImd5cHN5bW90aF9yZWxhdGVkbmVzc19mb3JfYnJtcy50eHQiLAogICAgICAgICAgICAgIHNob3dfY29sX3R5cGVzID0gRkFMU0UpCgpmaXRfTWFzcyA8LSBicm0oTWFzcyB+IFgzNDg2MiArIFgyOTUwNyArIFg0NDUyMiArIFg0MDg1NiArIFg4OTM2MiArIAogICAgICAgICAgICAgICAgICAoMXxncihzYW1wbGUsIGNvdiA9IEEpKSwKICAgICAgICAgICAgICAgIGRhdGEgPSBkYXQsCiAgICAgICAgICAgICAgICBmYW1pbHkgPSBnYXVzc2lhbigpLAogICAgICAgICAgICAgICAgc2V0X3ByaW9yKGhvcnNlc2hvZShkZiA9IDMsIHBhcl9yYXRpbyA9IDAuNSkpLAogICAgICAgICAgICAgICAgZGF0YTIgPSBsaXN0KEEgPSBBKSwKICAgICAgICAgICAgICAgIGl0ZXIgPSA1MDAwLAogICAgICAgICAgICAgICAgcmVmcmVzaCA9IDApICAKc3VtbWFyeShmaXRfTWFzcykKYGBgCgpUaGUgYCgxfGdyKHNhbXBsZSwgY292ID0gQSkpYCBpcyB3aGVyZSB0aGUgcmVsYXRlZG5lc3MgY29tZXMgaW4uIGBzYW1wbGVgIGlkZW50aWZpZXMgdGhlIHBhcnRpY3VsYXIgaW5kaXZpZHVhbCwgYW5kIGBBYCBpcyB0aGUgZ2VuZXRpYyByZWxhdGVkbmVzcyBtYXRyaXguIGBzZXRfcHJpb3IoaG9yc2VzaG9lKGRmID0gMywgcGFyX3JhdGlvID0gMC41KSlgIHNldHMgdGhlIGhvcnNlc2hvZSBwcmlvciBhcyBkZXNjcmliZWQgaW4gdGhlIFtsZWN0dXJlIG5vdGVzXShodHRwOi8vZGFyd2luLmVlYi51Y29ubi5lZHUvZWViMzQ4LW5vdGVzL2dlbm9taWMtcHJlZGljdGlvbi5odG1sKSwgd2hlcmUgYHBhcl9yYXRpb2AgaXMgb3VyIHByaW9yIGd1ZXNzIGF0IHdoYXQgZnJhY3Rpb24gb2YgdGhlIGxvY2kgbWlnaHQgYmUgaW1wb3J0YW50LiBZb3UnbGwgbm90aWNlIHRoYXQgaWYgd2UgY29tcGFyZSB0aGUgZXN0aW1hdGVzIGZyb20gdGhpcyBhbmFseXNpcyB3aXRoIHRob3NlIGZyb20gdGhlIGxvY3VzLWJ5LWxvY3VzIGFuYWx5c2lzIGxhc3Qgd2VlaywgdGhleSBzZWVtIHRvIGJlIHNtYWxsZXIgaW4gbWFnbml0dWRlIGJ1dCB0byBoYXZlIHRoZSBzYW1lIHNpZ24uCgojIyBBc3Nlc3NpbmcgbW9kZWwgZml0CgpOb3cgdGhhdCB3ZSd2ZSBlc3RpbWF0ZWQgYWxsZWxpYyBlZmZlY3RzIGF0IHRoZXNlIGZpdmUgbG9jaSB3ZSBjYW4gYXNrIGhvdyB3ZWxsIHRoZXkgYWNjb3VudCBmb3IgdmFyaWF0aW9uIGluIG1hc3MuIFdlJ2xsIGRvIHNvIGJ5IGVzdGltYXRpbmcgJFJeMiQsIHRoZSBwcm9wb3J0aW9uIG9mIHZhcmlhdGlvbiBpbiBtYXNzIHRoYXQgaXMgYWNjb3VudGVkIGZvciBieSB0aGUgbW9kZWwuIFRoZSAkUl4yJCB3ZSdsbCBjYWxjdWxhdGUgaXMgYSBiaXQgZGlmZmVyZW50IGZyb20gdGhvc2UgeW91IG1heSBoYXZlIGVuY291bnRlcmVkIHdpdGggb3RoZXIgbXVsdGlwbGUgcmVncmVzc2lvbiBtb2RlbHMsIGJlY2F1c2UgaXQncyBhIEJheWVzaWFuICRSXjIkLiBXZSBoYXZlIGFuIGVzdGltYXRlIGJvdGggb2YgdGhlIG1lYW4gJFJeMiQgYW5kIGNyZWRpYmxlIGludGVydmFscyBmb3IgdGhhdCBlc3RpbWF0ZS4KCmBgYHtyfQpiYXllc19SMihmaXRfTWFzcykKYGBgCgpTbyB3aXRoIG9ubHkgZml2ZSBsb2NpIHdlJ3ZlIGFjY291bnRlZCBmb3IgYWJvdXQgMjAlIG9mIHRoZSB2YXJpYXRpb24sIGFsdGhvdWdoIHRoZXJlIGlzIHF1aXRlIGEgYml0IG9mIHVuY2VydGFpbnR5IGFyb3VuZCB0aGF0IGVzdGltYXRlLgoKV2UgY2FuIGFsc28gdmlzdWFsaXplIHRoZSBmaXQgYnkgcGxvdHRpbmcgdGhlIG9ic2VydmVkIHZhbHVlcyBhZ2FpbnN0IHRob3NlIHByZWRpY3RlZCBmcm9tIHRoZSBtb2RlbC4KCmBgYHtyfQpwcmVkaWN0ZWRfTWFzcyA8LSBwcmVkaWN0KGZpdF9NYXNzKQoKbGlicmFyeShnZ3Bsb3QyKQoKZm9yX3Bsb3QgPC0gdGliYmxlKFByZWRpY3RlZCA9IHByZWRpY3RlZF9NYXNzWywgIkVzdGltYXRlIl0sCiAgICAgICAgICAgICAgICAgICBPYnNlcnZlZCA9IGRhdCRNYXNzKQpwIDwtIGdncGxvdChmb3JfcGxvdCwgYWVzKHggPSBQcmVkaWN0ZWQsIHkgPSBPYnNlcnZlZCkpICsKICBnZW9tX3BvaW50KCkgKwogIGdlb21fYWJsaW5lKHNsb3BlID0gMSwgaW50ZXJjZXB0ID0gMCwgY29sb3IgPSAicmVkIiwgCiAgICAgICAgICAgICAgbGluZXR5cGUgPSAiZGFzaGVkIikgKwogIHRoZW1lX2J3KCkKcApgYGAKCllvdSBjYW4gc2VlIHRoYXQgdGhlcmUncyBvbmUgb3V0bGllciBvbiBtYXNzIHRoYXQgd2UgcHJvYmFibHkgc2hvdWxkIGhhdmUgZWxpbWluYXRlZCBiZWZvcmUgYW5hbHl6aW5nIHRoZSBkYXRhLCBidXQgd2Ugd29uJ3Qgd29ycnkgYWJvdXQgdGhhdCBmb3Igb3VyIHB1cnBvc2VzLgoKIyMgSG93IG11Y2ggb2YgdGhlIHByZWRpY3Rpb24gaXMgZHVlIHRvIGJhY2tncm91bmQgZ2VuZXRpYyB2YXJpYXRpb24KCldoZW4gd2UgcHJlZGljdGVkIHRoZSB0cmFpdCBpbiB0aGUgbGFzdCBiaXQgb2YgY29kZSwgd2UgaW5jbHVkZWQgYWxsIG9mIHRoZSBpbmZvcm1hdGlvbiBhYm91dCBob3cgaW5kaXZpZHVhbHMgYXJlIHJlbGF0ZWQgdG8gb25lIGFub3RoZXIuIEl0J3MgcXVpdGUgZWFzeSB0byBsZWF2ZSB0aGF0IGluZm9ybWF0aW9uIG91dCB0byBzZWUgaG93IHdlbGwgYSBwdXJlIGdlbm9taWMgcHJlZGljdGlvbiB3b3VsZCB3b3JrIHdpdGggdGhlc2UgZGF0YS4KCmBgYHtyfQpwcmVkaWN0X3JlZHVjZWQgPC0gcHJlZGljdChmaXRfTWFzcywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgIHJlX2Zvcm11bGEgPSBOQSkKYGBgCgpXaXRoIHRoYXQgcHJlZGljdGlvbiBpbiBoYW5kLCBpdCdzIGVhc3kgdG8gY29tcGFyZSB0aGUgcHJlZGljdGlvbnMgdG8gb25lIGFub3RoZXIuICJGdWxsIiBpbiB0aGUgZm9sbG93aW5nIHBsb3QgcmVmZXJzIHRvIHRoZSBtb2RlbCB0aGF0IGluY2x1ZGVzIHJlbGF0ZWRuZXNzIGluZm9ybWF0aW9uLiAiUmVkdWNlZCIgcmVmZXJzIHRvIHRoZSBtb2RlbCB0aGF0IGRvZXNuJ3QgaW5jbHVkZSBpdC4KCmBgYHtyfQpmb3JfcGxvdCA8LSB0aWJibGUoUHJlZGljdGVkID0gYyhwcmVkaWN0ZWRfTWFzc1ssICJFc3RpbWF0ZSJdLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBwcmVkaWN0X3JlZHVjZWRbLCAiRXN0aW1hdGUiXSksCiAgICAgICAgICAgICAgICAgICBPYnNlcnZlZCA9IGMoZGF0JE1hc3MsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGF0JE1hc3MpLAogICAgICAgICAgICAgICAgICAgU291cmNlID0gYyhyZXAoIkZ1bGwiLCBucm93KGRhdCkpLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJlcCgiUmVkdWNlZCIsIG5yb3coZGF0KSkpCiAgICAgICAgICAgICAgICAgICApCnAgPC0gZ2dwbG90KGZvcl9wbG90LCBhZXMoeCA9IFByZWRpY3RlZCwgeSA9IE9ic2VydmVkKSkgKwogIGdlb21fcG9pbnQoKSArCiAgZ2VvbV9hYmxpbmUoc2xvcGUgPSAxLCBpbnRlcmNlcHQgPSAwLCBjb2xvciA9ICJyZWQiLCAKICAgICAgICAgICAgICBsaW5ldHlwZSA9ICJkYXNoZWQiKSArCiAgdGhlbWVfYncoKSArCiAgZmFjZXRfd3JhcCh+IFNvdXJjZSkKcApgYGAKCkl0IGxvb2tzIGFzIGlmIHBvaW50cyBpbiB0aGUgcmVkdWNlZCBtb2RlbCBhcmUgbm90IHF1aXRlIGFzIHRpZ2h0bHkgZ3JvdXBlZCBhcm91bmQgdGhlIDE6MSBsaW5lIGFzIHRoZXkgYXJlIGluIHRoZSBmdWxsIG1vZGVsLCBidXQgbGV0J3MgY29tcGFyZSB0aGUgJFJeMiQgb2YgdGhlIHR3byBtb2RlbHMgdG8gc2VlIGlmIHRoYXQgaW1wcmVzc2lvbiBpcyByaWdodC4KCmBgYHtyfQpjYXQoIkZ1bGwgbW9kZWwuLi5cbiIpCmJheWVzX1IyKGZpdF9NYXNzKQpjYXQoIlJlZHVjZWQgbW9kZWwuLi5cbiIpCmJheWVzX1IyKGZpdF9NYXNzLCAKICAgICAgICAgcmVfZm9ybXVsYSA9IE5BKQpgYGAKCk91ciB2aXN1YWwgaW1wcmVzc2lvbiB3YXMgcmlnaHQgdGhlIGZ1bGwgbW9kZWwgYWNjb3VudHMgZm9yIGFib3V0IDIwJSBvZiB0aGUgdmFyaWF0aW9uLCB3aGlsZSB0aGUgcmVkdWNlZCBtb2RlbCBhY2NvdW50cyBmb3Igb25seSBhYm91dCA5JS4gSW4gb3RoZXIgd29yZHMsIGFib3V0IGhhbGYgb2YgdGhlIHByZWRpY3RpdmUgcG93ZXIgZnJvbSB0aGUgbW9kZWwgaXMgY29taW5nIGZyb20gcmVsYXRlZG5lc3MgYW1vbmcgaW5kaXZpZHVhbHMgcmF0aGVyIHRoYW4gdGhlIGFsbGVsaWMgZWZmZWN0cyB0aGVtc2VsdmVzLgoKIyBQcm9ibGVtcyBhbmQgcXVlc3Rpb25zCgpEYXRhIGZpbGVzCgotIFtneXBzeW1vdGguY3N2XShodHRwOi8vZGFyd2luLmVlYi51Y29ubi5lZHUvZWViMzQ4LXJlc291cmNlcy9neXBzeW1vdGguY3N2KQotIFtneXBzeW1vdGhfcmVsYXRlZG5lc3NfZm9yX2JybXMudHh0XShodHRwOi8vZGFyd2luLmVlYi51Y29ubi5lZHUvZWViMzQ4LXJlc291cmNlcy9neXBzeW1vdGhfcmVsYXRlZG5lc3NfZm9yX2JybXMudHh0KQoKMS4gUGVyZm9ybSBhIGdlbm9taWMgcHJlZGljdGlvbiBhbmFseXNpcyBmb3IgbWFzcywgUEQsIGFuZCBURFQgdXNpbmcgdGhlIDEwIGxvY2kgaGF2aW5nIHRoZSBsYXJnZXN0IGVzdGltYXRlZCBhbGxlbGljIGVmZmVjdHMgb24gZWFjaCB0cmFpdC4KCjIuIEFzc2VzcyBob3cgd2VsbCB0aGUgbW9kZWwgYWNjb3VudHMgZm9yIHZhcmlhdGlvbiBpbiBlYWNoIHRyYWl0LgoKMy4gQXNzZXNzIGhvdyBtdWNoIGFkZGl0aW9uYWwgaW5mb3JtYXRpb24gdGhlIGFsbGVsaWMgZWZmZWN0cyBwcm92aWRlIGluIHByZWRpY3RpbmcgZWFjaCB0cmFpdCBiZXlvbmQgdGhlIGluZm9ybWF0aW9uIGFscmVhZHkgcHJvdmlkZWQgaW4gdGhlIHJlbGF0aW9uc2hpcCBtYXRyaXguCgo0LiAqKipCT05VUyBRVUVTVElPTiAoVU5HUkFERUQpKioqOiBIb3cgbWlnaHQgaW5jbHVkaW5nIHRoZSByZWxhdGlvbnNoaXAgbWF0cml4IGluIHRoaXMga2luZCBvZiBhbmFseXNpcyBsaW1pdCB5b3VyIGFiaWxpdHkgdG8gdXNlIHRoZSBhbGxlbGljIGVmZmVjdHMgZXN0aW1hdGVkIGluIHRoZXNlIHBvcHVsYXRpb25zIHRvIHByZWRpY3QgdHJhaXRzIGluIHBvcHVsYXRpb25zIHRoYXQgd2VyZW4ndCBpbmNsdWRlZCBpbiB0aGUgYW5hbHlzaXMuCgojIEhvdyBJIHByZXBhcmVkIHRoZSBkYXRhIGZvciBhbmFseXNpcwoKU2VlIGxhc3Qgd2VlaydzIGxhYiBmb3IgdGhlIGRhdGEgc291cmNlcy4gSGVyZSdzIHRoZSBjb2RlLCB3aGljaCBpcyBuZWFybHkgaWRlbnRpY2FsIHRvIHdoYXQgSSB1c2VkIGxhc3Qgd2Vlay4KCmBgYHtyfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKbGlicmFyeShwb3BraW4pCgpybShsaXN0ID0gbHMoKSkKCmdlbm9zIDwtIHJlYWRfdHN2KCJnbV9nZW5vdHlwZXNfMDEyX2ZpbmFsXzUxMjIwMTcudHh0IiwKICAgICAgICAgICAgICAgICAgY29sX25hbWVzID0gRkFMU0UsCiAgICAgICAgICAgICAgICAgIG5hID0gIi0xIiwKICAgICAgICAgICAgICAgICAgc2hvd19jb2xfdHlwZXMgPSBGQUxTRSkKcGhlbm9zIDwtIHJlYWRfY3N2KCJnbV9waGVub3R5cGVzLmNzdiIsCiAgICAgICAgICAgICAgICAgICBzaG93X2NvbF90eXBlcyA9IEZBTFNFKQoKIyMga2VlcCBvbmx5IGxvY2kgc2NvcmVkIGluIG1vcmUgdGhhbiAxMDAgaW5kaXZpZHVhbHMKIyMKbl9zY29yZWQgPC0gYXBwbHkoZ2Vub3MsIDIsIHN1bSwgbmEucm0gPSBUUlVFKQpnZW5vcyA8LSBnZW5vc1ssIG5fc2NvcmVkID4gMTAwXQoKIyMga2VlcCBvbmx5IGluZGl2aWR1YWxzIHNjb3JlZCBhdCBtb3JlIHRoYW4gNDAwMCBsb2NpCiMjCm5fc2NvcmVkIDwtIGFwcGx5KGdlbm9zLCAxLCBzdW0sIG5hLnJtID0gVFJVRSkKZ2Vub3MgPC0gZ2Vub3Nbbl9zY29yZWQgPiA0MDAwLCBdCnBoZW5vcyA8LSBwaGVub3Nbbl9zY29yZWQgPiA0MDAwLCBdCiMjIGFuZCBleGNsdWRlIGFueSByZW1haW5pbmcgbG9jaSB3aGVyZSBvbmUgb3IgbW9yZSBpbmRpdmlkdWFscyBpc24ndCBzY29yZWQKIyMKZ2Vub3MgPC0gZ2Vub3NbLCAhaXMubmEoYXBwbHkoZ2Vub3MsIDIsIHN1bSkpXQoKIyMgc3RyaXAgb2ZmIHRoZSBpbmRpdmlkdWFsIGlkcyB0byBpZGVudGlmeSBwb3B1bGF0aW9ucwojIwpwb3BzIDwtIGdzdWIoIl8uKiIsICIiLCBwaGVub3Mkc2FtcGxlKQoKIyMgYmluZCB0aGUgcG9vcHVsYXRpb24sIHBoZW5vdHlwZSwgYW5kIGdlbm90eXBlIGluZm9ybWF0aW9uIHRvZ2V0aGVyCiMjCmRhdCA8LSBjYmluZChwb3BzLCBwaGVub3MsIGdlbm9zKQoKIyMgZXN0aW1hdGUgdGhlIGtpbnNoaXAgYW1vbmcgaW5kaXZpZHVhbHMKIyMKQSA8LSBwb3BraW4odChhcy5tYXRyaXgoZ2Vub3MpKSwgCiAgICAgICAgICAgICAgc3VicG9wcyA9IHBvcHMpCmNvbG5hbWVzKEEpIDwtIGRhdCRzYW1wbGUKcm93bmFtZXMoQSkgPC0gZGF0JHNhbXBsZQp3cml0ZV90c3YoYXMuZGF0YS5mcmFtZShBKSwKICAgICAgICAgIGZpbGUgPSAiZ3lwc3ltb3RoX3JlbGF0ZWRuZXNzX2Zvcl9icm1zLnR4dCIpCmBgYA==