Kolmogorov-Smirnov test in R with ks.test()

The Kolmogorov-Smirnov test (KS test) is a generic test used to compare the empirical distribution of data against a theoretical distribution or to test if two samples follow the same distribution. R provides a function named ks.test to perform this test.

Quick answer

Use ks.test(x, "pnorm", mean = 0, sd = 1) to test if a sample follows a given distribution, or ks.test(x, y) to test if two samples come from the same distribution. If the p-value is lower than your significance level (usually 0.05), reject the null hypothesis that the distributions are equal.

set.seed(27)
x <- rnorm(50)
y <- rnorm(50, mean = 0.5)

# One sample: does 'x' follow a standard normal distribution?
ks.test(x, "pnorm", mean = 0, sd = 1)

# Two samples: do 'x' and 'y' come from the same distribution?
ks.test(x, y)

	Exact one-sample Kolmogorov-Smirnov test

data:  x
D = 0.15121, p-value = 0.1833
alternative hypothesis: two-sided


	Exact two-sample Kolmogorov-Smirnov test

data:  x and y
D = 0.32, p-value = 0.01151
alternative hypothesis: two-sided

The first test doesn’t reject that x is normal (p-value 0.1833), while the second one rejects that x and y have the same distribution (p-value 0.01151).

What the Kolmogorov-Smirnov test does

The KS test compares cumulative distribution functions (CDFs). It takes the empirical cumulative distribution function (ECDF) of your sample, \(F_n(x)\), which is the proportion of observations lower or equal than each value, and measures how far it is from a reference CDF.

The test statistic, named D, is the largest vertical distance between the two curves:

\[D = \sup_x |F_n(x) - F(x)|\]

where \(F(x)\) is the theoretical CDF in the one-sample test or the ECDF of the second sample in the two-sample test. D is always between 0 and 1:

• A D close to 0 means that the two curves are almost the same.
• A large D means that, at some point, the curves are far apart.

The p-value tells you if the observed D is larger than what you would expect by chance when the null hypothesis is true. The test can be used in two ways:

The KS test is non-parametric and doesn’t assume any specific distribution for the data, but the reference distribution of the one-sample test must be continuous and fully specified (with fixed parameters, not estimated from the same data). You will find more details about this in the normality section.

Syntax of ks.test

The syntax of the ks.test function is the following:

ks.test(x, y, ...,
        alternative = c("two.sided", "less", "greater"),
        exact = NULL, simulate.p.value = FALSE, B = 2000)

Being:

• x: a numeric vector of values.
• y: for a two-sample test, the second numeric vector of data. For a one sample test, a string containing a cumulative distribution function ("pnorm", "punif", "pexp", …).
• ...: the parameters of the distribution specified in y, such as mean and sd for "pnorm" or min and max for "punif".
• alternative: the alternative hypothesis. Possible values are "two.sided" (default), "less" (the CDF of x is lower than the CDF of y) and "greater" (the CDF of x is greater than the CDF of y).
• exact: whether to compute exact p-values. Possible values are TRUE, FALSE and NULL (the default, which calculates exact p-values in some scenarios).
• simulate.p.value: if TRUE computes p-values by Monte Carlo simulation. Defaults to FALSE.
• B: number of replicates of Monte Carlo simulation. Defaults to 2000.

One sample KS test

The one-sample Kolmogorov-Smirnov (KS) test is a non-parametric statistical method used to determine if a single sample of data follows a specified continuous distribution (e.g., normal, exponential, etc.) or if it significantly differs from that distribution.

Let F be a continuous distribution, the null and alternative hypothesis for the Kolmogorov-Smirnov test for a single sample are the following:

• \(H_0\): the distribution of a population IS F.
• \(H_1\): the distribution of a population IS NOT F.

In the example below we compare the empirical distribution function of a variable named x against the theoretical cumulative distribution function of the standardized normal distribution. The objective is to check if the distribution of the population is normal.

# Sample data
set.seed(27)
x <- rnorm(50)

# Grid of values
grid <- seq(-4, 4, length.out = 100)

# Plot of CDFs
plot(grid, pnorm(grid, mean = 0, sd = 1), type = "l",
     ylim = c(0, 1), ylab = "", lwd = 2, col = "red", xlab = "", main = "ECDFs")
lines(ecdf(x), verticals = TRUE, do.points = FALSE, col.01line = NULL)

To perform the test you will have to input the data and the name of the cumulative distribution function (such as pnorm for normal distribution, punif for uniform distribution, pexp for exponential distribution, …) and the desired parameters of the distribution.

# Sample data following a normal distribution
set.seed(27)
x <- rnorm(50)

# Does 'x' come from a standard normal distribution?
ks.test(x, "pnorm", mean = 0, sd = 1)

	Exact one-sample Kolmogorov-Smirnov test

data:  x
D = 0.15121, p-value = 0.1833
alternative hypothesis: two-sided

The p-value of the previous test is 0.1833, greater than the usual significance levels, implying there is no evidence to reject the null hypothesis that the distribution of the data is normal.

Now, we will perform the test again, but we will have a variable named y drawn from a normal distribution to be tested against a uniform distribution in the interval (-4, 4). In this scenario the cumulative distribution functions are the following:

# Sample data
set.seed(27)
y <- rnorm(50)

# Grid of values
grid <- seq(-5, 5, length.out = 100)

# Plot of CDFs
plot(grid, punif(grid, min = -4, max = 4), type = "l",
     ylim = c(0, 1), ylab = "", lwd = 2, col = "red", xlab = "", main = "ECDFs")
lines(ecdf(y), verticals = TRUE, do.points = FALSE, col.01line = NULL)

The test can be performed as shown in the example below:

# Sample data following a normal distribution
set.seed(27)
y <- rnorm(50)

# Does 'y' come from a uniform distribution in the interval (-4, 4)?
ks.test(y, "punif", min = -4, max = 4)

	Exact one-sample Kolmogorov-Smirnov test

data:  y
D = 0.23968, p-value = 0.005178
alternative hypothesis: two-sided

The test returns a p-value close to zero, which means there is statistical evidence to reject the null hypothesis that the data follows a uniform distribution.

How to interpret the output of ks.test

Let’s go through the output of the first one-sample test line by line:

	Exact one-sample Kolmogorov-Smirnov test

data:  x
D = 0.15121, p-value = 0.1833
alternative hypothesis: two-sided

• Exact one-sample Kolmogorov-Smirnov test: the type of test (one or two samples) and how the p-value was computed (exact or asymptotic, see exact vs asymptotic p-values).
• data: x: the data used in the test.
• D = 0.15121: the maximum distance between the ECDF of x and the normal CDF. At its worst point, the proportion of observations below a value differs by about 15 percentage points from the one expected under a standard normal distribution.
• p-value = 0.1833: the probability of observing a distance equal or larger than 0.15121 if the data really came from a standard normal distribution. As it is greater than 0.05, you can’t reject \(H_0\).
• alternative hypothesis: two-sided: the test looks for differences in any direction. See one-sided tests for the other options.

Keep in mind that not rejecting \(H_0\) doesn’t prove that the data is normal, it only means that there is not enough evidence against it. With small samples the test has low power and many distributions will pass it.

If you need the values for further calculations, save the result in an object and access its statistic and p.value elements:

set.seed(27)
x <- rnorm(50)

res <- ks.test(x, "pnorm", mean = 0, sd = 1)

res$statistic
res$p.value

       D
0.151207

[1] 0.1832976

You can also check that D is the largest vertical distance between the two CDFs computing it by hand. As the ECDF jumps at each observation, you need to compare the theoretical CDF with the ECDF right before and right after each jump:

n <- length(x)
F <- pnorm(sort(x))

max(max((1:n) / n - F), max(F - (0:(n - 1)) / n))

[1] 0.151207

Two sample KS test

The two-sample Kolmogorov-Smirnov (KS) test is a non-parametric method used to compare the empirical cumulative distribution functions (ECDFs) of two independent samples to determine if they are drawn from the same underlying distribution.

Let X and Y be two random independent variables the null and alternative hypotheses are the following:

• \(H_0\): the distribution of X IS EQUAL to the distribution of Y.
• \(H_1\): the distribution of X IS DIFFERENT to the distribution of Y.

The following block of code illustrates how to plot the cumulative distribution functions of two variables named x and y:

# Sample data
set.seed(27)
x <- runif(50, min = -1, max = 1)
y <- runif(50, min = -1, max = 1)

# Plot of CDFs
plot(ecdf(x), verticals = TRUE, do.points = FALSE, col.01line = NULL, xlab = "", main = "ECDFs")
lines(ecdf(y), verticals = TRUE, do.points = FALSE, col.01line = NULL, col = 4)

We can use ks.test to compare x and y to assess if they come from the same distribution.

# Sample data following a uniform distribution
set.seed(27)
x <- runif(50, min = -1, max = 1)
y <- runif(50, min = -1, max = 1)

# Two sample Kolmogorov-Smirnov test
# Do 'x' and 'y' come from the same distribution?
ks.test(x, y)

	Exact two-sample Kolmogorov-Smirnov test

data:  x and y
D = 0.2, p-value = 0.2719
alternative hypothesis: two-sided

The p-value is 0.2719, greater than the usual significance levels, so there is no evidence to reject the null hypothesis. This means that there is not enough statistical evidence to conclude that the distributions of the two samples (x and y) are significantly different from each other.

One-sided KS test

By default the test is two-sided: it rejects \(H_0\) if the CDFs differ in any direction. With the alternative argument you can test a specific direction:

• alternative = "greater": \(H_1\) is that the CDF of x lies above the CDF of y. The statistic is named D^+.
• alternative = "less": \(H_1\) is that the CDF of x lies below the CDF of y. The statistic is named D^-.

Be careful with the direction, as it is the opposite of what many people expect: a CDF that lies above means smaller values, because it reaches high cumulative probabilities earlier. In the following example y is shifted to the right (mean 0.5), so the CDF of x lies above the CDF of y:

set.seed(27)
x <- rnorm(50)
y <- rnorm(50, mean = 0.5)

# H1: the CDF of 'x' lies above that of 'y' ('x' tends to be smaller)
ks.test(x, y, alternative = "greater")

# H1: the CDF of 'x' lies below that of 'y' ('x' tends to be larger)
ks.test(x, y, alternative = "less")

	Exact two-sample Kolmogorov-Smirnov test

data:  x and y
D^+ = 0.32, p-value = 0.005756
alternative hypothesis: the CDF of x lies above that of y


	Exact two-sample Kolmogorov-Smirnov test

data:  x and y
D^- = 0, p-value = 1
alternative hypothesis: the CDF of x lies below that of y

The first test rejects \(H_0\) (p-value 0.005756), so there is evidence that the values of x tend to be smaller than the values of y. The second one doesn’t find any evidence in the opposite direction. The same argument works for the one-sample test, where y is the theoretical CDF.

Exact vs asymptotic p-values

The first line of the output tells you if the p-value is exact or asymptotic (an approximation that is accurate for large samples). With the default exact = NULL, R computes exact p-values:

• In the one-sample test, when the sample size is lower than 100 and there are no ties.
• In the two-sample test, when the product of both sample sizes is lower than 10000, with or without ties.

Otherwise the asymptotic distribution is used. You can force one method or the other with exact = TRUE or exact = FALSE:

set.seed(27)
z <- rnorm(150)

# n = 150, so the default p-value is asymptotic
ks.test(z, "pnorm")

# Force the exact p-value
ks.test(z, "pnorm", exact = TRUE)

	Asymptotic one-sample Kolmogorov-Smirnov test

data:  z
D = 0.038689, p-value = 0.9783
alternative hypothesis: two-sided


	Exact one-sample Kolmogorov-Smirnov test

data:  z
D = 0.038689, p-value = 0.9719
alternative hypothesis: two-sided

The statistic D doesn’t change, only the p-value. With moderate or large samples both p-values are very similar, so the default option is fine in most cases. The exact p-value is more useful with small samples, where the asymptotic approximation can be inaccurate.

Testing normality with ks.test: don’t estimate the parameters

A very common mistake is to test normality plugging the sample mean and standard deviation into ks.test:

ks.test(x, "pnorm", mean = mean(x), sd = sd(x))

The p-values of the KS test are only valid when the parameters of the distribution are fixed in advance. If you estimate them from the same data, the normal curve is fitted to your sample, D tends to be smaller and the p-value is too large, so the test will rarely reject normality even when the data is not normal.

In the following example the data comes from an exponential distribution, which is clearly skewed. The KS test with estimated parameters barely rejects normality, while the Lilliefors test, which corrects the p-value for the estimation of the mean and the standard deviation, and the Shapiro-Wilk test reject it with a very small p-value:

set.seed(27)
v <- rexp(40, rate = 1)

# Wrong: KS test with the mean and sd estimated from the data
ks.test(v, "pnorm", mean = mean(v), sd = sd(v))

# Lilliefors test (install.packages("nortest"))
library(nortest)
lillie.test(v)

# Shapiro-Wilk test
shapiro.test(v)

	Exact one-sample Kolmogorov-Smirnov test

data:  v
D = 0.21264, p-value = 0.04577
alternative hypothesis: two-sided


	Lilliefors (Kolmogorov-Smirnov) normality test

data:  v
D = 0.21264, p-value = 8.927e-05


	Shapiro-Wilk normality test

data:  v
W = 0.74113, p-value = 4.939e-07

Note that the Lilliefors statistic D is exactly the same, only the p-value changes. So, if you want to test if your data is normal, without knowing the mean and the variance, use the Lilliefors test or the Shapiro-Wilk test instead of ks.test. Use ks.test when the parameters come from theory or from a different data set.

Kolmogorov-Smirnov vs Shapiro-Wilk test

Both tests are used to check normality, but they are not interchangeable. The table below summarizes the main differences:

As a rule of thumb, use Shapiro-Wilk (or Lilliefors) to check if a sample is normal and use KS to compare a sample with a fully specified distribution of any kind or to compare two samples.

Common errors and warnings

Warning: ties should not be present for the Kolmogorov-Smirnov test

The one-sample KS test assumes a continuous distribution, where repeated values have zero probability. If your data contains ties (repeated values), usually because it was rounded, you will get the following warning:

set.seed(27)
w <- round(rnorm(50), 1)

ks.test(w, "pnorm")

	Asymptotic one-sample Kolmogorov-Smirnov test

data:  w
D = 0.16134, p-value = 0.148
alternative hypothesis: two-sided

Warning message:
In ks.test.default(w, "pnorm") :
  ties should not be present for the Kolmogorov-Smirnov test

The test still runs, but with ties R can’t compute the exact p-value and the asymptotic one is only approximate. Setting exact = FALSE doesn’t remove the warning and suppressWarnings only hides it. You can:

• Check how many values are repeated with sum(duplicated(w)). If there are only a few ties, the approximate p-value is usually good enough.
• Check if the conclusion is robust adding a tiny amount of noise to break the ties, for example with ks.test(jitter(w), "pnorm"). In this case the p-value is 0.1551, very similar to the previous one.
• If your data is really discrete (counts, Likert scales, …), the KS test is not appropriate. Use a test for discrete data, such as the chi-square goodness of fit test.

Note that in current versions of R the two-sample test supports ties and computes exact p-values without any warning.

Forgetting the parameters of the distribution

If you don’t pass any parameter, ks.test(x, "pnorm") compares your data with a standard normal distribution (mean 0 and standard deviation 1), not with any normal distribution. With data on a different scale the test will reject \(H_0\) even if the data is normal:

set.seed(27)
u <- rnorm(50, mean = 10, sd = 2)

ks.test(u, "pnorm")

	Exact one-sample Kolmogorov-Smirnov test

data:  u
D = 0.99997, p-value = 8.882e-16
alternative hypothesis: two-sided

A D close to 1 is a clear sign of this mistake. Specify the parameters, such as ks.test(u, "pnorm", mean = 10, sd = 2), and remember that they must not be estimated from the data, as explained in the normality section.

Using the wrong function name

The y argument must be the name of a cumulative distribution function, which starts with p ("pnorm", "pexp", "punif", …). If you pass "norm", R calls the norm function, which computes the norm of a matrix, and you get a confusing error:

ks.test(u, "norm")

Error in y(sort(x), ...) : 'A' must be a numeric matrix

Passing the density function, such as "dnorm", is worse, as there is no error but the result is meaningless. Always use the p functions.

Error: not enough ‘x’ data

This error appears when x has no valid numeric values, for example if it is empty or you selected the wrong column:

ks.test(numeric(0), "pnorm")

Error in ks.test.default(numeric(0), "pnorm") : not enough 'x' data

Check that the vector you pass to ks.test is numeric and not empty, for example with str(x) or length(x).