Speed up Kubernetes deployments with startup probes

Hello there 👋

Here is a small trick I use on our Kubernetes applications to make rollouts, autoscaling, and pod replacements a bit faster: I hack the startupProbe usage.

If you deploy applications on Kubernetes, this small configuration detail can make your pods serve traffic faster during deployments and scale-ups.

The official section from the Kubernetes documentation is called “Protect slow starting containers with startup probes”, and says:

Sometimes, you have to deal with applications that require additional startup time on their first initialization. In such cases, it can be tricky to set up liveness probe parameters without compromising the fast response to deadlocks that motivated such a probe. The solution is to set up a startup probe with the same command, HTTP or TCP check, with a failureThreshold * periodSeconds long enough to cover the worst case startup time.

I use the same mechanism for a slightly different reason: not only to protect slow starts, but to probe aggressively during startup without probing aggressively forever.

It is not magic. It will not make your application boot faster. If your Node.js application needs 2 seconds to load all its modules, warm some cache, connect to a database, or whatever else it does before being useful, Kubernetes cannot remove these 2 seconds for you.

But Kubernetes can avoid wasting time after the application is already ready, making your rollout sometimes up to 10s faster.

The annoying part: readiness probes waste seconds at startup#

In production, I do not want to ping my applications every second for readiness or liveness. It’s not necessary and would generate unnecessary network traffic, logs & metrics.

Most configs I see use a 10 to 15 second period for their probes.

Here is a common Pod config example:

apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
    - name: web
      image: example/web:latest
      ports:
        - name: http
          containerPort: 3000

      readinessProbe:
        httpGet:
          path: /healthz
          port: http
        periodSeconds: 10
        timeoutSeconds: 1
        failureThreshold: 3

      livenessProbe:
        httpGet:
          path: /healthz
          port: http
        periodSeconds: 10
        timeoutSeconds: 1
        failureThreshold: 3

That is reasonable. I do not need Kubernetes to ask every second if an application that has been running for 3 days is still ready.

But during startup, this can be a bit annoying 👀 (bear with me).

Imagine the container starts, Kubernetes checks readiness, and the application is not ready yet because it is still loading modules. The first probe ping fails. One or two seconds later, the application is actually ready to serve traffic. But depending on timing, Kubernetes may only notice it on the next probe (in 8s if we ping every 10s).

t=0scontainer starts

t=1sreadiness fails

t=2sapp ready

t=10sreadiness succeeds

The application is ready around t=2s, but traffic only starts after the next successful readiness check.

Without startupProbe

t=0scontainer starts

t=1sstartup fails

t=2sstartup succeeds

t=3sreadiness succeeds

Startup can be checked aggressively, then readiness/liveness go back to a quieter interval.

With startupProbe

With the startup probe hack, you end up shifting traffic to the new pod a few seconds earlier! (~8s in this example)

If you deploy often, autoscale often, or wait for many pods during a rollout, these seconds start to be very visible. Rollouts become way faster with the trick.

Pinging aggressively with the startupProbe#

Here is what a simple Pod config looks like with the startup probe hack:

apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
    - name: web
      image: example/web:latest
      ports:
        - name: http
          containerPort: 3000
      
      # the aggressive ping from the startup probe
      startupProbe:
        httpGet:
          path: /healthz
          port: http
        periodSeconds: 1
        timeoutSeconds: 1
        failureThreshold: 30

      # standard readiness probe
      readinessProbe:
        httpGet:
          path: /healthz
          port: http
        periodSeconds: 10
        timeoutSeconds: 1
        failureThreshold: 3

      # standard liveness probe
      livenessProbe:
        httpGet:
          path: /healthz
          port: http
        periodSeconds: 10
        timeoutSeconds: 1
        failureThreshold: 3

With this configuration, Kubernetes can check every second while the container is starting. Once the startup probe succeeds, Kubernetes enables the readiness and liveness probes.

That is the nice part: you can have periodSeconds: 1 during startup to ping aggressively, and once startup is done, it falls back to reasonable 10s pings.

Note: failureThreshold × periodSeconds is your startup budget. With periodSeconds: 1 and failureThreshold: 30, your application has 30 seconds to start. If it takes longer, Kubernetes kills the container and restarts it, so size these values to cover your worst-case startup time. Yes I am looking at you big Java applications.

A small nuance: how Kubernetes schedules readiness probes#

Kubernetes documentation says that while a container is not Ready, the readiness probe may be executed at times other than the configured periodSeconds interval. This is done to make the Pod ready faster.

So the timeline above is not a perfect scheduler trace. Kubernetes can be smarter than “wait exactly 10 seconds”.

But the main point still stands: startupProbe gives you a clean place to be aggressive during startup, without making readiness and liveness aggressive during the whole lifetime of the container.

Also, startup, readiness, and liveness do not mean the same thing:

• startupProbe: has the application finished starting?
• readinessProbe: should this pod receive traffic?
• livenessProbe: is this container broken enough that Kubernetes should restart it?

If you configure a startupProbe, Kubernetes delays liveness and readiness checks until the startup probe has succeeded. A successful startup probe does not directly make the pod Ready. The readiness probe still needs to succeed before traffic is sent to the pod.

Should the endpoints be the same?#

It depends.

For many HTTP services, using the same endpoint is fine:

apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
    - name: web
      image: example/web:latest
      ports:
        - name: http
          containerPort: 3000

      startupProbe:
        httpGet:
          path: /healthz
          port: http
        periodSeconds: 1
        failureThreshold: 30

      readinessProbe:
        httpGet:
          path: /healthz
          port: http
        periodSeconds: 10

This is useful when “the app has started” and “the app can serve traffic” are basically the same thing.

But sometimes they are not the same thing.

For example, your process might start an HTTP server quickly, but still need to load a model, run a migration check, build an in-memory cache, or wait for a dependency before it should receive real traffic. In that case, make the readiness endpoint stricter than the startup endpoint.

---

TL;DR: use the startup probe to aggressively ping every second your readiness endpoint.

I think it’s an easy win to speed up your deployments and autoscaling. Reach out to me on LinkedIn or Twitter if you were already doing this, if you start using it, or if you have any questions about it! :)