• Published:

Kube-OVN’s approach to CVE remediation has changed from occasional manual cleanup to a much more aggressive model: capture every layer of dependency, keep those dependencies moving, and let routine automation eliminate most security issues before anyone needs to treat them as a dedicated CVE task.

The dependencies involved are broader than application libraries alone. They include the operating system image, Go itself, Go modules, upstream components, and the surrounding ecosystem used in testing and integration, such as Kubernetes and KubeVirt. The goal is not merely to react to vulnerability reports, but to keep the entire dependency surface continuously updated so that the number of known fixable CVEs trends toward zero dynamically.

The remediation process so far

The workflow has gone through several stages:

  • Manual fixes during releases, or manual fixes when needed
  • CVE scanning on every commit, followed by manual remediation when fixable issues are found
  • Automatic dependency updates on master, partial automatic updates on release branches, and a small amount of remaining manual work
  • The target state: automatic updates for all dependencies, with manual remediation avoided as much as possible

Each stage reduced some pressure while exposing new problems. The current direction is to solve the larger dependency freshness problem rather than treating CVE fixing as a separate, repeated chore.

Fixing CVEs only when needed

The earliest model was simple: collect the security work and deal with it when a release was approaching or when a fix became necessary.

This had one clear advantage: compared with fixing every security issue separately, the number of remediation rounds was lower.

But the drawbacks were significant:

  • Release time became overloaded. Engineers already had to finish features, stabilize the release, run tests, and fix bugs; concentrated CVE cleanup added more pressure.
  • Dependency upgrades had not been validated continuously in daily development, so late updates introduced unknown risk.
  • Much of the manual security work did not provide meaningful product value, but still consumed engineering time.

This model reduced the frequency of the work, but it made releases more stressful and increased the uncertainty around dependency changes.

Scanning every commit

The next step was to move CVE detection into the regular development pipeline. Trivy scanning was added to CI, and code with security issues could not be merged.

Relevant workflow and example run:

  • https://github.com/kubeovn/kube-ovn/blob/master/.github/workflows/build-x86-image.yaml#L3437
  • https://github.com/kubeovn/kube-ovn/actions/runs/15760235247?pr=5376

This changed the nature of the work. Instead of discovering a pile of CVEs right before a release, fixes were spread across normal development.

The benefits were obvious:

  • CVE work no longer accumulated at release time, so release pressure was lower.
  • Fast releases became easier.
  • Most dependency updates had already gone through daily automated validation, reducing release risk.

However, this still had several problems:

  • There is always a time gap between the last commit scan and the release scan, so some CVEs can theoretically be missed.
  • Normal feature development and bug fixing can be interrupted by unrelated CVEs.
  • A large amount of remediation still requires manual work.

Commit-time scanning improved release hygiene, but it also made unrelated security noise part of everyday development.

Updating everything automatically

The newer strategy is more aggressive: whenever a dependency has a new version, try to update it, regardless of whether the change is specifically marked as a security fix.

This reframes the problem. Instead of asking “how do we fix this CVE?”, the system asks “how do we keep dependencies current?” If that larger problem is solved, most CVE fixes arrive naturally as a side effect.

Kube-OVN’s dependency surface includes:

  • OS image and system packages: ubuntu:24.04, apt-get install ....
  • Go version and Go module dependencies
  • Upstream dependencies: OVS, OVN
  • Other integration and ecosystem components: kubernetes, kubevirt, multus, metallb, cilium, talos

The update policy is intentionally aggressive. Even major version upgrades are attempted.

Daily rebuilds for the OS layer

For the operating system image, the pipeline rebuilds the base image every day. This automatically picks up fixes already available in the OS image and apt repositories.

Workflow:

  • https://github.com/kubeovn/kube-ovn/blob/master/.github/workflows/build-kube-ovn-base.yaml

This part runs daily without human intervention. It is meant to eliminate many OS-level and package-level CVEs without requiring an engineer to notice each one individually.

Renovate for Go and module updates

Go-related dependencies are handled with Renovate:

  • Go version updates
  • All dependency versions in go.mod
  • Real-time updates plus auto merge
  • Manual handling only when merge problems occur

Examples and configuration:

  • https://github.com/kubeovn/kube-ovn/pull/5354
  • https://github.com/kubeovn/kube-ovn/blob/master/renovate.json

No special configuration is needed for the basic Go dependency flow; Renovate’s automatic detection is enough.

Building upstream components more aggressively

For upstream components such as OVS and OVN, the policy is even more proactive. They are built daily, directly from upstream branches. Updates are consumed even when upstream has not published a formal release.

The assumption is that upstream changes are mostly bug fixes. Pulling them in early should improve stability rather than reduce it, provided that automated validation is strong enough.

Capturing non-standard dependencies

For other components, Renovate is used with custom regular expressions to detect and update versions embedded in places like Dockerfiles, Makefiles, and GitHub Actions workflows.

Tracking issue and example file:

  • https://github.com/kubeovn/kube-ovn/issues/5291
  • https://github.com/kubeovn/kube-ovn/blob/master/Makefile

This matters because many dependencies are not declared in standard package manager files. Without custom capture rules, those versions would remain invisible to automated update tooling.

What this improves

The automatic-update model brings several practical advantages:

  • Most CVEs are fixed without anyone explicitly noticing them. A small number can be fixed automatically within a day, and only special cases require manual handling.
  • Upstream bug fixes, performance improvements, and new features are pulled in continuously, improving overall software stability.
  • Compatibility validation for related components, such as Kubernetes version updates and KubeVirt version updates, happens automatically and earlier.
  • Manual intervention is greatly reduced.

This does not mean every dependency update is safe or effortless. It means the risk is moved earlier into continuous validation instead of being concentrated around release time.

The cost of an aggressive update policy

The approach also has trade-offs:

  • Frequent dependency updates create noise, so aggregation strategies are needed.
  • The update volume is too large for manual testing, which makes automated testing essential.
  • The project must actively adapt to upstream version changes.
  • New upstream versions can be unstable. In the past two years, this has happened twice.

The model depends heavily on CI quality. If automated testing is weak, automatic merging becomes dangerous. If automated testing is reliable, frequent updates become a way to detect breakage early and reduce late-stage surprises.

Why Renovate instead of Dependabot

Renovate has several advantages for this use case:

  • It can capture custom dependencies, including non-standard versions in Dockerfiles and Makefiles.
  • It can run on branches other than master.
  • It supports auto merge.

Those capabilities are important when the goal is to update more than standard language-package dependencies.

Remaining gaps

Some problems are still unresolved:

  • False positives in automated tests can prevent automatic merging.
  • Go-related updates on released branches are not yet automated.
  • The effect of Renovate’s security-only strategy is still unclear.
  • Related configuration: https://github.com/kubeovn/kube-ovn/blob/master/renovate.json#L45
  • Release branch strategy may need to be redefined.
  • Some dependencies are still not covered by automatic updates.

The direction is clear: expand automatic dependency capture, reduce manual CVE handling, and rely on continuous validation to surface risks early rather than during release crunch time.

Related Posts