Acceptance Testing Framework

This package provides a testing framework for validating vulnerability analyzers against known fixtures. It enables teams to verify that their analyzer correctly identifies vulnerabilities in container images using VEX documents.

Table of Contents

• Acceptance Testing Framework
  • Table of Contents
  • Overview
  • Prerequisites
  • Quick Start
  • Creating Fixtures
    • VEX Document Format
    • Expected Results CSV
    • Uploading Fixtures
      • Create Options
  • Running Tests
    • Running Integration Tests
    • Using Local OCI Layout
  • Testing with Claircore
  • Implementing a Custom Auditor
  • Media Types

Overview

The acceptance testing framework works by:

1. Loading test fixtures from an OCI registry (images with VEX documents and expected results attached as OCI referrers)
2. Running an analyzer (wrapped by the Auditor interface) against each fixture
3. Comparing the analyzer’s output against the expected results

This approach allows reproducible, automated testing of vulnerability detection without requiring access to the analyzer’s internal implementation.

Prerequisites

• Go 1.25+ installed
• Container registry access with push permissions (for creating fixtures)

Note: The integration tests use embedded PostgreSQL binaries that are downloaded automatically - no database installation required.

Quick Start

If you just want to run the existing tests against the default fixtures repository:

go test -tags=integration -v ./test/acceptance/...

The tests will automatically pull fixtures from quay.io/projectquay/clair-fixtures.

Creating Fixtures

A fixture consists of three parts:

1. A container image (the thing being analyzed)
2. One or more VEX documents describing vulnerabilities
3. An expected results CSV defining what the analyzer should find

VEX Document Format

VEX documents must be valid CSAF 2.0 with category: csaf_vex. Here’s a minimal example:

{
  "document": {
    "category": "csaf_vex",
    "csaf_version": "2.0",
    "publisher": {
      "category": "vendor",
      "name": "Your Organization",
      "namespace": "https://example.com"
    },
    "title": "CVE-2024-1234: Example vulnerability",
    "tracking": {
      "current_release_date": "2024-01-01T00:00:00Z",
      "id": "CVE-2024-1234",
      "initial_release_date": "2024-01-01T00:00:00Z",
      "revision_history": [
        {"date": "2024-01-01T00:00:00Z", "number": "1", "summary": "Initial"}
      ],
      "status": "final",
      "version": "1"
    }
  },
  "product_tree": {
    "branches": [
      {
        "category": "product_version",
        "name": "mypackage-1.2.3",
        "product": {
          "name": "My Package 1.2.3",
          "product_id": "mypackage-1.2.3",
          "product_identification_helper": {
            "purl": "pkg:pypi/mypackage@1.2.3"
          }
        }
      }
    ]
  },
  "vulnerabilities": [
    {
      "cve": "CVE-2024-1234",
      "product_status": {
        "known_affected": ["mypackage-1.2.3"]
      }
    }
  ]
}

Key fields:

Expected Results CSV

The expected results file is a CSV with three columns (no header row):

tracking_id,product_id,status

Example:

CVE-2024-1234,mypackage-1.2.3,affected
CVE-2024-5678,otherpackage-2.0.0,not_affected
CVE-2024-9999,fixedpackage-3.0.0,absent

Lines starting with # are treated as comments.

Uploading Fixtures

Use the test/acceptance/cmd/fixture tool to create fixtures:

# Create a fixture
go run ./test/acceptance/cmd/fixture create \
    -image registry.example.com/namespace/image@sha256:abc123... \
    -tag fixture-name \
    -vex path/to/vex.json \
    -manifest path/to/expected.csv \
    -repo registry.example.com/namespace/fixtures

# You can attach multiple VEX documents
go run ./test/acceptance/cmd/fixture create \
    -image registry.example.com/namespace/image@sha256:abc123... \
    -tag multi-vex-fixture \
    -vex vex1.json \
    -vex vex2.json \
    -vex vex3.json \
    -manifest expected.csv \
    -repo registry.example.com/namespace/fixtures

# Verify the fixture was created
go run ./test/acceptance/cmd/fixture list -image registry.example.com/namespace/fixtures:fixture-name

Create Options

Running Tests

Running Integration Tests

The integration tests automatically download and run embedded PostgreSQL binaries; no database setup required:

go test -tags=integration -v ./test/acceptance/...

Using Alternative Repository

The fixtures-repo flag can be passed to the test to run against an alternative repository of fixtures.

go test -tags=integration -v ./test/acceptance -fixtures-repo registry.example.com/namespace/fixtures:fixture-name

Using Local OCI Layout

For testing without a registry, fixtures can be loaded from local OCI Layout directories:

acceptance.Run(ctx, t, auditor, []string{
    "ocidir:///path/to/local/fixture",
})

Testing with Claircore

Most teams will use the built-in claircore auditor to test their images. See integration_test.go for a complete example showing how to set up and run acceptance tests with the ClaircoreAuditor.

To run:

go test -tags=integration -v ./test/acceptance/...

The test will:

1. Pull each fixture from the registry
2. Index the image with claircore
3. Load the VEX documents attached to the image
4. Match vulnerabilities against the indexed packages
5. Compare results to the expected results CSV
6. Report any mismatches or missing vulnerabilities

Implementing a Custom Auditor

If you’re building your own vulnerability analyzer (not using claircore), implement the Auditor interface:

See the Auditor interface and Result type documentation.

See example_test.go for a complete, type-checked example implementation.

Media Types

Compressed variants (...+zstd) are also supported.