Claircore

Claircore is the engine behind the Clair v4 container security solution. The Claircore package exports our domain models, interfaces necessary to plug into our business logic, and a default set of implementations. This default set of implementations define our support matrix and consists of the following distributions and languages:

Ubuntu
Debian
RHEL
Red Hat Container First content
SUSE
Oracle
Alpine
AWS Linux
VMWare Photon
Python
Java
Go
Ruby

Claircore relies on PostgreSQL for its persistence and the library will handle migrations if configured to do so.

The diagram below is a high level overview of Claircore's architecture.

graph LR
subgraph Indexer
im[Image Manifest]
libindex[Libindex]
iir[IndexReport]
im --> libindex --> iir
end
iir -.-> db[(Database)]

graph LR
subgraph Matcher
mir[IndexReport]
libvuln[Libvuln]
vr[VulnerabilityReport]
mir --> libvuln --> vr
end
db[(Database)] -.-> mir

When a claircore.Manifest is submitted to Libindex, the library will index its constituent parts and create a report with its findings.

When a claircore.IndexReport is provided to Libvuln, the library will discover vulnerabilities affecting it and generate a claircore.VulnerabilityReport.

Getting Started

The following documentation helps a beginner learn to use Claircore.

Libindex Usage

Libindex is the Go package responsible for fetching container image layers, identifying packages, distributions, and repositories within these layers, and computing a final coalesced Index Report.

An Index Report is primarily used as input to Libvuln's vulnerability matching process.

Usage

Libindex is runtime constructed via the libindex.New method. New requires an libindex.Opts struct.

Options

package libindex // import "github.com/quay/claircore/libindex"

type Options struct {
	// Store is the interface used to persist and retrieve results of indexing.
	Store indexer.Store
	// Locker provides system-wide locks. If the indexing work is distributed the
	// lock should be backed by a distributed store.
	Locker LockSource
	// FetchArena is an interface tied to the lifecycle of LibIndex to enable management
	// of the filesystem while separate processes are dealing with layers, for example:
	// you can reference count downloaded layer files to avoid racing.
	FetchArena indexer.FetchArena
	// ScanLockRetry specifies how often we should try to acquire a lock for scanning a
	// given manifest if lock is taken.
	ScanLockRetry time.Duration
	// LayerScanConcurrency specifies the number of layers to be scanned in parallel.
	LayerScanConcurrency int
	// LayerFetchOpt is unused and kept here for backwards compatibility.
	LayerFetchOpt interface{}
	// NoLayerValidation controls whether layers are checked to actually be
	// content-addressed. With this option toggled off, callers can trigger
	// layers to be indexed repeatedly by changing the identifier in the
	// manifest.
	NoLayerValidation bool
	// ControllerFactory provides an alternative method for creating a scanner during libindex runtime
	// if nil the default factory will be used. useful for testing purposes
	ControllerFactory ControllerFactory
	// Ecosystems a list of ecosystems to use which define which package databases and coalescing methods we use
	Ecosystems []*indexer.Ecosystem
	// ScannerConfig holds functions that can be passed into configurable
	// scanners. They're broken out by kind, and only used if a scanner
	// implements the appropriate interface.
	//
	// Providing a function for a scanner that's not expecting it is not a fatal
	// error.
	ScannerConfig struct {
		Package, Dist, Repo, File map[string]func(interface{}) error
	}
	Resolvers []indexer.Resolver
}
    Options are dependencies and options for constructing an instance of
    libindex

The above outlines the relevant bits of the Options structure.

Store is required needs to be an object that satisfies the indexer.Store interface.

Locker is required and needs to satisfy the LockSource interface.

FetchArena is required and needs to satify the FetchArena interface.

Providing a nil "Ecosystems" slice will supply the default set, instructing Libindex to index for all supported content in a layer, and is typically desired.

Construction

Constructing Libindex is straight forward.

	opts := new(libindex.Options)
	// Populate with desired settings...
	lib, err := libindex.New(ctx, opts, http.DefaultClient)
	if err != nil {
		panic(err)
	}
	defer lib.Close(ctx) // Remember to cleanup when done.

The constructing code should provide a valid Context tied to some lifetime.

Indexing

Indexing is the process of submitting a manifest to Libindex, fetching the manifest's layers, indexing their contents, and coalescing a final Index Report.

Coalescing is the act of computing a final set of contents (packages, distributions, repos) from a set of layers. Since layers maybe shared between many manifests, the final contents of a manifest must be computed.

To perform an Index you must provide a claircore.Manifest data struture to the Index method. The Manifest data structure describes an image manifest's layers and where they can be fetched from.

	m := new(claircore.Manifest)
	// Populate somehow ...
	ir, err := lib.Index(ctx, m)
	if err != nil {
		panic(err)
	}

The Index method will block until an claircore.IndexReport is returned. The context should be bound to some valid lifetime such as a request.

As the Indexer works on the manifest it will update its database throughout the process. You may view the status of an index report via the "IndexReport" method.

	ir, ok, err := lib.IndexReport(ctx, m.Hash)
	if err != nil {
		panic(err)
	}

Libindex performs its work incrementally and saves state as it goes along. If Libindex encounters an intermittent error during the index (for example, due to network failure while fetching a layer), when the manifest is resubmitted only the layers not yet indexed will be fetched and processed.

State

Libindex treats layers as content addressable. Once a layer identified by a particular hash is indexed its contents are definitively known. A request to re-index a known layer results in returning the previous successful response.

This comes in handy when dealing with base layers. The Ubuntu base layer is seen very often across container registries. Treating this layer as content addressable precludes the need to fetch and index the layer every time Libindex encounters it in a manifest.

There are times where re-indexing the same layer is necessary however. At the point where Libindex realizes a new version of a component has not indexed a layer being submitted it will perform the indexing operation.

A client must notice that Libindex has updated one of its components and subsequently resubmit Manifests. The State endpoint is implemented for this reason.

Clients may query the State endpoint to receive an opaque string acting as a cookie, identifying a unique state of Libindex. When a client sees this cookie change it should re-submit manifests to Libindex to obtain a new index report.

	state, err := lib.State(ctx)
	if err != nil {
		panic(err)
	}
	if state == prevState {
		// Nothing to do.
		return
	}
	// Otherwise, re-index manifest.

AffectedManifests

Libindex is capable of providing a client with all manifests affected by a set of vulnerabilities. This functionality is designed for use with a notification mechanism.

	var vulns []claircore.Vulnerability
	affected, err := lib.AffectedManifests(ctx, vulns)
	if err != nil {
		panic(err)
	}
	for manifest, vulns := range affected.VulnerableManifests {
		for _, vuln := range vulns {
			fmt.Printf("vuln affecting manifest %s: %+v", manifest, vuln)
		}
	}

The slice of vulnerabilities returned for each manifest hash will be sorted by claircore.NormalizedSeverity in "most severe" descending order.

Libvuln Usage

Libvuln is the Go package responsible for keeping the database of vulnerabilities consistent, matching container image contents with vulnerabilities, and reporting diffs between updates of the same security database.

Usage

Libvuln is runtime constructed via the libvuln.New method. New requires a libvuln.Opts struct.

Options

package libvuln // import "github.com/quay/claircore/libvuln"

type Options struct {
	// Store is the interface used to persist and retrieve vulnerabilities
	// for of matching.
	Store datastore.MatcherStore
	// Locker provides system-wide locks for the updater subsystem. If the
	// matching work is distributed the lock should be backed by a distributed
	// store.
	Locker LockSource
	// An interval on which Libvuln will check for new security database
	// updates.
	//
	// This duration will have jitter added to it, to help with smearing load on
	// installations.
	UpdateInterval time.Duration
	// A slice of strings representing which updaters libvuln will create.
	//
	// If nil all default UpdaterSets will be used.
	//
	// The following sets are supported:
	// "alpine"
	// "aws"
	// "debian"
	// "oracle"
	// "photon"
	// "pyupio"
	// "rhel"
	// "suse"
	// "ubuntu"
	UpdaterSets []string
	// A list of out-of-tree updaters to run.
	//
	// This list will be merged with any defined UpdaterSets.
	//
	// If you desire no updaters to run do not add an updater
	// into this slice.
	Updaters []driver.Updater
	// A slice of strings representing which
	// matchers will be used.
	//
	// If nil all default Matchers will be used
	//
	// The following names are supported by default:
	// "alpine"
	// "aws"
	// "debian"
	// "oracle"
	// "photon"
	// "python"
	// "rhel"
	// "suse"
	// "ubuntu"
	MatcherNames []string

	// Config holds configuration blocks for MatcherFactories and Matchers,
	// keyed by name.
	MatcherConfigs map[string]driver.MatcherConfigUnmarshaler

	// A list of out-of-tree matchers you'd like libvuln to
	// use.
	//
	// This list will me merged with the default matchers.
	Matchers []driver.Matcher

	// Enrichers is a slice of enrichers to use with all VulnerabilityReport
	// requests.
	Enrichers []driver.Enricher

	// UpdateWorkers controls the number of update workers running concurrently.
	// If less than or equal to zero, a sensible default will be used.
	UpdateWorkers int

	// UpdateRetention controls the number of updates to retain between
	// garbage collection periods.
	//
	// The lowest possible value is 2 in order to compare updates for notification
	// purposes.
	UpdateRetention int

	// If set to true, there will not be a goroutine launched to periodically
	// run updaters.
	DisableBackgroundUpdates bool

	// UpdaterConfigs is a map of functions for configuration of Updaters.
	UpdaterConfigs map[string]driver.ConfigUnmarshaler

	// Client is an http.Client for use by all updaters.
	//
	// Must be set.
	Client *http.Client
}

The above outlines the relevant bits of the Opts structure.

Construction

Constructing Libvuln is straight forward.

	opts := new(libvuln.Options)
	// Populate with desired settings...
	lib, err := libvuln.New(ctx, opts)
	if err != nil {
		panic(err)
	}
	defer lib.Close(ctx)

The constructing code should provide a valid Context tied to some lifetime.

On construction, New will block until the security databases are initialized. Expect some delay before this method returns.

Scanning

Scanning is the process of taking a claircore.IndexReport comprised of a Manifest's content and determining which vulnerabilities affect the Manifest. A claircore.VulnerabilityReport will be returned with these details.

	m := new(claircore.Manifest)
	// Populate somehow ...
	ir, err := indexer.Index(ctx, m)
	if err != nil {
		panic(err)
	}
	vr, err := lib.Scan(ctx, ir)
	if err != nil {
		panic(err)
	}

In the above example, Libindex is used to generate a claircore.IndexReport. The index report is then provided to Libvuln and a subsequent vulnerability report identifying any vulnerabilities affecting the manifest is returned.

Updates API

By default, Libvuln manages a set of long running updaters responsible for periodically fetching and loading new advisory contents into its database. The Updates API allows a client to view and manipulate aspects of the update operations that updaters perform.

In this getting started guide, we will only cover the two methods most interesting to new users.

UpdateOperations

This API provides a list of recent update operations performed by implemented updaters. The UpdateOperation slice returned will be sorted by latest timestamp descending.

	ops, err := lib.UpdateOperations(ctx, `updater`)
	if err != nil {
		panic(err)
	}
	for updater, ops := range ops {
		fmt.Printf("ops for updater %s, %+v", updater, ops)
	}

UpdateDiff

Mostly used by the Clair v4 notification subsystem, this endpoint will provide the caller with any removed or added vulnerabilities between two update operations. Typically a diff takes places against two versions of the same data source. This is useful to inform downstream applications what new vulnerabilities have entered the system.

	ops, err := lib.UpdateOperations(ctx, `updater`)
	if err != nil {
		panic(err)
	}
	for upd, ops := range ops {
		fmt.Printf("updater: %v", upd)
		diff, err := lib.UpdateDiff(ctx, ops[1].Ref, ops[0].Ref)
		if err != nil {
			panic(err)
		}
		for _, vuln := range diff.Added {
			fmt.Printf("vuln %+v added in %v", vuln, diff.Cur.Ref)
		}
	}

Concepts

The following sections give a conceptual overview of how Claircore works internally.

Vulnerability Matching

The following describes a successful scan.

Updaters have ran either in the background on an interval or an offline loader has been ran.
A Manifest is provided to Libindex. Libindex fetches all the layers, runs all scanner types on each layer, persists all artifacts found in each layer, and computes an IndexReport.
A IndexReport is provided to Libvuln.
Libvuln creates a stream of IndexRecord structs from the IndexReport and concurrently streams these structs to each configured Matcher.
Libvuln computes a VulnerabilityReport aggregating all vulnerabilities discovered by all configured Matcher implementations.
Sometime later the security advisory database is updated and a new request to Libvuln will present updated vulnerability data.

Indexer

claircore/indexer

The Indexer package performs Libindex's heavy lifting. It is responsible for retreiving Manifest layers, parsing the contents of each layer, and computing an IndexReport.

To perform this action in incremental steps the Indexer is implemented as a finite state machine. At each state transition the Indexer persists an updated IndexReport to its datastore.

States

The following diagram expresses the possible states of the Indexer:

stateDiagram-v2
	state if_indexed <<choice>>
	[*] --> CheckManifest
	CheckManifest --> if_indexed
	if_indexed --> [*]: Indexed
	if_indexed --> FetchLayers: Unindexed
	FetchLayers --> ScanLayers
	ScanLayers --> Coalesce
	Coalesce --> IndexManifest
	IndexManifest --> IndexFinished
	IndexFinished --> [*]
%% These notes make the diagram unreadable :/
%% note left of CheckManifest: Determine if this manifest has been indexed previously.
%% note right of FetchLayers: Determine which layers need to be indexed and fetch them.
%% note right of ScanLayers: Concurrently run needed Indexers on layers.
%% note right of Coalesce: Compute the final contents of the container image.
%% note right of IndexManifest: Associate all the discovered data.
%% note right of IndexFinished: Persist the results.

Data Model

The Indexer data model focuses on content addressable hashes as primary keys, the deduplication of package/distribution/repostitory information, and the recording of scan artifacts. Scan artifacts are unique artifacts found within a layer which point to a deduplicated general package/distribution/repository record.

The following diagram outlines the current Indexer data model.

%%{init: {"er":{"layoutDirection":"RL"}} }%%
erDiagram
	ManifestLayer many to 1 Manifest: ""
	ManifestLayer many to 1 Layer: ""
	ScannedLayer many to 1 Layer: ""
	ScannedLayer many to 1 Scanner: ""
	ScannedManifest many to 1 Manifest: ""
	ScannedManifest many to 1 Scanner: ""

	TYPE_ScanArtifact 1 to 1 Layer: ""
	TYPE_ScanArtifact 1 to 1 Scanner: ""
	TYPE_ScanArtifact 1 to 1 TYPE: ""

	ManifestIndex many to 1 Manifest: ""
	ManifestIndex 1 to zero or one TYPE: ""

	IndexReport 1 to 1 Manifest: "cached result"

Note that TYPE stands in for each of the Indexer types (i.e. Package, Repository, etc.).

HTTP Resources

Indexers as currently built may make network requests. This is an outstanding issue. The following are the URLs used.

https://search.maven.org/solrsearch/select
https://catalog.redhat.com/api/containers/
https://security.access.redhat.com/data/metrics/repository-to-cpe.json
https://security.access.redhat.com/data/metrics/container-name-repos-map.json

Matcher Architecture

claircore/internal/matcher claircore/libvuln/driver

The Matcher architecture is based on a data flow application.

graph TD
	start[Libvuln.Scan]
	finish[Merge into VulnerabilityReport]
	start ---> RemoteMatcher & Matcher ---> finish
	subgraph RemoteMatcher
		ra[Filter interested packages]
		api[Make API calls]
		rv[Decide vulnerability]
		ra --> api --> rv
	end
	subgraph Matcher
		dbv[Check versions in-database]
		ma[Filter interested packages]
		adv[Retrieve vulnerability information]
		mv[Deicide vulnerability]
		ma --> adv --> mv
		adv -.-> dbv -.-> mv
	end

When Libvuln's Scan method is called with an IndexReport it will begin the process of matching container contents with vulnerabilities.

Each configured Matcher will be instantiated concurrently. Depending on the interfaces the Matcher implements, one of the possible data flows will occur in the diagram above.

The provided IndexReport will be unpacked into a stream of IndexRecord structs. Each Matcher will evaluate each record in the stream and determine if the IndexRecord is vulnerable to a security advisory in their responsible databases.

Once each Matcher returns the set of vulnerabities, Libvuln will merge the results into a VulnerabilityReport and return this to the client.

HTTP Resources

NOTE: Remote matchers are being considered for removal.

"Remote matchers" may make HTTP requests during the matcher flow. These requests are time-bound and errors are not reported. The following are the URLs used.

Severity Mapping

Claircore will normalize a security databases's severity string to a set of defined values. Clients may use the NormalizedSeverity field on a claircore.Vulnerability to react to vulnerability severities without needing to know each security database's severity strings. All strings used in the mapping tables are identical to the strings found within the relevant security database.

Claircore Severity Strings

The following are severity strings Claircore will normalize others to. Clients can guarantee one of these strings will be associated with a claircore.Vulnerability.

Unknown
Negligible
Low
Medium
High
Critical

Alpine Mapping

The Alpine SecDB database does not provide severity information. All vulnerability severities will be Unknown.

Alpine Severity	Claircore Severity
*	Unknown

AWS Mapping

The AWS UpdateInfo database provides severity information.

AWS Severity	Claircore Severity
low	Low
medium	Medium
important	High
critical	Critical

Debian Mapping

The Debian security tracker data provides severity information.

Debian Severity	Claircore Severity
unimportant	Low
low	Medium
medium	High
high	Critical
*	Unknown

Oracle Mapping

The Oracle OVAL database provides severity information.

Oracle Severity	Claircore Severity
N/A	Unknown
LOW	Low
MODERATE	Medium
IMPORTANT	High
CRITICAL	Critical

RHEL Mapping

The RHEL OVAL database provides severity information.

RHEL Severity	Claircore Severity
None	Unknown
Low	Low
Moderate	Medium
Important	High
Critical	Critical

SUSE Mapping

The SUSE OVAL database provides severity information.

SUSE Severity	Claircore Severity
None	Unknown
Low	Low
Moderate	Medium
Important	High
Critical	Critical

Ubuntu Mapping

The Ubuntu OVAL database provides severity information.

Ubuntu Severity	Claircore Severity
Untriaged	Unknown
Negligible	Negligible
Low	Low
Medium	Medium
High	High
Critical	Critical

Photon Mapping

The Photon OVAL database provides severity information.

Photon Severity	Claircore Severity
Low	Low
Moderate	Medium
Important	High
Critical	Critical

OSV Mapping

OSV provides severity information via CVSS vectors, when applicable. These are normalized according to the NVD qualitative rating scale. If both v3 and v2 vectors are present, v3 is preferred.

CVSSv3

Base Score	Claircore Severity
0.0	Negligible
0.1-3.9	Low
4.0-6.9	Medium
7.0-8.9	High
9.0-10.0	Critical

CVSSv2

Base Score	Claircore Severity
0.0-3.9	Low
4.0-6.9	Medium
7.0-10.0	High

Updaters and Defaults

The default updaters are tracked in updater/defaults/defaults.go.

HTTP Resources

The following are the HTTP hosts and paths that Clair will attempt to talk to in a default configuration. This list is non-exhaustive, as some servers will issue redirects and some request URLs are constructed dynamically.

https://secdb.alpinelinux.org/
http://repo.us-west-2.amazonaws.com/2018.03/updates/x86_64/mirror.list
https://cdn.amazonlinux.com/2/core/latest/x86_64/mirror.list
https://cdn.amazonlinux.com/al2023/core/mirrors/latest/x86_64/mirror.list
https://deb.debian.org/
https://security-tracker.debian.org/tracker/data/json
https://nvd.nist.gov/feeds/json/cve/1.1/
https://linux.oracle.com/security/oval/com.oracle.elsa-*.xml.bz2
https://packages.vmware.com/photon/photon_oval_definitions/
https://security.access.redhat.com/data/oval/v2/PULP_MANIFEST
https://security.access.redhat.com/data/csaf/v2/vex/
https://ftp.suse.com/pub/projects/security/oval/
https://api.launchpad.net/1.0/
https://security-metadata.canonical.com/oval/com.ubuntu.*.cve.oval.xml
https://osv-vulnerabilities.storage.googleapis.com/

How Tos

The following sections provide instructions on accomplish specific goals in Claircore.

Adding Distribution Or Language Support

Adding Distribution And Language Support

Note: If terms in this document sound foreign check out Getting Started to acquaint yourself with "indexing", "scanning", and "matching"

The claircore team is always open to adding more distributions and languages to the library.

Generally, distributions or languages must provide a security tracker.

All officially supported distributions and languages provide a database of security vulnerabilities.

These databases are maintained by the distribution or language developers and reflect up-to-date CVE and advisory data for their packages.

If your distribution or language does not provide a security tracker or piggy-backs off another distribution's start an issue in our Github issue tracker to discuss further.

Implementing an Updater

The first step to adding your distribution or language to claircore is getting your security tracker's data ingested by Libvuln.

The Updater interfaces are responsible for this task.

An implementer must consider several design points:

Does the security database provide enough information to parse each entry into a claircore.Vulnerability?
- Each entry must parse into a claircore.Vulnerability.
- Each Vulnerability must contain a package and a repository or distribution field.
Will the Updater need to be configured at runtime?
- Your updater may implement the Configurable interface. Your matcher will have its "Configuration" method called before use, giving you an opportunity for run time configuration.
What fields in a parsed claircore.Vulnerability will be present when indexing layer artifacts.
- When implementing an updater you must keep in mind how packages/distributions/repositories will be parsed during index. When indexing a layer a common data model must exist between the possible package/distribution/repository and the parsed Vulnerabilitie's package/distribution/repository fields.

If you are having trouble figuring out these requirements do not hesitate to reach out to us for help.

After you have taken the design points into consideration, you are ready to implement your updater.

Typically you will create a new package named after the source you are adding support for.

Inside this package you can begin implementing the Updater and Updater Set Factory interfaces.

Optionally you may implement the Configurable interface if you need runtime configuration.

It will undoubtly be helpful to look at the examples in the "ubuntu", "rhel", and "debian" packages to get yourself started.

Implementing a Package Scanner

At this point you hopefully have your Updater working, writing vulnerability data into Libvuln's database.

We can now move our attention to package scanning.

A package scanner is responsible for taking a claircore.Layer and parsing the contents for a particular package database or set of files inside the provided tar archive. Once the target files are located the package scanner should parse these files into claircore.Packages and return a slice of these data structures.

Package scanning is context free, meaning no distribution classification has happened yet. This is because manifests are made up of layers, and a layer which holds a package database may not hold distribution information such as an os-release file. A package scanner need only parse a target package database and return claircore.Packages.

You need to implement the Package Scanner interface to achieve this.

Optionally, you may implement the Configurable Scanner if the scanner needs to perform runtime configuration before use.

Keep in mind that its very common for distributions to utilize an existing package manager such as RPM.

If this is the case there's a high likelihood that you can utilize the existing "rpm" or "dpkg" package scanner implementations.

Implementing a Distribution Scanner

Once the package scanner is implemented, tested, and working you can begin implementing a Distribution Scanner.

Implementing a distribution scanner is a design choice. Distributions and repositories are the way claircore matches packages to vulnerabilities.

If your implemented Updater parses vulnerabilities with distribution information you will likely need to implement a distribution scanner. Likewise, if your Updater parses vulnerabilities with repository information (typical with language vulnerabilities) you will likely need to implement a repository scanner.

A distribution scanner, like a package scanner, is provided a claircore.Layer.

The distribution scanner will parse the provided tar archive exhaustively searching for any clue that this layer was derived from your distribution. If you identify that it is, you should return a common distribution used by your Updater implementation. This ensures that claircore can match the output of your distribution scanner with your parsed vulnerabilities.

Optionally, you may implement the Configurable Scanner if the scanner needs to perform runtime configuration before use.

Implementing a Repository Scanner

As mentioned above, implementing a repository scanner is a design choice, often times applicable for language package managers.

If your Updater parses vulnerabilities with a repository field you will likely want to implement a repository scanner.

A repository scanner is used just like a distribution scanner however you will search for any clues that a layer contains your repository and if so return a common data model identifying the repository.

Optionally, you may implement the Configurable Scanner if the scanner needs to perform runtime configuration before use.

Implementing a Coalescer

As you may have noticed, the process of scanning a layer for packages, distribution, and repository information is distinct and separate.

At some point, claircore will need to take all the context-free information returned from layer scanners and create a complete view of the manifest. A coalescer performs this computation.

It's unlikely you will need to implement your own coalescer. Claircore provides a default "linux" coalescer which will work if your package database is rewritten when modified. For example, if a Dockerfile's RUN command causes a change to to dpkg's /var/lib/dpkg/status database, the resulting manifest will have a copy placed in the associated layer.

However, if your package database does not fit into this model, implementing a coalescer may be necessary.

To implement a coalescer, several details must be understood:

Each layer only provides a "piece" of the final manifest.
- Because manifests are comprised of multiple copy-on-write layers, some layers may contain package information, distribution information, repository information, any combination of those, or no information at all.
An OS may have a "dist-upgrade" performed and the implications of this on the package management system is distribution or language dependent.
- The coalescer must deal with distribution upgrades in a sane way. If your distribution or language does a dist-upgrade, are all packages bumped? Are they simply left alone? The coalescer must understand what happens and compute the final manifest's content correctly.
Packages may be removed and added between layers.
- When the package database is a regular file on disk, this case is simpler: the database file found in the most recent layer holds the ultimate set of packages for all previous layers. However, in the case where the package database is realized by adding and removing files on disk it becomes trickier. Claircore has no special handling of whiteout files, currently. We will address this in upcoming releases.

If your distribution or language cannot utilize a default coalescer, you will need to implement the Coalescer interface

Implementing or Adding To An Ecosystem

An Ecosystem provides a set of coalescers, package scanners, distribution scanners, and repository scanners to Libindex at the time of indexing.

Libindex will take the Ecosystem and scan each layer with all provided scanners. When Libindex is ready to coalesce the results of each scanner into an IndexReport the provided coalescer is given the output of the configured scanners.

This allows Libindex to segment the input to the coalescing step to particular scanners that a coalescer understands.

For instance, if we only wanted a (fictitious) Haskell coalescer to evaluate artifacts returned from a (fictitious) Haskell package and repository scanner we would create an ecosystem similar to:

// HaskellScanner returns a configured PackageScanner.
func haskellScanner() indexer.PackageScanner { return nil }

// HaskellCoalescer returns a configured Coalescer.
func haskellCoalescer() indexer.Coalescer { return nil }

// NewEcosystem provides the set of scanners and coalescers for the haskell
// ecosystem.
func NewEcosystem(ctx context.Context) *indexer.Ecosystem {
	return &indexer.Ecosystem{
		PackageScanners: func(ctx context.Context) ([]indexer.PackageScanner, error) {
			return []indexer.PackageScanner{haskellScanner()}, nil
		},
		DistributionScanners: func(ctx context.Context) ([]indexer.DistributionScanner, error) {
			return []indexer.DistributionScanner{}, nil
		},
		RepositoryScanners: func(ctx context.Context) ([]indexer.RepositoryScanner, error) {
			return []indexer.RepositoryScanner{}, nil
		},
		Coalescer: func(ctx context.Context) (indexer.Coalescer, error) {
			return haskellCoalescer(), nil
		},
	}
}

This ensures that Libindex will only provide Haskell artifacts to the Haskell coalescer and avoid calling the coalescer with rpm packages for example.

If your distribution uses an already implemented package manager such as "rpm" or "dpkg", it's likely you will simply add your scanners to the existing ecosystem in one of those packages.

Alternative Implementations

This how-to guide is a "perfect world" scenario.

Working on claircore has made us realize that this domain is a bit messy. Security trackers are not developed with package managers in mind, security databases do not follow correct specs, distribution maintainers spin their own tools, etc.

We understand that supporting your distribution or language may take some bending of claircore's architecture and business logic. If this is the case, start a conversation with us. We are open to design discussions.

Getting Help

At this point, you have implemented all the necessary components to integrate your distribution or language with claircore.

If you struggle with the design phase or are getting stuck at the implementation phases do not hesitate to reach out to us. Here are some links:

Reference

The follow sections provide a reference for developing with Claircore. Important interfaces and structs are outlined.

Coalescer

A coalescer must compute the final contents of a manifest given the artifacts found at each layer.

package indexer // import "github.com/quay/claircore/indexer"

type Coalescer interface {
	Coalesce(ctx context.Context, artifacts []*LayerArtifacts) (*claircore.IndexReport, error)
}
    Coalescer takes a set of layers and creates coalesced IndexReport.

    A coalesced IndexReport should provide only the packages present in the
    final container image once all layers were applied.

package indexer // import "github.com/quay/claircore/indexer"

type Coalescer interface {
	Coalesce(ctx context.Context, artifacts []*LayerArtifacts) (*claircore.IndexReport, error)
}
    Coalescer takes a set of layers and creates coalesced IndexReport.

    A coalesced IndexReport should provide only the packages present in the
    final container image once all layers were applied.

package indexer // import "github.com/quay/claircore/indexer"

type LayerArtifacts struct { Hash claircore.Digest Pkgs []*claircore.Package Dist []*claircore.Distribution // each layer can only have a single distribution Repos []*claircore.Repository Files []claircore.File } LayerArifact aggregates the artifacts found within a layer.


A `Coalsecer` implementation is free to determine this computation given the
artifacts found in a layer. A `Coalescer` is called with a slice of
`LayerArtifacts` structs. The manifest's layer ordering is preserved in the
provided slice.

ConfigurableScanner

A ConfigurableSanner is an optional interface a Scanner interface may implement. When implemented, the scanner's Configure method will be called with a ConfigDeserializer function. The Scanner may pass its config as an argument to the ConfigDeserializer function to populate the struct.

package indexer // import "github.com/quay/claircore/indexer"

type ConfigDeserializer func(interface{}) error
    ConfigDeserializer can be thought of as an Unmarshal function with the byte
    slice provided.

    This will typically be something like (*json.Decoder).Decode.

package indexer // import "github.com/quay/claircore/indexer"

type ConfigDeserializer func(interface{}) error
    ConfigDeserializer can be thought of as an Unmarshal function with the byte
    slice provided.

    This will typically be something like (*json.Decoder).Decode.

package indexer // import "github.com/quay/claircore/indexer"

type ConfigurableScanner interface { Configure(context.Context, ConfigDeserializer) error } ConfigurableScanner is an interface scanners can implement to receive configuration.

Distribution Scanner

A Distribution Scanner should identify any operating system distribution associated with the provided layer. It is OK for no distribution information to be discovered.

package indexer // import "github.com/quay/claircore/indexer"

type DistributionScanner interface {
	VersionedScanner
	Scan(context.Context, *claircore.Layer) ([]*claircore.Distribution, error)
}
    DistributionScanner reports the Distributions found in a given layer.

Ecosystem

An Ecosystem groups together scanners and coalescers which are often used together. Ecosystems are usually defined in a go package that corresponds to a package manager, such as dpkg. See dpkg/ecosystem.go for an example.

The Indexer will retrieve artifacts from the provided scanners and provide these scan artifacts to the coalescer in the Ecosystem.

package indexer // import "github.com/quay/claircore/indexer"

type Ecosystem struct {
	PackageScanners      func(ctx context.Context) ([]PackageScanner, error)
	DistributionScanners func(ctx context.Context) ([]DistributionScanner, error)
	RepositoryScanners   func(ctx context.Context) ([]RepositoryScanner, error)
	FileScanners         func(ctx context.Context) ([]FileScanner, error)
	Coalescer            func(ctx context.Context) (Coalescer, error)
	Name                 string
}
    Ecosystems group together scanners and a Coalescer which are commonly used
    together.

    A typical ecosystem is "dpkg" which will use the "dpkg" package indexer,
    the "os-release" distribution scanner and the "apt" repository scanner.

    A Controller will scan layers with all scanners present in its configured
    ecosystems.

Index Report

An Index Report defines the contents of a manifest. An Index Report can be unpacked into a slice of Index Records, each of which defines a package, distribution, repository tuple.

package claircore // import "github.com/quay/claircore"

type IndexReport struct {
	// the manifest hash this IndexReport is describing
	Hash Digest `json:"manifest_hash"`
	// the current state of the index operation
	State string `json:"state"`
	// all discovered packages in this manifest key'd by package id
	Packages map[string]*Package `json:"packages"`
	// all discovered distributions in this manifest key'd by distribution id
	Distributions map[string]*Distribution `json:"distributions"`
	// all discovered repositories in this manifest key'd by repository id
	Repositories map[string]*Repository `json:"repository"`
	// a list of environment details a package was discovered in key'd by package id
	Environments map[string][]*Environment `json:"environments"`
	// whether the index operation finished successfully
	Success bool `json:"success"`
	// an error string in the case the index did not succeed
	Err string `json:"err"`
	// Files doesn't end up in the json report but needs to be available at post-coalesce
	Files map[string]File `json:"-"`
}
    IndexReport provides a database for discovered artifacts in an image.

    IndexReports make heavy usage of lookup maps to associate information
    without repetition.

func (report *IndexReport) IndexRecords() []*IndexRecord

package claircore // import "github.com/quay/claircore"

type IndexReport struct {
	// the manifest hash this IndexReport is describing
	Hash Digest `json:"manifest_hash"`
	// the current state of the index operation
	State string `json:"state"`
	// all discovered packages in this manifest key'd by package id
	Packages map[string]*Package `json:"packages"`
	// all discovered distributions in this manifest key'd by distribution id
	Distributions map[string]*Distribution `json:"distributions"`
	// all discovered repositories in this manifest key'd by repository id
	Repositories map[string]*Repository `json:"repository"`
	// a list of environment details a package was discovered in key'd by package id
	Environments map[string][]*Environment `json:"environments"`
	// whether the index operation finished successfully
	Success bool `json:"success"`
	// an error string in the case the index did not succeed
	Err string `json:"err"`
	// Files doesn't end up in the json report but needs to be available at post-coalesce
	Files map[string]File `json:"-"`
}
    IndexReport provides a database for discovered artifacts in an image.

    IndexReports make heavy usage of lookup maps to associate information
    without repetition.

func (report *IndexReport) IndexRecords() []*IndexRecord

package claircore // import "github.com/quay/claircore"

type IndexRecord struct { Package *Package Distribution *Distribution Repository *Repository } IndexRecord is an entry in the IndexReport.

IndexRecords provide full access to contextual package structures such as
Distribution and Repository.

A list of these can be thought of as an "unpacked" IndexReport

Indexer Store

The indexer.Store interface defines all necessary persistence methods for Libindex to provide its functionality.

package indexer // import "github.com/quay/claircore/indexer"

type Store interface {
	Setter
	Querier
	Indexer
	// Close frees any resources associated with the Store.
	Close(context.Context) error
}
    Store is an interface for dealing with objects libindex needs to persist.
    Stores may be implemented per storage backend.

package indexer // import "github.com/quay/claircore/indexer"

type Store interface {
	Setter
	Querier
	Indexer
	// Close frees any resources associated with the Store.
	Close(context.Context) error
}
    Store is an interface for dealing with objects libindex needs to persist.
    Stores may be implemented per storage backend.

package indexer // import "github.com/quay/claircore/indexer"

type Setter interface { // PersistManifest must store the presence of a manifest and it's layers into the system. // // Typically this will write into identity tables so later methods have a foreign key // to reference and data integrity is applied. PersistManifest(ctx context.Context, manifest claircore.Manifest) error // DeleteManifests removes the manifests indicated by the passed digests // from the backing store. DeleteManifests(context.Context, ...claircore.Digest) ([]claircore.Digest, error)

// SetLayerScanned marks the provided layer hash successfully scanned by the provided versioned scanner.
//
// After this method is returned a call to Querier.LayerScanned with the same arguments must return true.
SetLayerScanned(ctx context.Context, hash claircore.Digest, scnr VersionedScanner) error
// RegisterPackageScanners registers the provided scanners with the persistence layer.
RegisterScanners(ctx context.Context, scnrs VersionedScanners) error
// SetIndexReport persists the current state of the IndexReport.
//
// IndexReports maybe in intermediate states to provide feedback for clients. this method should be
// used to communicate scanning state updates. to signal the scan has completely successfully
// see SetIndexFinished.
SetIndexReport(context.Context, *claircore.IndexReport) error
// SetIndexFinished marks a scan successfully completed.
//
// After this method returns a call to Querier.ManifestScanned with the manifest hash represted in the provided IndexReport
// must return true.
//
// Also a call to Querier.IndexReport with the manifest hash represted in the provided IndexReport must return the IndexReport
// in finished state.
SetIndexFinished(ctx context.Context, sr *claircore.IndexReport, scnrs VersionedScanners) error

} Setter interface provides the method set for required marking events, or registering components, associated with an Index operation.

package indexer // import "github.com/quay/claircore/indexer"

type Store interface { Setter Querier Indexer // Close frees any resources associated with the Store. Close(context.Context) error } Store is an interface for dealing with objects libindex needs to persist. Stores may be implemented per storage backend.

package indexer // import "github.com/quay/claircore/indexer"

type Setter interface {
	// PersistManifest must store the presence of a manifest and it's layers into the system.
	//
	// Typically this will write into identity tables so later methods have a foreign key
	// to reference and data integrity is applied.
	PersistManifest(ctx context.Context, manifest claircore.Manifest) error
	// DeleteManifests removes the manifests indicated by the passed digests
	// from the backing store.
	DeleteManifests(context.Context, ...claircore.Digest) ([]claircore.Digest, error)

	// SetLayerScanned marks the provided layer hash successfully scanned by the provided versioned scanner.
	//
	// After this method is returned a call to Querier.LayerScanned with the same arguments must return true.
	SetLayerScanned(ctx context.Context, hash claircore.Digest, scnr VersionedScanner) error
	// RegisterPackageScanners registers the provided scanners with the persistence layer.
	RegisterScanners(ctx context.Context, scnrs VersionedScanners) error
	// SetIndexReport persists the current state of the IndexReport.
	//
	// IndexReports maybe in intermediate states to provide feedback for clients. this method should be
	// used to communicate scanning state updates. to signal the scan has completely successfully
	// see SetIndexFinished.
	SetIndexReport(context.Context, *claircore.IndexReport) error
	// SetIndexFinished marks a scan successfully completed.
	//
	// After this method returns a call to Querier.ManifestScanned with the manifest hash represted in the provided IndexReport
	// must return true.
	//
	// Also a call to Querier.IndexReport with the manifest hash represted in the provided IndexReport must return the IndexReport
	// in finished state.
	SetIndexFinished(ctx context.Context, sr *claircore.IndexReport, scnrs VersionedScanners) error
}
    Setter interface provides the method set for required marking events,
    or registering components, associated with an Index operation.

package indexer // import "github.com/quay/claircore/indexer"

type Querier interface { // ManifestScanned returns whether the given manifest was scanned by the provided scanners. ManifestScanned(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) (bool, error) // LayerScanned returns whether the given layer was scanned by the provided scanner. LayerScanned(ctx context.Context, hash claircore.Digest, scnr VersionedScanner) (bool, error) // PackagesByLayer gets all the packages found in a layer limited by the provided scanners. PackagesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Package, error) // DistributionsByLayer gets all the distributions found in a layer limited by the provided scanners. DistributionsByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Distribution, error) // RepositoriesByLayer gets all the repositories found in a layer limited by the provided scanners. RepositoriesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Repository, error) // FilesByLayer gets all the interesting files found in a layer limited by the provided scanners. FilesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]claircore.File, error) // IndexReport attempts to retrieve a persisted IndexReport. IndexReport(ctx context.Context, hash claircore.Digest) (*claircore.IndexReport, bool, error) // AffectedManifests returns a list of manifest digests which the target vulnerability // affects. AffectedManifests(ctx context.Context, v claircore.Vulnerability, f claircore.CheckVulnernableFunc) ([]claircore.Digest, error) } Querier interface provides the method set to ascertain indexed artifacts and query whether a layer or manifest has been scanned.

package indexer // import "github.com/quay/claircore/indexer"

package indexer // import "github.com/quay/claircore/indexer"

type Setter interface {
	// PersistManifest must store the presence of a manifest and it's layers into the system.
	//
	// Typically this will write into identity tables so later methods have a foreign key
	// to reference and data integrity is applied.
	PersistManifest(ctx context.Context, manifest claircore.Manifest) error
	// DeleteManifests removes the manifests indicated by the passed digests
	// from the backing store.
	DeleteManifests(context.Context, ...claircore.Digest) ([]claircore.Digest, error)

	// SetLayerScanned marks the provided layer hash successfully scanned by the provided versioned scanner.
	//
	// After this method is returned a call to Querier.LayerScanned with the same arguments must return true.
	SetLayerScanned(ctx context.Context, hash claircore.Digest, scnr VersionedScanner) error
	// RegisterPackageScanners registers the provided scanners with the persistence layer.
	RegisterScanners(ctx context.Context, scnrs VersionedScanners) error
	// SetIndexReport persists the current state of the IndexReport.
	//
	// IndexReports maybe in intermediate states to provide feedback for clients. this method should be
	// used to communicate scanning state updates. to signal the scan has completely successfully
	// see SetIndexFinished.
	SetIndexReport(context.Context, *claircore.IndexReport) error
	// SetIndexFinished marks a scan successfully completed.
	//
	// After this method returns a call to Querier.ManifestScanned with the manifest hash represted in the provided IndexReport
	// must return true.
	//
	// Also a call to Querier.IndexReport with the manifest hash represted in the provided IndexReport must return the IndexReport
	// in finished state.
	SetIndexFinished(ctx context.Context, sr *claircore.IndexReport, scnrs VersionedScanners) error
}
    Setter interface provides the method set for required marking events,
    or registering components, associated with an Index operation.

package indexer // import "github.com/quay/claircore/indexer"

package indexer // import "github.com/quay/claircore/indexer"

type Indexer interface {
	// IndexPackages indexes a package into the persistence layer.
	IndexPackages(ctx context.Context, pkgs []*claircore.Package, layer *claircore.Layer, scnr VersionedScanner) error
	// IndexDistributions indexes distributions into the persistence layer.
	IndexDistributions(ctx context.Context, dists []*claircore.Distribution, layer *claircore.Layer, scnr VersionedScanner) error
	// IndexRepositories indexes repositories into the persistence layer.
	IndexRepositories(ctx context.Context, repos []*claircore.Repository, layer *claircore.Layer, scnr VersionedScanner) error
	// IndexFiles indexes the interesting files into the persistence layer.
	IndexFiles(ctx context.Context, files []claircore.File, layer *claircore.Layer, scnr VersionedScanner) error
	// IndexManifest should index the coalesced manifest's content given an IndexReport.
	IndexManifest(ctx context.Context, ir *claircore.IndexReport) error
}
    Indexer interface provide the method set required for indexing layer and
    manifest contents into a persistent store.
```

Matcher Store

The datastore.MatcherStore interface defines all necessary persistence methods for Libvuln to provide its functionality.

package datastore // import "github.com/quay/claircore/datastore"

type MatcherStore interface {
	Updater
	Vulnerability
	Enrichment
}
    MatcherStore aggregates all interface types

package datastore // import "github.com/quay/claircore/datastore"

type MatcherStore interface {
	Updater
	Vulnerability
	Enrichment
}
    MatcherStore aggregates all interface types

package datastore // import "github.com/quay/claircore/datastore"

type Updater interface { EnrichmentUpdater

// UpdateVulnerabilities creates a new UpdateOperation, inserts the provided
// vulnerabilities, and ensures vulnerabilities from previous updates are
// not queried by clients.
UpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
// UpdateVulnerabilitiesIter performs the same operation as
// UpdateVulnerabilities, but accepting an iterator function.
UpdateVulnerabilitiesIter(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulnIter VulnerabilityIter) (uuid.UUID, error)
// DeltaUpdateVulnerabilities creates a new UpdateOperation consisting of existing
// vulnerabilities and new vulnerabilities. It also takes an array of deleted
// vulnerability names which should no longer be available to query.
DeltaUpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability, deletedVulns []string) (uuid.UUID, error)
// GetUpdateOperations returns a list of UpdateOperations in date descending
// order for the given updaters.
//
// The returned map is keyed by Updater implementation's unique names.
//
// If no updaters are specified, all UpdateOperations are returned.
GetUpdateOperations(context.Context, driver.UpdateKind, ...string) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRefs reports the latest update reference for every known
// updater.
GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRef reports the latest update reference of any known
// updater.
GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
// DeleteUpdateOperations removes an UpdateOperation.
// A call to GC must be run after this to garbage collect vulnerabilities associated
// with the UpdateOperation.
//
// The number of UpdateOperations deleted is returned.
DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
// Operations.
//
// In diff(1) terms, this is like
//
//	diff prev cur
//
GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID) (*driver.UpdateDiff, error)
// GC will delete any update operations for an updater which exceeds the provided keep
// value.
//
// Implementations may throttle the GC process for datastore efficiency reasons.
//
// The returned int64 value indicates the remaining number of update operations needing GC.
// Running this method till the returned value is 0 accomplishes a full GC of the vulnstore.
GC(ctx context.Context, keep int) (int64, error)
// Initialized reports whether the vulnstore contains vulnerabilities.
Initialized(context.Context) (bool, error)
// RecordUpdaterStatus records that an updater is up to date with vulnerabilities at this time
RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime time.Time, fingerprint driver.Fingerprint, updaterError error) error
// RecordUpdaterSetStatus records that all updaters from an updater set are up to date with vulnerabilities at this time
RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime time.Time) error

} Updater is an interface exporting the necessary methods for updating a vulnerability database.

package datastore // import "github.com/quay/claircore/datastore"

type MatcherStore interface { Updater Vulnerability Enrichment } MatcherStore aggregates all interface types

package datastore // import "github.com/quay/claircore/datastore"

type Updater interface {
	EnrichmentUpdater

	// UpdateVulnerabilities creates a new UpdateOperation, inserts the provided
	// vulnerabilities, and ensures vulnerabilities from previous updates are
	// not queried by clients.
	UpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
	// UpdateVulnerabilitiesIter performs the same operation as
	// UpdateVulnerabilities, but accepting an iterator function.
	UpdateVulnerabilitiesIter(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulnIter VulnerabilityIter) (uuid.UUID, error)
	// DeltaUpdateVulnerabilities creates a new UpdateOperation consisting of existing
	// vulnerabilities and new vulnerabilities. It also takes an array of deleted
	// vulnerability names which should no longer be available to query.
	DeltaUpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability, deletedVulns []string) (uuid.UUID, error)
	// GetUpdateOperations returns a list of UpdateOperations in date descending
	// order for the given updaters.
	//
	// The returned map is keyed by Updater implementation's unique names.
	//
	// If no updaters are specified, all UpdateOperations are returned.
	GetUpdateOperations(context.Context, driver.UpdateKind, ...string) (map[string][]driver.UpdateOperation, error)
	// GetLatestUpdateRefs reports the latest update reference for every known
	// updater.
	GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string][]driver.UpdateOperation, error)
	// GetLatestUpdateRef reports the latest update reference of any known
	// updater.
	GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
	// DeleteUpdateOperations removes an UpdateOperation.
	// A call to GC must be run after this to garbage collect vulnerabilities associated
	// with the UpdateOperation.
	//
	// The number of UpdateOperations deleted is returned.
	DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
	// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
	// Operations.
	//
	// In diff(1) terms, this is like
	//
	//	diff prev cur
	//
	GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID) (*driver.UpdateDiff, error)
	// GC will delete any update operations for an updater which exceeds the provided keep
	// value.
	//
	// Implementations may throttle the GC process for datastore efficiency reasons.
	//
	// The returned int64 value indicates the remaining number of update operations needing GC.
	// Running this method till the returned value is 0 accomplishes a full GC of the vulnstore.
	GC(ctx context.Context, keep int) (int64, error)
	// Initialized reports whether the vulnstore contains vulnerabilities.
	Initialized(context.Context) (bool, error)
	// RecordUpdaterStatus records that an updater is up to date with vulnerabilities at this time
	RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime time.Time, fingerprint driver.Fingerprint, updaterError error) error
	// RecordUpdaterSetStatus records that all updaters from an updater set are up to date with vulnerabilities at this time
	RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime time.Time) error
}
    Updater is an interface exporting the necessary methods for updating a
    vulnerability database.

package datastore // import "github.com/quay/claircore/datastore"

type EnrichmentUpdater interface { // UpdateEnrichments creates a new EnrichmentUpdateOperation, inserts the provided // EnrichmentRecord(s), and ensures enrichments from previous updates are not // queries by clients. UpdateEnrichments(ctx context.Context, kind string, fingerprint driver.Fingerprint, enrichments []driver.EnrichmentRecord) (uuid.UUID, error) // UpdateEnrichmentsIter performs the same operation as UpdateEnrichments, but // accepting an iterator function. UpdateEnrichmentsIter(ctx context.Context, kind string, fingerprint driver.Fingerprint, enIter EnrichmentIter) (uuid.UUID, error) } EnrichmentUpdater is an interface exporting the necessary methods for storing and querying Enrichments.

package datastore // import "github.com/quay/claircore/datastore"

type MatcherStore interface { Updater Vulnerability Enrichment } MatcherStore aggregates all interface types

package datastore // import "github.com/quay/claircore/datastore"

type Updater interface {
	EnrichmentUpdater

	// UpdateVulnerabilities creates a new UpdateOperation, inserts the provided
	// vulnerabilities, and ensures vulnerabilities from previous updates are
	// not queried by clients.
	UpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
	// UpdateVulnerabilitiesIter performs the same operation as
	// UpdateVulnerabilities, but accepting an iterator function.
	UpdateVulnerabilitiesIter(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulnIter VulnerabilityIter) (uuid.UUID, error)
	// DeltaUpdateVulnerabilities creates a new UpdateOperation consisting of existing
	// vulnerabilities and new vulnerabilities. It also takes an array of deleted
	// vulnerability names which should no longer be available to query.
	DeltaUpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability, deletedVulns []string) (uuid.UUID, error)
	// GetUpdateOperations returns a list of UpdateOperations in date descending
	// order for the given updaters.
	//
	// The returned map is keyed by Updater implementation's unique names.
	//
	// If no updaters are specified, all UpdateOperations are returned.
	GetUpdateOperations(context.Context, driver.UpdateKind, ...string) (map[string][]driver.UpdateOperation, error)
	// GetLatestUpdateRefs reports the latest update reference for every known
	// updater.
	GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string][]driver.UpdateOperation, error)
	// GetLatestUpdateRef reports the latest update reference of any known
	// updater.
	GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
	// DeleteUpdateOperations removes an UpdateOperation.
	// A call to GC must be run after this to garbage collect vulnerabilities associated
	// with the UpdateOperation.
	//
	// The number of UpdateOperations deleted is returned.
	DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
	// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
	// Operations.
	//
	// In diff(1) terms, this is like
	//
	//	diff prev cur
	//
	GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID) (*driver.UpdateDiff, error)
	// GC will delete any update operations for an updater which exceeds the provided keep
	// value.
	//
	// Implementations may throttle the GC process for datastore efficiency reasons.
	//
	// The returned int64 value indicates the remaining number of update operations needing GC.
	// Running this method till the returned value is 0 accomplishes a full GC of the vulnstore.
	GC(ctx context.Context, keep int) (int64, error)
	// Initialized reports whether the vulnstore contains vulnerabilities.
	Initialized(context.Context) (bool, error)
	// RecordUpdaterStatus records that an updater is up to date with vulnerabilities at this time
	RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime time.Time, fingerprint driver.Fingerprint, updaterError error) error
	// RecordUpdaterSetStatus records that all updaters from an updater set are up to date with vulnerabilities at this time
	RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime time.Time) error
}
    Updater is an interface exporting the necessary methods for updating a
    vulnerability database.

package datastore // import "github.com/quay/claircore/datastore"

package datastore // import "github.com/quay/claircore/datastore"

type Vulnerability interface {
	// Get finds the vulnerabilities which match each package provided in the
	// [IndexRecord]s. This may be a one-to-many relationship. A map of Package
	// ID to Vulnerabilities is returned.
	Get(ctx context.Context, records []*claircore.IndexRecord, opts GetOpts) (map[string][]*claircore.Vulnerability, error)
}
    Vulnerability is the interface for querying stored Vulnerabilities.
```
```
package datastore // import "github.com/quay/claircore/datastore"

type MatcherStore interface {
	Updater
	Vulnerability
	Enrichment
}
    MatcherStore aggregates all interface types

package datastore // import "github.com/quay/claircore/datastore"

type Updater interface { EnrichmentUpdater

// UpdateVulnerabilities creates a new UpdateOperation, inserts the provided
// vulnerabilities, and ensures vulnerabilities from previous updates are
// not queried by clients.
UpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
// UpdateVulnerabilitiesIter performs the same operation as
// UpdateVulnerabilities, but accepting an iterator function.
UpdateVulnerabilitiesIter(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulnIter VulnerabilityIter) (uuid.UUID, error)
// DeltaUpdateVulnerabilities creates a new UpdateOperation consisting of existing
// vulnerabilities and new vulnerabilities. It also takes an array of deleted
// vulnerability names which should no longer be available to query.
DeltaUpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability, deletedVulns []string) (uuid.UUID, error)
// GetUpdateOperations returns a list of UpdateOperations in date descending
// order for the given updaters.
//
// The returned map is keyed by Updater implementation's unique names.
//
// If no updaters are specified, all UpdateOperations are returned.
GetUpdateOperations(context.Context, driver.UpdateKind, ...string) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRefs reports the latest update reference for every known
// updater.
GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRef reports the latest update reference of any known
// updater.
GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
// DeleteUpdateOperations removes an UpdateOperation.
// A call to GC must be run after this to garbage collect vulnerabilities associated
// with the UpdateOperation.
//
// The number of UpdateOperations deleted is returned.
DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
// Operations.
//
// In diff(1) terms, this is like
//
//	diff prev cur
//
GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID) (*driver.UpdateDiff, error)
// GC will delete any update operations for an updater which exceeds the provided keep
// value.
//
// Implementations may throttle the GC process for datastore efficiency reasons.
//
// The returned int64 value indicates the remaining number of update operations needing GC.
// Running this method till the returned value is 0 accomplishes a full GC of the vulnstore.
GC(ctx context.Context, keep int) (int64, error)
// Initialized reports whether the vulnstore contains vulnerabilities.
Initialized(context.Context) (bool, error)
// RecordUpdaterStatus records that an updater is up to date with vulnerabilities at this time
RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime time.Time, fingerprint driver.Fingerprint, updaterError error) error
// RecordUpdaterSetStatus records that all updaters from an updater set are up to date with vulnerabilities at this time
RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime time.Time) error

} Updater is an interface exporting the necessary methods for updating a vulnerability database.

package datastore // import "github.com/quay/claircore/datastore"

type EnrichmentUpdater interface {
	// UpdateEnrichments creates a new EnrichmentUpdateOperation, inserts the provided
	// EnrichmentRecord(s), and ensures enrichments from previous updates are not
	// queries by clients.
	UpdateEnrichments(ctx context.Context, kind string, fingerprint driver.Fingerprint, enrichments []driver.EnrichmentRecord) (uuid.UUID, error)
	// UpdateEnrichmentsIter performs the same operation as UpdateEnrichments, but
	// accepting an iterator function.
	UpdateEnrichmentsIter(ctx context.Context, kind string, fingerprint driver.Fingerprint, enIter EnrichmentIter) (uuid.UUID, error)
}
    EnrichmentUpdater is an interface exporting the necessary methods for
    storing and querying Enrichments.

package datastore // import "github.com/quay/claircore/datastore"

type Vulnerability interface { // Get finds the vulnerabilities which match each package provided in the // [IndexRecord]s. This may be a one-to-many relationship. A map of Package // ID to Vulnerabilities is returned. Get(ctx context.Context, records []*claircore.IndexRecord, opts GetOpts) (map[string][]*claircore.Vulnerability, error) } Vulnerability is the interface for querying stored Vulnerabilities.

package datastore // import "github.com/quay/claircore/datastore"

type Enrichment interface {
	GetEnrichment(ctx context.Context, kind string, tags []string) ([]driver.EnrichmentRecord, error)
}
    Enrichment is an interface for querying enrichments from the store.
```

Manifest

A Manifest is analogous to an OCI Image Manifest: it defines the order of layers and how to retrieve the them.

package claircore // import "github.com/quay/claircore"

type Manifest struct {
	// content addressable hash. should be able to be computed via
	// the hashes of all included layers
	Hash Digest `json:"hash"`
	// an array of filesystem layers indexed in the same order as the corresponding image
	Layers []*Layer `json:"layers"`
}
    Manifest represents a docker image. Layers array MUST be indexed in the
    order that image layers are stacked.

Matcher

A Matcher performs the heavy lifting of matching manifest contents to relevant vulnerabilities. These implementations provide the smarts for understanding if a particular artifact in a layer is vulnerable to a particular advisory in the database.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Matcher interface {
	// a unique name for the matcher
	Name() string
	// Filter informs the Controller if the implemented Matcher is interested in the provided IndexRecord.
	Filter(record *claircore.IndexRecord) bool
	// Query informs the Controller how it should match packages with vulnerabilities.
	// All conditions are logical AND'd together.
	Query() []MatchConstraint
	// Vulnerable informs the Controller if the given package is affected by the given vulnerability.
	// for example checking the "FixedInVersion" field.
	Vulnerable(ctx context.Context, record *claircore.IndexRecord, vuln *claircore.Vulnerability) (bool, error)
}
    Matcher is an interface which a Controller uses to query the vulnstore for
    vulnerabilities.

The Filter method is used to inform Libvuln the provided artifact is interesting. The Query method tells Libvuln how to query the security advisory database. The Vulnerable method reports whether the provided package is vulnerable to the provided vulnerability. Typically, this would perform a version check between the artifact and the vulnerability in question.

Package Scanner

A Package Scanner should discover any packages found within the given layer. It is OK for to discover no packages within a layer.

package indexer // import "github.com/quay/claircore/indexer"

type PackageScanner interface {
	VersionedScanner
	// Scan performs a package scan on the given layer and returns all
	// the found packages
	Scan(context.Context, *claircore.Layer) ([]*claircore.Package, error)
}
    PackageScanner provides an interface for unique identification or a
    PackageScanner and a Scan method for extracting installed packages from an
    individual container layer

func NewPackageScannerMock(name, version, kind string) PackageScanner

RemoteMatcher

RemoteMatcher is an additional interface a Matcher may implement to skip the database for matching results and use an external API.

package driver // import "github.com/quay/claircore/libvuln/driver"

type RemoteMatcher interface {
	QueryRemoteMatcher(ctx context.Context, records []*claircore.IndexRecord) (map[string][]*claircore.Vulnerability, error)
}
    RemoteMatcher is an additional interface that a Matcher can implement.

    When called the interface can invoke the remote matcher using an HTTP API to
    fetch new vulnerabilities associated with the given IndexRecords.

    The information retrieved from this interface won't be persisted into the
    claircore database.

Repository Scanner

A RepositoryScanner should identify any repositories discovered in the provided layer. It is OK for the scanner to identify no repositories.

package indexer // import "github.com/quay/claircore/indexer"

type RepositoryScanner interface {
	VersionedScanner
	Scan(context.Context, *claircore.Layer) ([]*claircore.Repository, error)
}

Resolver

A Resolver is used to analyze and modify the post-coalesced index report. This is useful for operations that need all context from an index report.

package indexer // import "github.com/quay/claircore/indexer"

type Resolver interface {
	Resolve(context.Context, *claircore.IndexReport, []*claircore.Layer) *claircore.IndexReport
}
    Resolver is used for any reasoning that needs to be done with all the layers
    in context.

    Resolvers are called at the end of the coalesce step when reports from
    separate scanners are merged.

Any Resolvers' Resolve() methods are called (in no set order) at the end of the coalesce step after reports from separate scanners are merged.

RPCScanner

RPCScanner is an optional interface a Scanner may implement. When implemented, the scanner's Configure method will be called with a ConfigDeserializer function and an HTTP client. The Scanner may pass its config as an argument to the ConfigDeserializer function to populate the struct and use the HTTP client for any remote access necessary during the scanning process.

package indexer // import "github.com/quay/claircore/indexer"

type RPCScanner interface {
	Configure(context.Context, ConfigDeserializer, *http.Client) error
}
    RPCScanner is an interface scanners can implement to receive configuration
    and denote that they expect to be able to talk to the network at run time.

package indexer // import "github.com/quay/claircore/indexer"

type RPCScanner interface {
	Configure(context.Context, ConfigDeserializer, *http.Client) error
}
    RPCScanner is an interface scanners can implement to receive configuration
    and denote that they expect to be able to talk to the network at run time.

package indexer // import "github.com/quay/claircore/indexer"

type ConfigDeserializer func(interface{}) error ConfigDeserializer can be thought of as an Unmarshal function with the byte slice provided.

This will typically be something like (*json.Decoder).Decode.

Updater

An Updater is responsible for performing run-time fetching and parsing of a security database. The returned vulnerabilities will be written to claircore's database and used in vulnerability matching.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface {
	Name() string
	Fetcher
	Parser
}
    Updater is an aggregate interface combining the method set of a Fetcher and
    a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface {
	Name() string
	Fetcher
	Parser
}
    Updater is an aggregate interface combining the method set of a Fetcher and
    a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fetcher interface { Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error) } Fetcher is an interface which is embedded into the Updater interface.

When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed

If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.

If the conent has not change an Unchanged error should be returned.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface { Name() string Fetcher Parser } Updater is an aggregate interface combining the method set of a Fetcher and a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fetcher interface {
	Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error)
}
    Fetcher is an interface which is embedded into the Updater interface.

    When called the interface should determine if new security advisory data
    is available. Fingerprint may be passed into in order for the Fetcher to
    determine if the contents has changed

    If there is new content Fetcher should return a io.ReadCloser where the new
    content can be read. Optionally a fingerprint can be returned which uniquely
    identifies the new content.

    If the conent has not change an Unchanged error should be returned.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Parser interface { // Parse should take an io.ReadCloser, read the contents, parse the contents // into a list of claircore.Vulnerability structs and then return // the list. Parse should assume contents are uncompressed and ready for parsing. Parse(ctx context.Context, contents io.ReadCloser) ([]*claircore.Vulnerability, error) } Parser is an interface which is embedded into the Updater interface.

Parse should be called with an io.ReadCloser struct where the contents
of a security advisory database can be read and parsed into an array of
*claircore.Vulnerability

package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface { Name() string Fetcher Parser } Updater is an aggregate interface combining the method set of a Fetcher and a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fetcher interface {
	Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error)
}
    Fetcher is an interface which is embedded into the Updater interface.

    When called the interface should determine if new security advisory data
    is available. Fingerprint may be passed into in order for the Fetcher to
    determine if the contents has changed

    If there is new content Fetcher should return a io.ReadCloser where the new
    content can be read. Optionally a fingerprint can be returned which uniquely
    identifies the new content.

    If the conent has not change an Unchanged error should be returned.

package driver // import "github.com/quay/claircore/libvuln/driver"

Parse should be called with an io.ReadCloser struct where the contents
of a security advisory database can be read and parsed into an array of
*claircore.Vulnerability

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fingerprint string
    Fingerprint is some identifying information about a vulnerability database.
```
```
package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface {
	Name() string
	Fetcher
	Parser
}
    Updater is an aggregate interface combining the method set of a Fetcher and
    a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fetcher interface { Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error) } Fetcher is an interface which is embedded into the Updater interface.

When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed

If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.

If the conent has not change an Unchanged error should be returned.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Parser interface {
	// Parse should take an io.ReadCloser, read the contents, parse the contents
	// into a list of claircore.Vulnerability structs and then return
	// the list. Parse should assume contents are uncompressed and ready for parsing.
	Parse(ctx context.Context, contents io.ReadCloser) ([]*claircore.Vulnerability, error)
}
    Parser is an interface which is embedded into the Updater interface.

    Parse should be called with an io.ReadCloser struct where the contents
    of a security advisory database can be read and parsed into an array of
    *claircore.Vulnerability

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fingerprint string Fingerprint is some identifying information about a vulnerability database.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Configurable interface {
	Configure(context.Context, ConfigUnmarshaler, *http.Client) error
}
    Configurable is an interface that Updaters can implement to opt-in to having
    their configuration provided dynamically.
```
```
package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface {
	Name() string
	Fetcher
	Parser
}
    Updater is an aggregate interface combining the method set of a Fetcher and
    a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fetcher interface { Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error) } Fetcher is an interface which is embedded into the Updater interface.

When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed

If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.

If the conent has not change an Unchanged error should be returned.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Parser interface {
	// Parse should take an io.ReadCloser, read the contents, parse the contents
	// into a list of claircore.Vulnerability structs and then return
	// the list. Parse should assume contents are uncompressed and ready for parsing.
	Parse(ctx context.Context, contents io.ReadCloser) ([]*claircore.Vulnerability, error)
}
    Parser is an interface which is embedded into the Updater interface.

    Parse should be called with an io.ReadCloser struct where the contents
    of a security advisory database can be read and parsed into an array of
    *claircore.Vulnerability

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fingerprint string Fingerprint is some identifying information about a vulnerability database.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Configurable interface {
	Configure(context.Context, ConfigUnmarshaler, *http.Client) error
}
    Configurable is an interface that Updaters can implement to opt-in to having
    their configuration provided dynamically.

package driver // import "github.com/quay/claircore/libvuln/driver"

type ConfigUnmarshaler func(interface{}) error ConfigUnmarshaler can be thought of as an Unmarshal function with the byte slice provided, or a Decode function.

The function should populate a passed struct with any configuration
information.

UpdaterSetFactory

An UpdaterSetFactory is a factory for runtime construction and configuration for Updaters.

package driver // import "github.com/quay/claircore/libvuln/driver"

type UpdaterSetFactory interface {
	UpdaterSet(context.Context) (UpdaterSet, error)
}
    UpdaterSetFactory is used to construct updaters at run-time.

func StaticSet(s UpdaterSet) UpdaterSetFactory

package driver // import "github.com/quay/claircore/libvuln/driver"

type UpdaterSetFactory interface {
	UpdaterSet(context.Context) (UpdaterSet, error)
}
    UpdaterSetFactory is used to construct updaters at run-time.

func StaticSet(s UpdaterSet) UpdaterSetFactory

package driver // import "github.com/quay/claircore/libvuln/driver"

type UpdaterSet struct { // Has unexported fields. } UpdaterSet holds a deduplicated set of updaters.

func NewUpdaterSet() UpdaterSet func (s *UpdaterSet) Add(u Updater) error func (s *UpdaterSet) Merge(set UpdaterSet) error func (s *UpdaterSet) RegexFilter(regex string) error func (s *UpdaterSet) Updaters() []Updater

VersionFilter

VersionFilter is an additional interface a Matcher may implement. If implemented, Libvuln will attempt to use the database and the normalized version field of a package to filter vulnerabilities in the database. This is an opt-in optimization for when a package manager's version scheme can be normalized into a claircore.Version.

package driver // import "github.com/quay/claircore/libvuln/driver"

type VersionFilter interface {
	VersionFilter()
	// VersionAuthoritative reports whether the Matcher trusts the database-side
	// filtering to be authoritative.
	//
	// A Matcher may return false if it's using a versioning scheme that can't
	// be completely normalized into a claircore.Version.
	VersionAuthoritative() bool
}
    VersionFilter is an additional interface that a Matcher can implement to
    opt-in to using normalized version information in database queries.

Versioned Scanner

A versioned scanner is typically embedded into other scanner types. It drives claircore's ability to register and understand when updaters have been changed. Functions that want to work with a generic scanner type should use a VersionedScanner.

Implementers of this interface must provide a unique name. Making changes to a scanner's implementation must return a new value from Version. Implementers must return the correct kind: one of "package", "distribution", or "repository"

package indexer // import "github.com/quay/claircore/indexer"

type VersionedScanner interface {
	// unique name of the distribution scanner.
	Name() string
	// version of this scanner. this information will be persisted with the scan.
	Version() string
	// the kind of scanner. currently only package is implemented
	Kind() string
}
    VersionedScanner can be embedded into specific scanner types. This allows
    for methods and functions which only need to compare names and versions of
    scanners not to require each scanner type as an argument.

Vulnerability Report

A Vulnerability Report is a structure describing a specific manifest, its contents, and the vulnerabilities affecting its contents.

package claircore // import "github.com/quay/claircore"

type VulnerabilityReport struct {
	// the manifest hash this vulnerability report is describing
	Hash Digest `json:"manifest_hash"`
	// all discovered packages in this manifest keyed by package id
	Packages map[string]*Package `json:"packages"`
	// all discovered distributions in this manifest keyed by distribution id
	Distributions map[string]*Distribution `json:"distributions"`
	// all discovered repositories in this manifest keyed by repository id
	Repositories map[string]*Repository `json:"repository"`
	// a list of environment details a package was discovered in keyed by package id
	Environments map[string][]*Environment `json:"environments"`
	// all discovered vulnerabilities affecting this manifest
	Vulnerabilities map[string]*Vulnerability `json:"vulnerabilities"`
	// a lookup table associating package ids with 1 or more vulnerability ids. keyed by package id
	PackageVulnerabilities map[string][]string `json:"package_vulnerabilities"`
	// a map of enrichments keyed by a type.
	Enrichments map[string][]json.RawMessage `json:"enrichments"`
}
    VulnerabilityReport provides a report of packages and their associated
    vulnerabilities.

A Vulnerability Report is package focused.

Unpacking a report is done by mapping the keys in the PackageVulnerabilities field to the data structures in other lookup maps.

For example:

	for pkgID, vulnIDS := range report.PackageVulnerabilities {
		// get package data structure
		pkg := report.Packages[pkgID]

		for _, vulnID := range vulnIDS {
			vuln := report.Vulnerabilities[vulnID]
			fmt.Printf("package %+v affected by vuln %+v", pkg, vuln)
		}
	}

Contributors

These topics cover helpful tips for contributing to Claircore.

How to contribute

The preferred workflow is to fork the quay/claircore repository, push a feature branch to the new fork, then open a pull request. All pull requests should be targeted to the main branch outside of exceptional circumstances.

Testing

As many tests as possible should run with the standard go test invocations. Adding the special tag integration (e.g. go test -tags integration ./...) will also run "integration" tests. The project interprets "integration" tests to mean any test that would need external resources, such as:

External web servers
External network access
Out-of-process databases
Large test fixtures

After at least one run with the integration tag, the tests should cache needed resources and run as many tests as possible. See also the test/integration package.

Pull Requests

The Pull Request (PR) is the unit of code review. Claircore's review flow treats a feature branch as a stack of patches to be applied. That is to say, the feature branch should be rebased onto the target branch and have well-organized commits. Merge commits are disallowed. If the author would prefer to not rewrite commit history while working through reviews, fixup commits are the suggested way to achieve that. As many requirements as possible are enforced by CI, like:

Commits being signed off
Commit messages having a properly formed subject
Go modules being tidied

Please use the "draft" option if the branch is not ready. Please enable the "allow edits by maintainers" option.

The maintainers may rebase, push, and merge contributors' branches. This may necessitate doing a git reset <remote>/<branch> to update a local branch.

Conventions

Git commits should be formatted like "subject: summary" and avoid going over 80 characters per line. The "subject" is usually the package affected by the commit (like jar or rhel -- the relative path isn't needed) but sometimes a broader category (like docs, all, or cicd) is OK.

All the helper scripts should handle the "normal" convention (origin is quay/claircore and fork is one's personal fork) and the "British" convention (origin is one's personal fork and upstream is quay/claircore).

More detailed contributor documentation can be found in the project documentation.

The claircore project has switched from a git log based changelog to a git notes based changelog.

This has the benefit of making the changelog more human-friendly, as it can have prose describing changes now, but makes adding entries a bit more involved. A full understanding of git notes is helpful for working with the changelog, but not required. If the reader has worked with the notes feature before, the changelog entries are stored under the changelog ref. For other users, there are some helper scripts in .github/scripts.

Basics of `git notes`

Git notes is a mechanism for storing additional information alongside commits without modifying the commits. It does this by creating a ref full of files named after the commits, with their contents being the notes. This scheme requires some special care and tooling -- see the documentation for more information.

Helper scripts

The primary helper script is changelog-edit. It allows a user to sync down notes, edit an entry, or both. See the output of the h flag for more information.

The other script of interest is changelog-render, which can be used to render out the changelog on demand, assuming the changelog notes have been pulled locally.

The changelog-update script uses changelog-render to add to the CHANGELOG.md file in the repository root.

Formatting

Broadly, changelog entries should be formatted like commit messages without any trailers. Entries are turned into list items, with the subject being the bullet point and the body of the entry being the "body" of the item, or hidden behind details elements when using HTML-enabled output.

The entries are almost always rendered as markdown, so using minimal markdown is OK. Anything requiring excessive markdown is probably better served as documentation proper, rather than a changelog entry.

Commit Style

The Claircore project utilizes well structured commits to keep the history useful and help with release automation. We suggest signing off on your commits as well.

A typical commit will take on the following structure:

<scope>: <subject>

<body>
Fixes #1
Pull Request #2

Signed-Off By: <email>

The header of the commit is regexp checked before commit and your commit will be kicked back if it does not conform.

Scope

This is the section of code this commit influences.

You will often see scopes such as "notifier", "auth", "chore", "cicd".

Subject

Subject is a short and concise summary of the change the commit is introducing. It should be a sentence fragment without starting capitalization and ending punctuation and limited to about 60 characters, to allow for the scope prefix and decoration in the git log.

Body

Body should be full of detail.

Explain what this commit is doing and why it is necessary.

You may include references to issues and pull requests as well.

Local Development

A local development environment is implemented via docker-compose.

Usage

Several make targets are defined for working with the local development environment.

claircore-db-up - creates just the claircore database useful for running integration tests without test servers
claircore-db-restart - destroys and recreates a fresh database. localhost:5434

Tests

Several make targets are defined for working with tests.

integration - run the integration test suite. requires the claircore-db to be up. run `make clair-db-up` before this target
unit - run the unit test suite.
bench -  runs the benchmarks
integration-v - runs the integration test suite with verbose
unit-v - runs the unit test suite with verbose

Logging

All the logging in claircore is done with zerolog via context.Context values. The zlog package takes OpenTelemetry labels and attaches them to zerolog events.

This allows for claircore's logging to be used consistently throughout all the packages without having unintended prints to stderr.

How to Log

Adding Context

In a function, use zlog to add key-value pairs of any relevant context:

	ctx = zlog.ContextWithValues(ctx,
		"component", "Example.Logger")

Alternatively, the go.opentelemetry.io/otel/baggage package can be used for more explicit control around the baggage values.

Logging style

Constant Messages

Zerolog emits lines when the Msg or Msgf methods are called. Project style is to not use Msgf. Any variable data should be set as key-value pairs on the Event object.

For example, don't do this:

	zlog.Info(ctx).Msgf("done at: %v", time.Now())

Do this instead:

	zlog.Info(ctx).
		Time("time", time.Now()).
		Msgf("done")

Grammar

When noting the change during a chunk of work, make sure that the log messages scan as visually similar. Usually, this means formatting messages into "${process} ${event}". For example:

frob start
frob initialized
frob ready
frob success
frob done

Is much easier to scan than:

starting to frob
initialized frobber
ready for frobbing
did frob
done with frobing

Don't log and return

When handling an error, code should only log it if it does not propagate it. The code that ultimately handles the error is responsible for deciding what to do with it. Logging and returning ends up with the same message repeated multiple times in the logs.

Levels

Claircore attempts to have consistent leveled logging. The rules for figuring out what level to use is:

Panic

There's some occurrence that means the process won't work correctly.
Fatal

Unused, because it prevents defers from running.
Error

Something unexpected occurred and the process can continue, but a human needs to be notified. An error will be returned for this request.
Warn

Something unexpected occurred and the process can continue. An error will be returned for this request.
Info

Some information that may be useful to an operator. Examples include a timer-based process starting and ending, a user request starting and ending, or a summary of work done.
Debug

Some information that may be useful to a developer. Examples include entering and exiting functions, stepping through a function, or specific file paths used while work is being done.

Here's various codebase conventions that don't have dedicated pages:

URLs URLs in code should be annotated with a //doc:url directive comment. See the the internal/cmd/mdbook-injecturls command for documentation on how the preprocessor works. The list of keywords isn't an allowlist, so an invocation like the following should list the ones actually used in the documentation using a command like git grep injecturls -- :/docs/*.md.

Tests

Tests in the claircore module may use various helpers underneath the test directory. Using these packages outside of testing code is disallowed. Assert packages are disallowed; the go-cmp package is the only external package helper allowed.

Tests that use external resources or generate test fixtures should be annotated according to the integration package.

Caching

Tests using the integration package cache generated and downloaded assets into a directory named clair-testing inside the directory reported by os.UserCacheDir. For example, on a Linux system, the cache directory will be (in sh notation) ${XDG_CACHE_HOME-$HOME/.cache}/clair-testing.

Go Version

Claircore endeavors to declare a go version in the go.mod directive that is the minimum required for all the language features to work as expected. The go version will also be updated if there is deemed to be a security fix that could directly affect claircore's codebase.

Claircore Documentation