Claircore
Claircore is the engine behind the Clair v4 container security solution. The Claircore package exports our domain models, interfaces necessary to plug into our business logic, and a default set of implementations. This default set of implementations define our support matrix and consists of the following distributions and languages:
- Ubuntu
- Debian
- RHEL
- Red Hat Container First content
- SUSE
- Oracle
- Alpine
- AWS Linux
- VMWare Photon
- Python
- Java
- Go
- Ruby
Claircore relies on PostgreSQL for its persistence and the library will handle migrations if configured to do so.
The diagram below is a high level overview of Claircore's architecture.
graph LR subgraph Indexer im[Image Manifest] libindex[Libindex] iir[IndexReport] im --> libindex --> iir end iir -.-> db[(Database)]
graph LR subgraph Matcher mir[IndexReport] libvuln[Libvuln] vr[VulnerabilityReport] mir --> libvuln --> vr end db[(Database)] -.-> mir
When a claircore.Manifest is submitted to Libindex, the library will index its constituent parts and create a report with its findings.
When a claircore.IndexReport is provided to Libvuln, the library will discover vulnerabilities affecting it and generate a claircore.VulnerabilityReport.
Getting Started
The following documentation helps a beginner learn to use Claircore.
Libindex Usage
Libindex is the Go package responsible for fetching container image layers, identifying packages, distributions, and repositories within these layers, and computing a final coalesced Index Report.
An Index Report is primarily used as input to Libvuln's vulnerability matching process.
Usage
Libindex is runtime constructed via the libindex.New
method. New requires an libindex.Opts
struct.
Options
package libindex // import "github.com/quay/claircore/libindex"
type Options struct {
// Store is the interface used to persist and retrieve results of indexing.
Store indexer.Store
// Locker provides system-wide locks. If the indexing work is distributed the
// lock should be backed by a distributed store.
Locker LockSource
// FetchArena is an interface tied to the lifecycle of LibIndex to enable management
// of the filesystem while separate processes are dealing with layers, for example:
// you can reference count downloaded layer files to avoid racing.
FetchArena indexer.FetchArena
// ScanLockRetry specifies how often we should try to acquire a lock for scanning a
// given manifest if lock is taken.
ScanLockRetry time.Duration
// LayerScanConcurrency specifies the number of layers to be scanned in parallel.
LayerScanConcurrency int
// LayerFetchOpt is unused and kept here for backwards compatibility.
LayerFetchOpt interface{}
// NoLayerValidation controls whether layers are checked to actually be
// content-addressed. With this option toggled off, callers can trigger
// layers to be indexed repeatedly by changing the identifier in the
// manifest.
NoLayerValidation bool
// ControllerFactory provides an alternative method for creating a scanner during libindex runtime
// if nil the default factory will be used. useful for testing purposes
ControllerFactory ControllerFactory
// Ecosystems a list of ecosystems to use which define which package databases and coalescing methods we use
Ecosystems []*indexer.Ecosystem
// ScannerConfig holds functions that can be passed into configurable
// scanners. They're broken out by kind, and only used if a scanner
// implements the appropriate interface.
//
// Providing a function for a scanner that's not expecting it is not a fatal
// error.
ScannerConfig struct {
Package, Dist, Repo, File map[string]func(interface{}) error
}
Resolvers []indexer.Resolver
}
Options are dependencies and options for constructing an instance of
libindex
The above outlines the relevant bits of the Options structure.
Store is required needs to be an object that satisfies the indexer.Store interface.
Locker is required and needs to satisfy the LockSource interface.
FetchArena is required and needs to satify the FetchArena interface.
Providing a nil "Ecosystems" slice will supply the default set, instructing Libindex to index for all supported content in a layer, and is typically desired.
Construction
Constructing Libindex is straight forward.
opts := new(libindex.Options)
// Populate with desired settings...
lib, err := libindex.New(ctx, opts, http.DefaultClient)
if err != nil {
panic(err)
}
defer lib.Close(ctx) // Remember to cleanup when done.
The constructing code should provide a valid Context tied to some lifetime.
Indexing
Indexing is the process of submitting a manifest to Libindex, fetching the manifest's layers, indexing their contents, and coalescing a final Index Report.
Coalescing is the act of computing a final set of contents (packages, distributions, repos) from a set of layers. Since layers maybe shared between many manifests, the final contents of a manifest must be computed.
To perform an Index you must provide a claircore.Manifest data struture to the Index method. The Manifest data structure describes an image manifest's layers and where they can be fetched from.
m := new(claircore.Manifest)
// Populate somehow ...
ir, err := lib.Index(ctx, m)
if err != nil {
panic(err)
}
The Index method will block until an claircore.IndexReport is returned. The context should be bound to some valid lifetime such as a request.
As the Indexer works on the manifest it will update its database throughout the process. You may view the status of an index report via the "IndexReport" method.
ir, ok, err := lib.IndexReport(ctx, m.Hash)
if err != nil {
panic(err)
}
Libindex performs its work incrementally and saves state as it goes along. If Libindex encounters an intermittent error during the index (for example, due to network failure while fetching a layer), when the manifest is resubmitted only the layers not yet indexed will be fetched and processed.
State
Libindex treats layers as content addressable. Once a layer identified by a particular hash is indexed its contents are definitively known. A request to re-index a known layer results in returning the previous successful response.
This comes in handy when dealing with base layers. The Ubuntu base layer is seen very often across container registries. Treating this layer as content addressable precludes the need to fetch and index the layer every time Libindex encounters it in a manifest.
There are times where re-indexing the same layer is necessary however. At the point where Libindex realizes a new version of a component has not indexed a layer being submitted it will perform the indexing operation.
A client must notice that Libindex has updated one of its components and subsequently resubmit Manifests. The State endpoint is implemented for this reason.
Clients may query the State endpoint to receive an opaque string acting as a cookie, identifying a unique state of Libindex. When a client sees this cookie change it should re-submit manifests to Libindex to obtain a new index report.
state, err := lib.State(ctx)
if err != nil {
panic(err)
}
if state == prevState {
// Nothing to do.
return
}
// Otherwise, re-index manifest.
AffectedManifests
Libindex is capable of providing a client with all manifests affected by a set of vulnerabilities. This functionality is designed for use with a notification mechanism.
var vulns []claircore.Vulnerability
affected, err := lib.AffectedManifests(ctx, vulns)
if err != nil {
panic(err)
}
for manifest, vulns := range affected.VulnerableManifests {
for _, vuln := range vulns {
fmt.Printf("vuln affecting manifest %s: %+v", manifest, vuln)
}
}
The slice of vulnerabilities returned for each manifest hash will be sorted by
claircore.NormalizedSeverity
in "most severe" descending order.
Libvuln Usage
Libvuln is the Go package responsible for keeping the database of vulnerabilities consistent, matching container image contents with vulnerabilities, and reporting diffs between updates of the same security database.
Usage
Libvuln
is runtime constructed via the libvuln.New
method. New
requires a
libvuln.Opts
struct.
Options
package libvuln // import "github.com/quay/claircore/libvuln"
type Options struct {
// Store is the interface used to persist and retrieve vulnerabilities
// for of matching.
Store datastore.MatcherStore
// Locker provides system-wide locks for the updater subsystem. If the
// matching work is distributed the lock should be backed by a distributed
// store.
Locker LockSource
// An interval on which Libvuln will check for new security database
// updates.
//
// This duration will have jitter added to it, to help with smearing load on
// installations.
UpdateInterval time.Duration
// A slice of strings representing which updaters libvuln will create.
//
// If nil all default UpdaterSets will be used.
//
// The following sets are supported:
// "alpine"
// "aws"
// "debian"
// "oracle"
// "photon"
// "pyupio"
// "rhel"
// "suse"
// "ubuntu"
UpdaterSets []string
// A list of out-of-tree updaters to run.
//
// This list will be merged with any defined UpdaterSets.
//
// If you desire no updaters to run do not add an updater
// into this slice.
Updaters []driver.Updater
// A slice of strings representing which
// matchers will be used.
//
// If nil all default Matchers will be used
//
// The following names are supported by default:
// "alpine"
// "aws"
// "debian"
// "oracle"
// "photon"
// "python"
// "rhel"
// "suse"
// "ubuntu"
MatcherNames []string
// Config holds configuration blocks for MatcherFactories and Matchers,
// keyed by name.
MatcherConfigs map[string]driver.MatcherConfigUnmarshaler
// A list of out-of-tree matchers you'd like libvuln to
// use.
//
// This list will me merged with the default matchers.
Matchers []driver.Matcher
// Enrichers is a slice of enrichers to use with all VulnerabilityReport
// requests.
Enrichers []driver.Enricher
// UpdateWorkers controls the number of update workers running concurrently.
// If less than or equal to zero, a sensible default will be used.
UpdateWorkers int
// UpdateRetention controls the number of updates to retain between
// garbage collection periods.
//
// The lowest possible value is 2 in order to compare updates for notification
// purposes.
UpdateRetention int
// If set to true, there will not be a goroutine launched to periodically
// run updaters.
DisableBackgroundUpdates bool
// UpdaterConfigs is a map of functions for configuration of Updaters.
UpdaterConfigs map[string]driver.ConfigUnmarshaler
// Client is an http.Client for use by all updaters.
//
// Must be set.
Client *http.Client
}
The above outlines the relevant bits of the Opts structure.
Construction
Constructing Libvuln is straight forward.
opts := new(libvuln.Options)
// Populate with desired settings...
lib, err := libvuln.New(ctx, opts)
if err != nil {
panic(err)
}
defer lib.Close(ctx)
The constructing code should provide a valid Context tied to some lifetime.
On construction, New
will block until the security databases are initialized.
Expect some delay before this method returns.
Scanning
Scanning is the process of taking a claircore.IndexReport
comprised of a
Manifest's content and determining which vulnerabilities affect the Manifest. A
claircore.VulnerabilityReport
will be returned with these details.
m := new(claircore.Manifest)
// Populate somehow ...
ir, err := indexer.Index(ctx, m)
if err != nil {
panic(err)
}
vr, err := lib.Scan(ctx, ir)
if err != nil {
panic(err)
}
In the above example, Libindex
is used to generate a claircore.IndexReport
.
The index report is then provided to Libvuln
and a subsequent vulnerability
report identifying any vulnerabilities affecting the manifest is returned.
Updates API
By default, Libvuln manages a set of long running updaters responsible for periodically fetching and loading new advisory contents into its database. The Updates API allows a client to view and manipulate aspects of the update operations that updaters perform.
In this getting started guide, we will only cover the two methods most interesting to new users.
UpdateOperations
This API provides a list of recent update operations performed by implemented updaters.
The UpdateOperation
slice returned will be sorted by latest timestamp descending.
ops, err := lib.UpdateOperations(ctx, `updater`)
if err != nil {
panic(err)
}
for updater, ops := range ops {
fmt.Printf("ops for updater %s, %+v", updater, ops)
}
UpdateDiff
Mostly used by the Clair v4 notification subsystem, this endpoint will provide the caller with any removed or added vulnerabilities between two update operations. Typically a diff takes places against two versions of the same data source. This is useful to inform downstream applications what new vulnerabilities have entered the system.
ops, err := lib.UpdateOperations(ctx, `updater`)
if err != nil {
panic(err)
}
for upd, ops := range ops {
fmt.Printf("updater: %v", upd)
diff, err := lib.UpdateDiff(ctx, ops[1].Ref, ops[0].Ref)
if err != nil {
panic(err)
}
for _, vuln := range diff.Added {
fmt.Printf("vuln %+v added in %v", vuln, diff.Cur.Ref)
}
}
Concepts
The following sections give a conceptual overview of how Claircore works internally.
Vulnerability Matching
The following describes a successful scan.
- Updaters have ran either in the background on an interval or an offline loader has been ran.
- A Manifest is provided to Libindex. Libindex fetches all the layers, runs all scanner types on each layer, persists all artifacts found in each layer, and computes an IndexReport.
- A IndexReport is provided to Libvuln.
- Libvuln creates a stream of IndexRecord structs from the IndexReport and concurrently streams these structs to each configured Matcher.
- Libvuln computes a VulnerabilityReport aggregating all vulnerabilities discovered by all configured Matcher implementations.
- Sometime later the security advisory database is updated and a new request to Libvuln will present updated vulnerability data.
Indexer
claircore/indexer
The Indexer package performs Libindex's heavy lifting. It is responsible for retreiving Manifest layers, parsing the contents of each layer, and computing an IndexReport.
To perform this action in incremental steps the Indexer is implemented as a finite state machine. At each state transition the Indexer persists an updated IndexReport to its datastore.
States
The following diagram expresses the possible states of the Indexer:
stateDiagram-v2 state if_indexed <<choice>> [*] --> CheckManifest CheckManifest --> if_indexed if_indexed --> [*]: Indexed if_indexed --> FetchLayers: Unindexed FetchLayers --> ScanLayers ScanLayers --> Coalesce Coalesce --> IndexManifest IndexManifest --> IndexFinished IndexFinished --> [*] %% These notes make the diagram unreadable :/ %% note left of CheckManifest: Determine if this manifest has been indexed previously. %% note right of FetchLayers: Determine which layers need to be indexed and fetch them. %% note right of ScanLayers: Concurrently run needed Indexers on layers. %% note right of Coalesce: Compute the final contents of the container image. %% note right of IndexManifest: Associate all the discovered data. %% note right of IndexFinished: Persist the results.
Data Model
The Indexer data model focuses on content addressable hashes as primary keys, the deduplication of package/distribution/repostitory information, and the recording of scan artifacts. Scan artifacts are unique artifacts found within a layer which point to a deduplicated general package/distribution/repository record.
The following diagram outlines the current Indexer data model.
%%{init: {"er":{"layoutDirection":"RL"}} }%% erDiagram ManifestLayer many to 1 Manifest: "" ManifestLayer many to 1 Layer: "" ScannedLayer many to 1 Layer: "" ScannedLayer many to 1 Scanner: "" ScannedManifest many to 1 Manifest: "" ScannedManifest many to 1 Scanner: "" TYPE_ScanArtifact 1 to 1 Layer: "" TYPE_ScanArtifact 1 to 1 Scanner: "" TYPE_ScanArtifact 1 to 1 TYPE: "" ManifestIndex many to 1 Manifest: "" ManifestIndex 1 to zero or one TYPE: "" IndexReport 1 to 1 Manifest: "cached result"
Note that TYPE
stands in for each of the Indexer types (i.e. Package
, Repository
, etc.).
HTTP Resources
Indexers as currently built may make network requests. This is an outstanding issue. The following are the URLs used.
- https://search.maven.org/solrsearch/select
- https://catalog.redhat.com/api/containers/
- https://security.access.redhat.com/data/metrics/repository-to-cpe.json
- https://security.access.redhat.com/data/metrics/container-name-repos-map.json
Matcher Architecture
claircore/internal/matcher
claircore/libvuln/driver
The Matcher architecture is based on a data flow application.
graph TD start[Libvuln.Scan] finish[Merge into VulnerabilityReport] start ---> RemoteMatcher & Matcher ---> finish subgraph RemoteMatcher ra[Filter interested packages] api[Make API calls] rv[Decide vulnerability] ra --> api --> rv end subgraph Matcher dbv[Check versions in-database] ma[Filter interested packages] adv[Retrieve vulnerability information] mv[Deicide vulnerability] ma --> adv --> mv adv -.-> dbv -.-> mv end
When Libvuln's Scan method is called with an IndexReport it will begin the process of matching container contents with vulnerabilities.
Each configured Matcher will be instantiated concurrently. Depending on the interfaces the Matcher implements, one of the possible data flows will occur in the diagram above.
The provided IndexReport will be unpacked into a stream of IndexRecord structs. Each Matcher will evaluate each record in the stream and determine if the IndexRecord is vulnerable to a security advisory in their responsible databases.
Once each Matcher returns the set of vulnerabities, Libvuln will merge the results into a VulnerabilityReport and return this to the client.
HTTP Resources
NOTE: Remote matchers are being considered for removal.
"Remote matchers" may make HTTP requests during the matcher flow. These requests are time-bound and errors are not reported. The following are the URLs used.
Severity Mapping
Claircore will normalize a security databases's severity string to a set of defined values.
Clients may use the NormalizedSeverity
field on a claircore.Vulnerability
to react to vulnerability severities without needing to know each security database's severity strings.
All strings used in the mapping tables are identical to the strings found within the relevant security database.
Claircore Severity Strings
The following are severity strings Claircore will normalize others to. Clients can guarantee one of these strings will be associated with a claircore.Vulnerability.
- Unknown
- Negligible
- Low
- Medium
- High
- Critical
Alpine Mapping
The Alpine SecDB database does not provide severity information. All vulnerability severities will be Unknown.
Alpine Severity | Claircore Severity |
---|---|
* | Unknown |
AWS Mapping
The AWS UpdateInfo database provides severity information.
AWS Severity | Claircore Severity |
---|---|
low | Low |
medium | Medium |
important | High |
critical | Critical |
Debian Mapping
The Debian security tracker data provides severity information.
Debian Severity | Claircore Severity |
---|---|
unimportant | Low |
low | Medium |
medium | High |
high | Critical |
* | Unknown |
Oracle Mapping
The Oracle OVAL database provides severity information.
Oracle Severity | Claircore Severity |
---|---|
N/A | Unknown |
LOW | Low |
MODERATE | Medium |
IMPORTANT | High |
CRITICAL | Critical |
RHEL Mapping
The RHEL OVAL database provides severity information.
RHEL Severity | Claircore Severity |
---|---|
None | Unknown |
Low | Low |
Moderate | Medium |
Important | High |
Critical | Critical |
SUSE Mapping
The SUSE OVAL database provides severity information.
SUSE Severity | Claircore Severity |
---|---|
None | Unknown |
Low | Low |
Moderate | Medium |
Important | High |
Critical | Critical |
Ubuntu Mapping
The Ubuntu OVAL database provides severity information.
Ubuntu Severity | Claircore Severity |
---|---|
Untriaged | Unknown |
Negligible | Negligible |
Low | Low |
Medium | Medium |
High | High |
Critical | Critical |
Photon Mapping
The Photon OVAL database provides severity information.
Photon Severity | Claircore Severity |
---|---|
Low | Low |
Moderate | Medium |
Important | High |
Critical | Critical |
OSV Mapping
OSV provides severity information via CVSS vectors, when applicable. These are normalized according to the NVD qualitative rating scale. If both v3 and v2 vectors are present, v3 is preferred.
CVSSv3
Base Score | Claircore Severity |
---|---|
0.0 | Negligible |
0.1-3.9 | Low |
4.0-6.9 | Medium |
7.0-8.9 | High |
9.0-10.0 | Critical |
CVSSv2
Base Score | Claircore Severity |
---|---|
0.0-3.9 | Low |
4.0-6.9 | Medium |
7.0-10.0 | High |
Updaters and Defaults
The default updaters are tracked in updater/defaults/defaults.go
.
HTTP Resources
The following are the HTTP hosts and paths that Clair will attempt to talk to in a default configuration. This list is non-exhaustive, as some servers will issue redirects and some request URLs are constructed dynamically.
- https://secdb.alpinelinux.org/
- http://repo.us-west-2.amazonaws.com/2018.03/updates/x86_64/mirror.list
- https://cdn.amazonlinux.com/2/core/latest/x86_64/mirror.list
- https://cdn.amazonlinux.com/al2023/core/mirrors/latest/x86_64/mirror.list
- https://deb.debian.org/
- https://security-tracker.debian.org/tracker/data/json
- https://nvd.nist.gov/feeds/json/cve/1.1/
- https://linux.oracle.com/security/oval/com.oracle.elsa-*.xml.bz2
- https://packages.vmware.com/photon/photon_oval_definitions/
- https://security.access.redhat.com/data/metrics/cvemap.xml
- https://access.redhat.com/security/cve/
- https://security.access.redhat.com/data/oval/v2/PULP_MANIFEST
- https://security.access.redhat.com/data/csaf/v2/vex/
- https://support.novell.com/security/oval/
- https://api.launchpad.net/1.0/
- https://security-metadata.canonical.com/oval/com.ubuntu.*.cve.oval.xml
- https://osv-vulnerabilities.storage.googleapis.com/
How Tos
The following sections provide instructions on accomplish specific goals in Claircore.
Adding Distribution And Language Support
Note: If terms in this document sound foreign check out Getting Started to acquaint yourself with "indexing", "scanning", and "matching"
The claircore team is always open to adding more distributions and languages to the library.
Generally, distributions or languages must provide a security tracker.
All officially supported distributions and languages provide a database of security vulnerabilities.
These databases are maintained by the distribution or language developers and reflect up-to-date CVE and advisory data for their packages.
If your distribution or language does not provide a security tracker or piggy-backs off another distribution's start an issue in our Github issue tracker to discuss further.
Implementing an Updater
The first step to adding your distribution or language to claircore is getting your security tracker's data ingested by Libvuln.
The Updater interfaces are responsible for this task.
An implementer must consider several design points:
- Does the security database provide enough information to parse each entry into a claircore.Vulnerability?
- Each entry must parse into a claircore.Vulnerability.
- Each Vulnerability must contain a package and a repository or distribution field.
- Will the Updater need to be configured at runtime?
- Your updater may implement the Configurable interface. Your matcher will have its "Configuration" method called before use, giving you an opportunity for run time configuration.
- What fields in a parsed claircore.Vulnerability will be present when indexing layer artifacts.
- When implementing an updater you must keep in mind how packages/distributions/repositories will be parsed during index. When indexing a layer a common data model must exist between the possible package/distribution/repository and the parsed Vulnerabilitie's package/distribution/repository fields.
If you are having trouble figuring out these requirements do not hesitate to reach out to us for help.
After you have taken the design points into consideration, you are ready to implement your updater.
Typically you will create a new package named after the source you are adding support for.
Inside this package you can begin implementing the Updater and Updater Set Factory interfaces.
Optionally you may implement the Configurable interface if you need runtime configuration.
It will undoubtly be helpful to look at the examples in the "ubuntu", "rhel", and "debian" packages to get yourself started.
Implementing a Package Scanner
At this point you hopefully have your Updater working, writing vulnerability data into Libvuln's database.
We can now move our attention to package scanning.
A package scanner is responsible for taking a claircore.Layer and parsing the contents for a particular package database or set of files inside the provided tar archive. Once the target files are located the package scanner should parse these files into claircore.Packages and return a slice of these data structures.
Package scanning is context free, meaning no distribution classification has happened yet. This is because manifests are made up of layers, and a layer which holds a package database may not hold distribution information such as an os-release file. A package scanner need only parse a target package database and return claircore.Packages.
You need to implement the Package Scanner interface to achieve this.
Optionally, you may implement the Configurable Scanner if the scanner needs to perform runtime configuration before use.
Keep in mind that its very common for distributions to utilize an existing package manager such as RPM.
If this is the case there's a high likelihood that you can utilize the existing "rpm" or "dpkg" package scanner implementations.
Implementing a Distribution Scanner
Once the package scanner is implemented, tested, and working you can begin implementing a Distribution Scanner.
Implementing a distribution scanner is a design choice. Distributions and repositories are the way claircore matches packages to vulnerabilities.
If your implemented Updater parses vulnerabilities with distribution information you will likely need to implement a distribution scanner. Likewise, if your Updater parses vulnerabilities with repository information (typical with language vulnerabilities) you will likely need to implement a repository scanner.
A distribution scanner, like a package scanner, is provided a claircore.Layer.
The distribution scanner will parse the provided tar archive exhaustively searching for any clue that this layer was derived from your distribution. If you identify that it is, you should return a common distribution used by your Updater implementation. This ensures that claircore can match the output of your distribution scanner with your parsed vulnerabilities.
Optionally, you may implement the Configurable Scanner if the scanner needs to perform runtime configuration before use.
Implementing a Repository Scanner
As mentioned above, implementing a repository scanner is a design choice, often times applicable for language package managers.
If your Updater parses vulnerabilities with a repository field you will likely want to implement a repository scanner.
A repository scanner is used just like a distribution scanner however you will search for any clues that a layer contains your repository and if so return a common data model identifying the repository.
Optionally, you may implement the Configurable Scanner if the scanner needs to perform runtime configuration before use.
Implementing a Coalescer
As you may have noticed, the process of scanning a layer for packages, distribution, and repository information is distinct and separate.
At some point, claircore will need to take all the context-free information returned from layer scanners and create a complete view of the manifest. A coalescer performs this computation.
It's unlikely you will need to implement your own coalescer.
Claircore provides a default "linux" coalescer which will work if your package database is rewritten when modified.
For example, if a Dockerfile's RUN
command causes a change to to dpkg's /var/lib/dpkg/status
database, the resulting manifest will have a copy placed in the associated layer.
However, if your package database does not fit into this model, implementing a coalescer may be necessary.
To implement a coalescer, several details must be understood:
- Each layer only provides a "piece" of the final manifest.
- Because manifests are comprised of multiple copy-on-write layers, some layers may contain package information, distribution information, repository information, any combination of those, or no information at all.
- An OS may have a "dist-upgrade" performed and the implications of this on the package management system is distribution or language dependent.
- The coalescer must deal with distribution upgrades in a sane way. If your distribution or language does a dist-upgrade, are all packages bumped? Are they simply left alone? The coalescer must understand what happens and compute the final manifest's content correctly.
- Packages may be removed and added between layers.
- When the package database is a regular file on disk, this case is simpler: the database file found in the most recent layer holds the ultimate set of packages for all previous layers. However, in the case where the package database is realized by adding and removing files on disk it becomes trickier. Claircore has no special handling of whiteout files, currently. We will address this in upcoming releases.
If your distribution or language cannot utilize a default coalescer, you will need to implement the Coalescer interface
Implementing or Adding To An Ecosystem
An Ecosystem provides a set of coalescers, package scanners, distribution scanners, and repository scanners to Libindex at the time of indexing.
Libindex will take the Ecosystem and scan each layer with all provided scanners. When Libindex is ready to coalesce the results of each scanner into an IndexReport the provided coalescer is given the output of the configured scanners.
This allows Libindex to segment the input to the coalescing step to particular scanners that a coalescer understands.
For instance, if we only wanted a (fictitious) Haskell coalescer to evaluate artifacts returned from a (fictitious) Haskell package and repository scanner we would create an ecosystem similar to:
// HaskellScanner returns a configured PackageScanner.
func haskellScanner() indexer.PackageScanner { return nil }
// HaskellCoalescer returns a configured Coalescer.
func haskellCoalescer() indexer.Coalescer { return nil }
// NewEcosystem provides the set of scanners and coalescers for the haskell
// ecosystem.
func NewEcosystem(ctx context.Context) *indexer.Ecosystem {
return &indexer.Ecosystem{
PackageScanners: func(ctx context.Context) ([]indexer.PackageScanner, error) {
return []indexer.PackageScanner{haskellScanner()}, nil
},
DistributionScanners: func(ctx context.Context) ([]indexer.DistributionScanner, error) {
return []indexer.DistributionScanner{}, nil
},
RepositoryScanners: func(ctx context.Context) ([]indexer.RepositoryScanner, error) {
return []indexer.RepositoryScanner{}, nil
},
Coalescer: func(ctx context.Context) (indexer.Coalescer, error) {
return haskellCoalescer(), nil
},
}
}
This ensures that Libindex will only provide Haskell artifacts to the Haskell coalescer and avoid calling the coalescer with rpm packages for example.
If your distribution uses an already implemented package manager such as "rpm" or "dpkg", it's likely you will simply add your scanners to the existing ecosystem in one of those packages.
Alternative Implementations
This how-to guide is a "perfect world" scenario.
Working on claircore has made us realize that this domain is a bit messy. Security trackers are not developed with package managers in mind, security databases do not follow correct specs, distribution maintainers spin their own tools, etc.
We understand that supporting your distribution or language may take some bending of claircore's architecture and business logic. If this is the case, start a conversation with us. We are open to design discussions.
Getting Help
At this point, you have implemented all the necessary components to integrate your distribution or language with claircore.
If you struggle with the design phase or are getting stuck at the implementation phases do not hesitate to reach out to us. Here are some links:
Reference
The follow sections provide a reference for developing with Claircore. Important interfaces and structs are outlined.
- Coalescer
- Configurable Scanner
- Distribution Scanner
- Ecosystem
- Libindex Store
- Matcher
- Package Scanner
- Remote Scanner
- Repository Scanner
- RPC Scanner
- Updater Set Factory
- Version Filter
- Versioned Scanner
- Resolver
Coalescer
A coalescer must compute the final contents of a manifest given the artifacts found at each layer.
package indexer // import "github.com/quay/claircore/indexer"
type Coalescer interface {
Coalesce(ctx context.Context, artifacts []*LayerArtifacts) (*claircore.IndexReport, error)
}
Coalescer takes a set of layers and creates coalesced IndexReport.
A coalesced IndexReport should provide only the packages present in the
final container image once all layers were applied.
package indexer // import "github.com/quay/claircore/indexer"
type Coalescer interface {
Coalesce(ctx context.Context, artifacts []*LayerArtifacts) (*claircore.IndexReport, error)
}
Coalescer takes a set of layers and creates coalesced IndexReport.
A coalesced IndexReport should provide only the packages present in the
final container image once all layers were applied.
package indexer // import "github.com/quay/claircore/indexer"
type LayerArtifacts struct { Hash claircore.Digest Pkgs []*claircore.Package Dist []*claircore.Distribution // each layer can only have a single distribution Repos []*claircore.Repository Files []claircore.File } LayerArifact aggregates the artifacts found within a layer.
A `Coalsecer` implementation is free to determine this computation given the
artifacts found in a layer. A `Coalescer` is called with a slice of
`LayerArtifacts` structs. The manifest's layer ordering is preserved in the
provided slice.
ConfigurableScanner
A ConfigurableSanner
is an optional interface a Scanner
interface may
implement. When implemented, the scanner's Configure
method will be called
with a ConfigDeserializer
function. The Scanner
may pass its config as an
argument to the ConfigDeserializer
function to populate the struct.
package indexer // import "github.com/quay/claircore/indexer"
type ConfigDeserializer func(interface{}) error
ConfigDeserializer can be thought of as an Unmarshal function with the byte
slice provided.
This will typically be something like (*json.Decoder).Decode.
package indexer // import "github.com/quay/claircore/indexer"
type ConfigDeserializer func(interface{}) error
ConfigDeserializer can be thought of as an Unmarshal function with the byte
slice provided.
This will typically be something like (*json.Decoder).Decode.
package indexer // import "github.com/quay/claircore/indexer"
type ConfigurableScanner interface { Configure(context.Context, ConfigDeserializer) error } ConfigurableScanner is an interface scanners can implement to receive configuration.
Distribution Scanner
A Distribution Scanner should identify any operating system distribution associated with the provided layer. It is OK for no distribution information to be discovered.
package indexer // import "github.com/quay/claircore/indexer"
type DistributionScanner interface {
VersionedScanner
Scan(context.Context, *claircore.Layer) ([]*claircore.Distribution, error)
}
DistributionScanner reports the Distributions found in a given layer.
Ecosystem
An Ecosystem groups together scanners and coalescers which are often used
together. Ecosystems are usually defined in a go package that corresponds to a
package manager, such as dpkg
. See dpkg/ecosystem.go
for an example.
The Indexer will retrieve artifacts from the provided scanners and provide these scan artifacts to the coalescer in the Ecosystem.
package indexer // import "github.com/quay/claircore/indexer"
type Ecosystem struct {
PackageScanners func(ctx context.Context) ([]PackageScanner, error)
DistributionScanners func(ctx context.Context) ([]DistributionScanner, error)
RepositoryScanners func(ctx context.Context) ([]RepositoryScanner, error)
FileScanners func(ctx context.Context) ([]FileScanner, error)
Coalescer func(ctx context.Context) (Coalescer, error)
Name string
}
Ecosystems group together scanners and a Coalescer which are commonly used
together.
A typical ecosystem is "dpkg" which will use the "dpkg" package indexer,
the "os-release" distribution scanner and the "apt" repository scanner.
A Controller will scan layers with all scanners present in its configured
ecosystems.
Index Report
An Index Report defines the contents of a manifest. An Index Report can be unpacked into a slice of Index Records, each of which defines a package, distribution, repository tuple.
package claircore // import "github.com/quay/claircore"
type IndexReport struct {
// the manifest hash this IndexReport is describing
Hash Digest `json:"manifest_hash"`
// the current state of the index operation
State string `json:"state"`
// all discovered packages in this manifest key'd by package id
Packages map[string]*Package `json:"packages"`
// all discovered distributions in this manifest key'd by distribution id
Distributions map[string]*Distribution `json:"distributions"`
// all discovered repositories in this manifest key'd by repository id
Repositories map[string]*Repository `json:"repository"`
// a list of environment details a package was discovered in key'd by package id
Environments map[string][]*Environment `json:"environments"`
// whether the index operation finished successfully
Success bool `json:"success"`
// an error string in the case the index did not succeed
Err string `json:"err"`
// Files doesn't end up in the json report but needs to be available at post-coalesce
Files map[string]File `json:"-"`
}
IndexReport provides a database for discovered artifacts in an image.
IndexReports make heavy usage of lookup maps to associate information
without repetition.
func (report *IndexReport) IndexRecords() []*IndexRecord
package claircore // import "github.com/quay/claircore"
type IndexReport struct {
// the manifest hash this IndexReport is describing
Hash Digest `json:"manifest_hash"`
// the current state of the index operation
State string `json:"state"`
// all discovered packages in this manifest key'd by package id
Packages map[string]*Package `json:"packages"`
// all discovered distributions in this manifest key'd by distribution id
Distributions map[string]*Distribution `json:"distributions"`
// all discovered repositories in this manifest key'd by repository id
Repositories map[string]*Repository `json:"repository"`
// a list of environment details a package was discovered in key'd by package id
Environments map[string][]*Environment `json:"environments"`
// whether the index operation finished successfully
Success bool `json:"success"`
// an error string in the case the index did not succeed
Err string `json:"err"`
// Files doesn't end up in the json report but needs to be available at post-coalesce
Files map[string]File `json:"-"`
}
IndexReport provides a database for discovered artifacts in an image.
IndexReports make heavy usage of lookup maps to associate information
without repetition.
func (report *IndexReport) IndexRecords() []*IndexRecord
package claircore // import "github.com/quay/claircore"
type IndexRecord struct { Package *Package Distribution *Distribution Repository *Repository } IndexRecord is an entry in the IndexReport.
IndexRecords provide full access to contextual package structures such as
Distribution and Repository.
A list of these can be thought of as an "unpacked" IndexReport
Indexer Store
The indexer.Store
interface defines all necessary persistence methods for
Libindex
to provide its functionality.
package indexer // import "github.com/quay/claircore/indexer"
type Store interface {
Setter
Querier
Indexer
// Close frees any resources associated with the Store.
Close(context.Context) error
}
Store is an interface for dealing with objects libindex needs to persist.
Stores may be implemented per storage backend.
package indexer // import "github.com/quay/claircore/indexer"
type Store interface {
Setter
Querier
Indexer
// Close frees any resources associated with the Store.
Close(context.Context) error
}
Store is an interface for dealing with objects libindex needs to persist.
Stores may be implemented per storage backend.
package indexer // import "github.com/quay/claircore/indexer"
type Setter interface { // PersistManifest must store the presence of a manifest and it's layers into the system. // // Typically this will write into identity tables so later methods have a foreign key // to reference and data integrity is applied. PersistManifest(ctx context.Context, manifest claircore.Manifest) error // DeleteManifests removes the manifests indicated by the passed digests // from the backing store. DeleteManifests(context.Context, ...claircore.Digest) ([]claircore.Digest, error)
// SetLayerScanned marks the provided layer hash successfully scanned by the provided versioned scanner.
//
// After this method is returned a call to Querier.LayerScanned with the same arguments must return true.
SetLayerScanned(ctx context.Context, hash claircore.Digest, scnr VersionedScanner) error
// RegisterPackageScanners registers the provided scanners with the persistence layer.
RegisterScanners(ctx context.Context, scnrs VersionedScanners) error
// SetIndexReport persists the current state of the IndexReport.
//
// IndexReports maybe in intermediate states to provide feedback for clients. this method should be
// used to communicate scanning state updates. to signal the scan has completely successfully
// see SetIndexFinished.
SetIndexReport(context.Context, *claircore.IndexReport) error
// SetIndexFinished marks a scan successfully completed.
//
// After this method returns a call to Querier.ManifestScanned with the manifest hash represted in the provided IndexReport
// must return true.
//
// Also a call to Querier.IndexReport with the manifest hash represted in the provided IndexReport must return the IndexReport
// in finished state.
SetIndexFinished(ctx context.Context, sr *claircore.IndexReport, scnrs VersionedScanners) error
} Setter interface provides the method set for required marking events, or registering components, associated with an Index operation.
package indexer // import "github.com/quay/claircore/indexer"
type Store interface { Setter Querier Indexer // Close frees any resources associated with the Store. Close(context.Context) error } Store is an interface for dealing with objects libindex needs to persist. Stores may be implemented per storage backend.
package indexer // import "github.com/quay/claircore/indexer"
type Setter interface {
// PersistManifest must store the presence of a manifest and it's layers into the system.
//
// Typically this will write into identity tables so later methods have a foreign key
// to reference and data integrity is applied.
PersistManifest(ctx context.Context, manifest claircore.Manifest) error
// DeleteManifests removes the manifests indicated by the passed digests
// from the backing store.
DeleteManifests(context.Context, ...claircore.Digest) ([]claircore.Digest, error)
// SetLayerScanned marks the provided layer hash successfully scanned by the provided versioned scanner.
//
// After this method is returned a call to Querier.LayerScanned with the same arguments must return true.
SetLayerScanned(ctx context.Context, hash claircore.Digest, scnr VersionedScanner) error
// RegisterPackageScanners registers the provided scanners with the persistence layer.
RegisterScanners(ctx context.Context, scnrs VersionedScanners) error
// SetIndexReport persists the current state of the IndexReport.
//
// IndexReports maybe in intermediate states to provide feedback for clients. this method should be
// used to communicate scanning state updates. to signal the scan has completely successfully
// see SetIndexFinished.
SetIndexReport(context.Context, *claircore.IndexReport) error
// SetIndexFinished marks a scan successfully completed.
//
// After this method returns a call to Querier.ManifestScanned with the manifest hash represted in the provided IndexReport
// must return true.
//
// Also a call to Querier.IndexReport with the manifest hash represted in the provided IndexReport must return the IndexReport
// in finished state.
SetIndexFinished(ctx context.Context, sr *claircore.IndexReport, scnrs VersionedScanners) error
}
Setter interface provides the method set for required marking events,
or registering components, associated with an Index operation.
package indexer // import "github.com/quay/claircore/indexer"
type Querier interface { // ManifestScanned returns whether the given manifest was scanned by the provided scanners. ManifestScanned(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) (bool, error) // LayerScanned returns whether the given layer was scanned by the provided scanner. LayerScanned(ctx context.Context, hash claircore.Digest, scnr VersionedScanner) (bool, error) // PackagesByLayer gets all the packages found in a layer limited by the provided scanners. PackagesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Package, error) // DistributionsByLayer gets all the distributions found in a layer limited by the provided scanners. DistributionsByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Distribution, error) // RepositoriesByLayer gets all the repositories found in a layer limited by the provided scanners. RepositoriesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Repository, error) // FilesByLayer gets all the interesting files found in a layer limited by the provided scanners. FilesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]claircore.File, error) // IndexReport attempts to retrieve a persisted IndexReport. IndexReport(ctx context.Context, hash claircore.Digest) (*claircore.IndexReport, bool, error) // AffectedManifests returns a list of manifest digests which the target vulnerability // affects. AffectedManifests(ctx context.Context, v claircore.Vulnerability, f claircore.CheckVulnernableFunc) ([]claircore.Digest, error) } Querier interface provides the method set to ascertain indexed artifacts and query whether a layer or manifest has been scanned.
package indexer // import "github.com/quay/claircore/indexer"
type Store interface { Setter Querier Indexer // Close frees any resources associated with the Store. Close(context.Context) error } Store is an interface for dealing with objects libindex needs to persist. Stores may be implemented per storage backend.
package indexer // import "github.com/quay/claircore/indexer"
type Setter interface {
// PersistManifest must store the presence of a manifest and it's layers into the system.
//
// Typically this will write into identity tables so later methods have a foreign key
// to reference and data integrity is applied.
PersistManifest(ctx context.Context, manifest claircore.Manifest) error
// DeleteManifests removes the manifests indicated by the passed digests
// from the backing store.
DeleteManifests(context.Context, ...claircore.Digest) ([]claircore.Digest, error)
// SetLayerScanned marks the provided layer hash successfully scanned by the provided versioned scanner.
//
// After this method is returned a call to Querier.LayerScanned with the same arguments must return true.
SetLayerScanned(ctx context.Context, hash claircore.Digest, scnr VersionedScanner) error
// RegisterPackageScanners registers the provided scanners with the persistence layer.
RegisterScanners(ctx context.Context, scnrs VersionedScanners) error
// SetIndexReport persists the current state of the IndexReport.
//
// IndexReports maybe in intermediate states to provide feedback for clients. this method should be
// used to communicate scanning state updates. to signal the scan has completely successfully
// see SetIndexFinished.
SetIndexReport(context.Context, *claircore.IndexReport) error
// SetIndexFinished marks a scan successfully completed.
//
// After this method returns a call to Querier.ManifestScanned with the manifest hash represted in the provided IndexReport
// must return true.
//
// Also a call to Querier.IndexReport with the manifest hash represted in the provided IndexReport must return the IndexReport
// in finished state.
SetIndexFinished(ctx context.Context, sr *claircore.IndexReport, scnrs VersionedScanners) error
}
Setter interface provides the method set for required marking events,
or registering components, associated with an Index operation.
package indexer // import "github.com/quay/claircore/indexer"
type Querier interface { // ManifestScanned returns whether the given manifest was scanned by the provided scanners. ManifestScanned(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) (bool, error) // LayerScanned returns whether the given layer was scanned by the provided scanner. LayerScanned(ctx context.Context, hash claircore.Digest, scnr VersionedScanner) (bool, error) // PackagesByLayer gets all the packages found in a layer limited by the provided scanners. PackagesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Package, error) // DistributionsByLayer gets all the distributions found in a layer limited by the provided scanners. DistributionsByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Distribution, error) // RepositoriesByLayer gets all the repositories found in a layer limited by the provided scanners. RepositoriesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Repository, error) // FilesByLayer gets all the interesting files found in a layer limited by the provided scanners. FilesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]claircore.File, error) // IndexReport attempts to retrieve a persisted IndexReport. IndexReport(ctx context.Context, hash claircore.Digest) (*claircore.IndexReport, bool, error) // AffectedManifests returns a list of manifest digests which the target vulnerability // affects. AffectedManifests(ctx context.Context, v claircore.Vulnerability, f claircore.CheckVulnernableFunc) ([]claircore.Digest, error) } Querier interface provides the method set to ascertain indexed artifacts and query whether a layer or manifest has been scanned.
package indexer // import "github.com/quay/claircore/indexer"
type Indexer interface {
// IndexPackages indexes a package into the persistence layer.
IndexPackages(ctx context.Context, pkgs []*claircore.Package, layer *claircore.Layer, scnr VersionedScanner) error
// IndexDistributions indexes distributions into the persistence layer.
IndexDistributions(ctx context.Context, dists []*claircore.Distribution, layer *claircore.Layer, scnr VersionedScanner) error
// IndexRepositories indexes repositories into the persistence layer.
IndexRepositories(ctx context.Context, repos []*claircore.Repository, layer *claircore.Layer, scnr VersionedScanner) error
// IndexFiles indexes the interesting files into the persistence layer.
IndexFiles(ctx context.Context, files []claircore.File, layer *claircore.Layer, scnr VersionedScanner) error
// IndexManifest should index the coalesced manifest's content given an IndexReport.
IndexManifest(ctx context.Context, ir *claircore.IndexReport) error
}
Indexer interface provide the method set required for indexing layer and
manifest contents into a persistent store.
```
Matcher Store
The datastore.MatcherStore
interface defines all necessary persistence methods for Libvuln
to provide its functionality.
package datastore // import "github.com/quay/claircore/datastore"
type MatcherStore interface {
Updater
Vulnerability
Enrichment
}
MatcherStore aggregates all interface types
package datastore // import "github.com/quay/claircore/datastore"
type MatcherStore interface {
Updater
Vulnerability
Enrichment
}
MatcherStore aggregates all interface types
package datastore // import "github.com/quay/claircore/datastore"
type Updater interface { EnrichmentUpdater
// UpdateVulnerabilities creates a new UpdateOperation, inserts the provided
// vulnerabilities, and ensures vulnerabilities from previous updates are
// not queried by clients.
UpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
// UpdateVulnerabilitiesIter performs the same operation as
// UpdateVulnerabilities, but accepting an iterator function.
UpdateVulnerabilitiesIter(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulnIter VulnerabilityIter) (uuid.UUID, error)
// DeltaUpdateVulnerabilities creates a new UpdateOperation consisting of existing
// vulnerabilities and new vulnerabilities. It also takes an array of deleted
// vulnerability names which should no longer be available to query.
DeltaUpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability, deletedVulns []string) (uuid.UUID, error)
// GetUpdateOperations returns a list of UpdateOperations in date descending
// order for the given updaters.
//
// The returned map is keyed by Updater implementation's unique names.
//
// If no updaters are specified, all UpdateOperations are returned.
GetUpdateOperations(context.Context, driver.UpdateKind, ...string) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRefs reports the latest update reference for every known
// updater.
GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRef reports the latest update reference of any known
// updater.
GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
// DeleteUpdateOperations removes an UpdateOperation.
// A call to GC must be run after this to garbage collect vulnerabilities associated
// with the UpdateOperation.
//
// The number of UpdateOperations deleted is returned.
DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
// Operations.
//
// In diff(1) terms, this is like
//
// diff prev cur
//
GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID) (*driver.UpdateDiff, error)
// GC will delete any update operations for an updater which exceeds the provided keep
// value.
//
// Implementations may throttle the GC process for datastore efficiency reasons.
//
// The returned int64 value indicates the remaining number of update operations needing GC.
// Running this method till the returned value is 0 accomplishes a full GC of the vulnstore.
GC(ctx context.Context, keep int) (int64, error)
// Initialized reports whether the vulnstore contains vulnerabilities.
Initialized(context.Context) (bool, error)
// RecordUpdaterStatus records that an updater is up to date with vulnerabilities at this time
RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime time.Time, fingerprint driver.Fingerprint, updaterError error) error
// RecordUpdaterSetStatus records that all updaters from an updater set are up to date with vulnerabilities at this time
RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime time.Time) error
} Updater is an interface exporting the necessary methods for updating a vulnerability database.
package datastore // import "github.com/quay/claircore/datastore"
type MatcherStore interface { Updater Vulnerability Enrichment } MatcherStore aggregates all interface types
package datastore // import "github.com/quay/claircore/datastore"
type Updater interface {
EnrichmentUpdater
// UpdateVulnerabilities creates a new UpdateOperation, inserts the provided
// vulnerabilities, and ensures vulnerabilities from previous updates are
// not queried by clients.
UpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
// UpdateVulnerabilitiesIter performs the same operation as
// UpdateVulnerabilities, but accepting an iterator function.
UpdateVulnerabilitiesIter(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulnIter VulnerabilityIter) (uuid.UUID, error)
// DeltaUpdateVulnerabilities creates a new UpdateOperation consisting of existing
// vulnerabilities and new vulnerabilities. It also takes an array of deleted
// vulnerability names which should no longer be available to query.
DeltaUpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability, deletedVulns []string) (uuid.UUID, error)
// GetUpdateOperations returns a list of UpdateOperations in date descending
// order for the given updaters.
//
// The returned map is keyed by Updater implementation's unique names.
//
// If no updaters are specified, all UpdateOperations are returned.
GetUpdateOperations(context.Context, driver.UpdateKind, ...string) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRefs reports the latest update reference for every known
// updater.
GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRef reports the latest update reference of any known
// updater.
GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
// DeleteUpdateOperations removes an UpdateOperation.
// A call to GC must be run after this to garbage collect vulnerabilities associated
// with the UpdateOperation.
//
// The number of UpdateOperations deleted is returned.
DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
// Operations.
//
// In diff(1) terms, this is like
//
// diff prev cur
//
GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID) (*driver.UpdateDiff, error)
// GC will delete any update operations for an updater which exceeds the provided keep
// value.
//
// Implementations may throttle the GC process for datastore efficiency reasons.
//
// The returned int64 value indicates the remaining number of update operations needing GC.
// Running this method till the returned value is 0 accomplishes a full GC of the vulnstore.
GC(ctx context.Context, keep int) (int64, error)
// Initialized reports whether the vulnstore contains vulnerabilities.
Initialized(context.Context) (bool, error)
// RecordUpdaterStatus records that an updater is up to date with vulnerabilities at this time
RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime time.Time, fingerprint driver.Fingerprint, updaterError error) error
// RecordUpdaterSetStatus records that all updaters from an updater set are up to date with vulnerabilities at this time
RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime time.Time) error
}
Updater is an interface exporting the necessary methods for updating a
vulnerability database.
package datastore // import "github.com/quay/claircore/datastore"
type EnrichmentUpdater interface { // UpdateEnrichments creates a new EnrichmentUpdateOperation, inserts the provided // EnrichmentRecord(s), and ensures enrichments from previous updates are not // queries by clients. UpdateEnrichments(ctx context.Context, kind string, fingerprint driver.Fingerprint, enrichments []driver.EnrichmentRecord) (uuid.UUID, error) // UpdateEnrichmentsIter performs the same operation as UpdateEnrichments, but // accepting an iterator function. UpdateEnrichmentsIter(ctx context.Context, kind string, fingerprint driver.Fingerprint, enIter EnrichmentIter) (uuid.UUID, error) } EnrichmentUpdater is an interface exporting the necessary methods for storing and querying Enrichments.
package datastore // import "github.com/quay/claircore/datastore"
type MatcherStore interface { Updater Vulnerability Enrichment } MatcherStore aggregates all interface types
package datastore // import "github.com/quay/claircore/datastore"
type Updater interface {
EnrichmentUpdater
// UpdateVulnerabilities creates a new UpdateOperation, inserts the provided
// vulnerabilities, and ensures vulnerabilities from previous updates are
// not queried by clients.
UpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
// UpdateVulnerabilitiesIter performs the same operation as
// UpdateVulnerabilities, but accepting an iterator function.
UpdateVulnerabilitiesIter(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulnIter VulnerabilityIter) (uuid.UUID, error)
// DeltaUpdateVulnerabilities creates a new UpdateOperation consisting of existing
// vulnerabilities and new vulnerabilities. It also takes an array of deleted
// vulnerability names which should no longer be available to query.
DeltaUpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability, deletedVulns []string) (uuid.UUID, error)
// GetUpdateOperations returns a list of UpdateOperations in date descending
// order for the given updaters.
//
// The returned map is keyed by Updater implementation's unique names.
//
// If no updaters are specified, all UpdateOperations are returned.
GetUpdateOperations(context.Context, driver.UpdateKind, ...string) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRefs reports the latest update reference for every known
// updater.
GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRef reports the latest update reference of any known
// updater.
GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
// DeleteUpdateOperations removes an UpdateOperation.
// A call to GC must be run after this to garbage collect vulnerabilities associated
// with the UpdateOperation.
//
// The number of UpdateOperations deleted is returned.
DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
// Operations.
//
// In diff(1) terms, this is like
//
// diff prev cur
//
GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID) (*driver.UpdateDiff, error)
// GC will delete any update operations for an updater which exceeds the provided keep
// value.
//
// Implementations may throttle the GC process for datastore efficiency reasons.
//
// The returned int64 value indicates the remaining number of update operations needing GC.
// Running this method till the returned value is 0 accomplishes a full GC of the vulnstore.
GC(ctx context.Context, keep int) (int64, error)
// Initialized reports whether the vulnstore contains vulnerabilities.
Initialized(context.Context) (bool, error)
// RecordUpdaterStatus records that an updater is up to date with vulnerabilities at this time
RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime time.Time, fingerprint driver.Fingerprint, updaterError error) error
// RecordUpdaterSetStatus records that all updaters from an updater set are up to date with vulnerabilities at this time
RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime time.Time) error
}
Updater is an interface exporting the necessary methods for updating a
vulnerability database.
package datastore // import "github.com/quay/claircore/datastore"
type EnrichmentUpdater interface { // UpdateEnrichments creates a new EnrichmentUpdateOperation, inserts the provided // EnrichmentRecord(s), and ensures enrichments from previous updates are not // queries by clients. UpdateEnrichments(ctx context.Context, kind string, fingerprint driver.Fingerprint, enrichments []driver.EnrichmentRecord) (uuid.UUID, error) // UpdateEnrichmentsIter performs the same operation as UpdateEnrichments, but // accepting an iterator function. UpdateEnrichmentsIter(ctx context.Context, kind string, fingerprint driver.Fingerprint, enIter EnrichmentIter) (uuid.UUID, error) } EnrichmentUpdater is an interface exporting the necessary methods for storing and querying Enrichments.
package datastore // import "github.com/quay/claircore/datastore"
type Vulnerability interface {
// Get finds the vulnerabilities which match each package provided in the
// [IndexRecord]s. This may be a one-to-many relationship. A map of Package
// ID to Vulnerabilities is returned.
Get(ctx context.Context, records []*claircore.IndexRecord, opts GetOpts) (map[string][]*claircore.Vulnerability, error)
}
Vulnerability is the interface for querying stored Vulnerabilities.
```
```
package datastore // import "github.com/quay/claircore/datastore"
type MatcherStore interface {
Updater
Vulnerability
Enrichment
}
MatcherStore aggregates all interface types
package datastore // import "github.com/quay/claircore/datastore"
type Updater interface { EnrichmentUpdater
// UpdateVulnerabilities creates a new UpdateOperation, inserts the provided
// vulnerabilities, and ensures vulnerabilities from previous updates are
// not queried by clients.
UpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
// UpdateVulnerabilitiesIter performs the same operation as
// UpdateVulnerabilities, but accepting an iterator function.
UpdateVulnerabilitiesIter(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulnIter VulnerabilityIter) (uuid.UUID, error)
// DeltaUpdateVulnerabilities creates a new UpdateOperation consisting of existing
// vulnerabilities and new vulnerabilities. It also takes an array of deleted
// vulnerability names which should no longer be available to query.
DeltaUpdateVulnerabilities(ctx context.Context, updater string, fingerprint driver.Fingerprint, vulns []*claircore.Vulnerability, deletedVulns []string) (uuid.UUID, error)
// GetUpdateOperations returns a list of UpdateOperations in date descending
// order for the given updaters.
//
// The returned map is keyed by Updater implementation's unique names.
//
// If no updaters are specified, all UpdateOperations are returned.
GetUpdateOperations(context.Context, driver.UpdateKind, ...string) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRefs reports the latest update reference for every known
// updater.
GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRef reports the latest update reference of any known
// updater.
GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
// DeleteUpdateOperations removes an UpdateOperation.
// A call to GC must be run after this to garbage collect vulnerabilities associated
// with the UpdateOperation.
//
// The number of UpdateOperations deleted is returned.
DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
// Operations.
//
// In diff(1) terms, this is like
//
// diff prev cur
//
GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID) (*driver.UpdateDiff, error)
// GC will delete any update operations for an updater which exceeds the provided keep
// value.
//
// Implementations may throttle the GC process for datastore efficiency reasons.
//
// The returned int64 value indicates the remaining number of update operations needing GC.
// Running this method till the returned value is 0 accomplishes a full GC of the vulnstore.
GC(ctx context.Context, keep int) (int64, error)
// Initialized reports whether the vulnstore contains vulnerabilities.
Initialized(context.Context) (bool, error)
// RecordUpdaterStatus records that an updater is up to date with vulnerabilities at this time
RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime time.Time, fingerprint driver.Fingerprint, updaterError error) error
// RecordUpdaterSetStatus records that all updaters from an updater set are up to date with vulnerabilities at this time
RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime time.Time) error
} Updater is an interface exporting the necessary methods for updating a vulnerability database.
package datastore // import "github.com/quay/claircore/datastore"
type EnrichmentUpdater interface {
// UpdateEnrichments creates a new EnrichmentUpdateOperation, inserts the provided
// EnrichmentRecord(s), and ensures enrichments from previous updates are not
// queries by clients.
UpdateEnrichments(ctx context.Context, kind string, fingerprint driver.Fingerprint, enrichments []driver.EnrichmentRecord) (uuid.UUID, error)
// UpdateEnrichmentsIter performs the same operation as UpdateEnrichments, but
// accepting an iterator function.
UpdateEnrichmentsIter(ctx context.Context, kind string, fingerprint driver.Fingerprint, enIter EnrichmentIter) (uuid.UUID, error)
}
EnrichmentUpdater is an interface exporting the necessary methods for
storing and querying Enrichments.
package datastore // import "github.com/quay/claircore/datastore"
type Vulnerability interface { // Get finds the vulnerabilities which match each package provided in the // [IndexRecord]s. This may be a one-to-many relationship. A map of Package // ID to Vulnerabilities is returned. Get(ctx context.Context, records []*claircore.IndexRecord, opts GetOpts) (map[string][]*claircore.Vulnerability, error) } Vulnerability is the interface for querying stored Vulnerabilities.
package datastore // import "github.com/quay/claircore/datastore"
type Enrichment interface {
GetEnrichment(ctx context.Context, kind string, tags []string) ([]driver.EnrichmentRecord, error)
}
Enrichment is an interface for querying enrichments from the store.
```
Manifest
A Manifest is analogous to an OCI Image Manifest: it defines the order of layers and how to retrieve the them.
package claircore // import "github.com/quay/claircore"
type Manifest struct {
// content addressable hash. should be able to be computed via
// the hashes of all included layers
Hash Digest `json:"hash"`
// an array of filesystem layers indexed in the same order as the corresponding image
Layers []*Layer `json:"layers"`
}
Manifest represents a docker image. Layers array MUST be indexed in the
order that image layers are stacked.
Matcher
A Matcher performs the heavy lifting of matching manifest contents to relevant vulnerabilities. These implementations provide the smarts for understanding if a particular artifact in a layer is vulnerable to a particular advisory in the database.
package driver // import "github.com/quay/claircore/libvuln/driver"
type Matcher interface {
// a unique name for the matcher
Name() string
// Filter informs the Controller if the implemented Matcher is interested in the provided IndexRecord.
Filter(record *claircore.IndexRecord) bool
// Query informs the Controller how it should match packages with vulnerabilities.
// All conditions are logical AND'd together.
Query() []MatchConstraint
// Vulnerable informs the Controller if the given package is affected by the given vulnerability.
// for example checking the "FixedInVersion" field.
Vulnerable(ctx context.Context, record *claircore.IndexRecord, vuln *claircore.Vulnerability) (bool, error)
}
Matcher is an interface which a Controller uses to query the vulnstore for
vulnerabilities.
The Filter
method is used to inform Libvuln
the provided artifact is
interesting.
The Query
method tells Libvuln
how to query the security advisory database.
The Vulnerable
method reports whether the provided package is vulnerable to
the provided vulnerability. Typically, this would perform a version check
between the artifact and the vulnerability in question.
Package Scanner
A Package Scanner should discover any packages found within the given layer. It is OK for to discover no packages within a layer.
package indexer // import "github.com/quay/claircore/indexer"
type PackageScanner interface {
VersionedScanner
// Scan performs a package scan on the given layer and returns all
// the found packages
Scan(context.Context, *claircore.Layer) ([]*claircore.Package, error)
}
PackageScanner provides an interface for unique identification or a
PackageScanner and a Scan method for extracting installed packages from an
individual container layer
func NewPackageScannerMock(name, version, kind string) PackageScanner
RemoteMatcher
RemoteMatcher is an additional interface a Matcher may implement to skip the database for matching results and use an external API.
package driver // import "github.com/quay/claircore/libvuln/driver"
type RemoteMatcher interface {
QueryRemoteMatcher(ctx context.Context, records []*claircore.IndexRecord) (map[string][]*claircore.Vulnerability, error)
}
RemoteMatcher is an additional interface that a Matcher can implement.
When called the interface can invoke the remote matcher using an HTTP API to
fetch new vulnerabilities associated with the given IndexRecords.
The information retrieved from this interface won't be persisted into the
claircore database.
Repository Scanner
A RepositoryScanner should identify any repositories discovered in the provided layer. It is OK for the scanner to identify no repositories.
package indexer // import "github.com/quay/claircore/indexer"
type RepositoryScanner interface {
VersionedScanner
Scan(context.Context, *claircore.Layer) ([]*claircore.Repository, error)
}
Resolver
A Resolver is used to analyze and modify the post-coalesced index report. This is useful for operations that need all context from an index report.
package indexer // import "github.com/quay/claircore/indexer"
type Resolver interface {
Resolve(context.Context, *claircore.IndexReport, []*claircore.Layer) *claircore.IndexReport
}
Resolver is used for any reasoning that needs to be done with all the layers
in context.
Resolvers are called at the end of the coalesce step when reports from
separate scanners are merged.
Any Resolvers' Resolve()
methods are called (in no set order) at the end of the coalesce step after reports from separate scanners are merged.
RPCScanner
RPCScanner
is an optional interface a Scanner
may implement.
When implemented, the scanner's Configure
method will be called with a
ConfigDeserializer
function and an HTTP client.
The Scanner
may pass its config as an argument to the ConfigDeserializer
function to populate the struct and use the HTTP client for any remote access
necessary during the scanning process.
package indexer // import "github.com/quay/claircore/indexer"
type RPCScanner interface {
Configure(context.Context, ConfigDeserializer, *http.Client) error
}
RPCScanner is an interface scanners can implement to receive configuration
and denote that they expect to be able to talk to the network at run time.
package indexer // import "github.com/quay/claircore/indexer"
type RPCScanner interface {
Configure(context.Context, ConfigDeserializer, *http.Client) error
}
RPCScanner is an interface scanners can implement to receive configuration
and denote that they expect to be able to talk to the network at run time.
package indexer // import "github.com/quay/claircore/indexer"
type ConfigDeserializer func(interface{}) error ConfigDeserializer can be thought of as an Unmarshal function with the byte slice provided.
This will typically be something like (*json.Decoder).Decode.
Updater
An Updater is responsible for performing run-time fetching and parsing of a security database. The returned vulnerabilities will be written to claircore's database and used in vulnerability matching.
package driver // import "github.com/quay/claircore/libvuln/driver"
type Updater interface {
Name() string
Fetcher
Parser
}
Updater is an aggregate interface combining the method set of a Fetcher and
a Parser and forces a Name() to be provided
package driver // import "github.com/quay/claircore/libvuln/driver"
type Updater interface {
Name() string
Fetcher
Parser
}
Updater is an aggregate interface combining the method set of a Fetcher and
a Parser and forces a Name() to be provided
package driver // import "github.com/quay/claircore/libvuln/driver"
type Fetcher interface { Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error) } Fetcher is an interface which is embedded into the Updater interface.
When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed
If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.
If the conent has not change an Unchanged error should be returned.
package driver // import "github.com/quay/claircore/libvuln/driver"
type Updater interface { Name() string Fetcher Parser } Updater is an aggregate interface combining the method set of a Fetcher and a Parser and forces a Name() to be provided
package driver // import "github.com/quay/claircore/libvuln/driver"
type Fetcher interface {
Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error)
}
Fetcher is an interface which is embedded into the Updater interface.
When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed
If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.
If the conent has not change an Unchanged error should be returned.
package driver // import "github.com/quay/claircore/libvuln/driver"
type Parser interface { // Parse should take an io.ReadCloser, read the contents, parse the contents // into a list of claircore.Vulnerability structs and then return // the list. Parse should assume contents are uncompressed and ready for parsing. Parse(ctx context.Context, contents io.ReadCloser) ([]*claircore.Vulnerability, error) } Parser is an interface which is embedded into the Updater interface.
Parse should be called with an io.ReadCloser struct where the contents
of a security advisory database can be read and parsed into an array of
*claircore.Vulnerability
package driver // import "github.com/quay/claircore/libvuln/driver"
type Updater interface { Name() string Fetcher Parser } Updater is an aggregate interface combining the method set of a Fetcher and a Parser and forces a Name() to be provided
package driver // import "github.com/quay/claircore/libvuln/driver"
type Fetcher interface {
Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error)
}
Fetcher is an interface which is embedded into the Updater interface.
When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed
If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.
If the conent has not change an Unchanged error should be returned.
package driver // import "github.com/quay/claircore/libvuln/driver"
type Parser interface { // Parse should take an io.ReadCloser, read the contents, parse the contents // into a list of claircore.Vulnerability structs and then return // the list. Parse should assume contents are uncompressed and ready for parsing. Parse(ctx context.Context, contents io.ReadCloser) ([]*claircore.Vulnerability, error) } Parser is an interface which is embedded into the Updater interface.
Parse should be called with an io.ReadCloser struct where the contents
of a security advisory database can be read and parsed into an array of
*claircore.Vulnerability
package driver // import "github.com/quay/claircore/libvuln/driver"
type Fingerprint string
Fingerprint is some identifying information about a vulnerability database.
```
```
package driver // import "github.com/quay/claircore/libvuln/driver"
type Updater interface {
Name() string
Fetcher
Parser
}
Updater is an aggregate interface combining the method set of a Fetcher and
a Parser and forces a Name() to be provided
package driver // import "github.com/quay/claircore/libvuln/driver"
type Fetcher interface { Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error) } Fetcher is an interface which is embedded into the Updater interface.
When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed
If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.
If the conent has not change an Unchanged error should be returned.
package driver // import "github.com/quay/claircore/libvuln/driver"
type Parser interface {
// Parse should take an io.ReadCloser, read the contents, parse the contents
// into a list of claircore.Vulnerability structs and then return
// the list. Parse should assume contents are uncompressed and ready for parsing.
Parse(ctx context.Context, contents io.ReadCloser) ([]*claircore.Vulnerability, error)
}
Parser is an interface which is embedded into the Updater interface.
Parse should be called with an io.ReadCloser struct where the contents
of a security advisory database can be read and parsed into an array of
*claircore.Vulnerability
package driver // import "github.com/quay/claircore/libvuln/driver"
type Fingerprint string Fingerprint is some identifying information about a vulnerability database.
package driver // import "github.com/quay/claircore/libvuln/driver"
type Configurable interface {
Configure(context.Context, ConfigUnmarshaler, *http.Client) error
}
Configurable is an interface that Updaters can implement to opt-in to having
their configuration provided dynamically.
```
```
package driver // import "github.com/quay/claircore/libvuln/driver"
type Updater interface {
Name() string
Fetcher
Parser
}
Updater is an aggregate interface combining the method set of a Fetcher and
a Parser and forces a Name() to be provided
package driver // import "github.com/quay/claircore/libvuln/driver"
type Fetcher interface { Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error) } Fetcher is an interface which is embedded into the Updater interface.
When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed
If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.
If the conent has not change an Unchanged error should be returned.
package driver // import "github.com/quay/claircore/libvuln/driver"
type Parser interface {
// Parse should take an io.ReadCloser, read the contents, parse the contents
// into a list of claircore.Vulnerability structs and then return
// the list. Parse should assume contents are uncompressed and ready for parsing.
Parse(ctx context.Context, contents io.ReadCloser) ([]*claircore.Vulnerability, error)
}
Parser is an interface which is embedded into the Updater interface.
Parse should be called with an io.ReadCloser struct where the contents
of a security advisory database can be read and parsed into an array of
*claircore.Vulnerability
package driver // import "github.com/quay/claircore/libvuln/driver"
type Fingerprint string Fingerprint is some identifying information about a vulnerability database.
package driver // import "github.com/quay/claircore/libvuln/driver"
type Configurable interface {
Configure(context.Context, ConfigUnmarshaler, *http.Client) error
}
Configurable is an interface that Updaters can implement to opt-in to having
their configuration provided dynamically.
package driver // import "github.com/quay/claircore/libvuln/driver"
type ConfigUnmarshaler func(interface{}) error ConfigUnmarshaler can be thought of as an Unmarshal function with the byte slice provided, or a Decode function.
The function should populate a passed struct with any configuration
information.
UpdaterSetFactory
An UpdaterSetFactory is a factory for runtime construction and configuration for Updaters.
package driver // import "github.com/quay/claircore/libvuln/driver"
type UpdaterSetFactory interface {
UpdaterSet(context.Context) (UpdaterSet, error)
}
UpdaterSetFactory is used to construct updaters at run-time.
func StaticSet(s UpdaterSet) UpdaterSetFactory
package driver // import "github.com/quay/claircore/libvuln/driver"
type UpdaterSetFactory interface {
UpdaterSet(context.Context) (UpdaterSet, error)
}
UpdaterSetFactory is used to construct updaters at run-time.
func StaticSet(s UpdaterSet) UpdaterSetFactory
package driver // import "github.com/quay/claircore/libvuln/driver"
type UpdaterSet struct { // Has unexported fields. } UpdaterSet holds a deduplicated set of updaters.
func NewUpdaterSet() UpdaterSet func (s *UpdaterSet) Add(u Updater) error func (s *UpdaterSet) Merge(set UpdaterSet) error func (s *UpdaterSet) RegexFilter(regex string) error func (s *UpdaterSet) Updaters() []Updater
VersionFilter
VersionFilter
is an additional interface a Matcher
may implement.
If implemented, Libvuln
will attempt to use the database and the normalized
version field of a package to filter vulnerabilities in the database.
This is an opt-in optimization for when a package manager's version scheme can
be normalized into a claircore.Version
.
package driver // import "github.com/quay/claircore/libvuln/driver"
type VersionFilter interface {
VersionFilter()
// VersionAuthoritative reports whether the Matcher trusts the database-side
// filtering to be authoritative.
//
// A Matcher may return false if it's using a versioning scheme that can't
// be completely normalized into a claircore.Version.
VersionAuthoritative() bool
}
VersionFilter is an additional interface that a Matcher can implement to
opt-in to using normalized version information in database queries.
Versioned Scanner
A versioned scanner is typically embedded into other scanner types.
It drives claircore's ability to register and understand when updaters have been changed.
Functions that want to work with a generic scanner type should use a VersionedScanner
.
Implementers of this interface must provide a unique name.
Making changes to a scanner's implementation must return a new value from Version
.
Implementers must return the correct kind: one of "package", "distribution", or "repository"
package indexer // import "github.com/quay/claircore/indexer"
type VersionedScanner interface {
// unique name of the distribution scanner.
Name() string
// version of this scanner. this information will be persisted with the scan.
Version() string
// the kind of scanner. currently only package is implemented
Kind() string
}
VersionedScanner can be embedded into specific scanner types. This allows
for methods and functions which only need to compare names and versions of
scanners not to require each scanner type as an argument.
Vulnerability Report
A Vulnerability Report is a structure describing a specific manifest, its contents, and the vulnerabilities affecting its contents.
package claircore // import "github.com/quay/claircore"
type VulnerabilityReport struct {
// the manifest hash this vulnerability report is describing
Hash Digest `json:"manifest_hash"`
// all discovered packages in this manifest keyed by package id
Packages map[string]*Package `json:"packages"`
// all discovered distributions in this manifest keyed by distribution id
Distributions map[string]*Distribution `json:"distributions"`
// all discovered repositories in this manifest keyed by repository id
Repositories map[string]*Repository `json:"repository"`
// a list of environment details a package was discovered in keyed by package id
Environments map[string][]*Environment `json:"environments"`
// all discovered vulnerabilities affecting this manifest
Vulnerabilities map[string]*Vulnerability `json:"vulnerabilities"`
// a lookup table associating package ids with 1 or more vulnerability ids. keyed by package id
PackageVulnerabilities map[string][]string `json:"package_vulnerabilities"`
// a map of enrichments keyed by a type.
Enrichments map[string][]json.RawMessage `json:"enrichments"`
}
VulnerabilityReport provides a report of packages and their associated
vulnerabilities.
A Vulnerability Report is package focused.
Unpacking a report is done by mapping the keys in the PackageVulnerabilities field to the data structures in other lookup maps.
For example:
for pkgID, vulnIDS := range report.PackageVulnerabilities {
// get package data structure
pkg := report.Packages[pkgID]
for _, vulnID := range vulnIDS {
vuln := report.Vulnerabilities[vulnID]
fmt.Printf("package %+v affected by vuln %+v", pkg, vuln)
}
}
Contributors
These topics cover helpful tips for contributing to Claircore.
How to contribute
The preferred workflow is to fork the quay/claircore
repository, push a feature branch to the new fork, then open a pull request.
All pull requests should be targeted to the main
branch outside of exceptional circumstances.
Testing
As many tests as possible should run with the standard go test
invocations.
Adding the special tag integration
(e.g. go test -tags integration ./...
) will also run "integration" tests.
The project interprets "integration" tests to mean any test that would need external resources, such as:
- External web servers
- External network access
- Out-of-process databases
- Large test fixtures
After at least one run with the integration
tag, the tests should cache needed resources and run as many tests as possible.
See also the test/integration
package.
Pull Requests
The Pull Request (PR) is the unit of code review. Claircore's review flow treats a feature branch as a stack of patches to be applied. That is to say, the feature branch should be rebased onto the target branch and have well-organized commits. Merge commits are disallowed. If the author would prefer to not rewrite commit history while working through reviews, fixup commits are the suggested way to achieve that. As many requirements as possible are enforced by CI, like:
- Commits being signed off
- Commit messages having a properly formed subject
- Go modules being tidied
Please use the "draft" option if the branch is not ready. Please enable the "allow edits by maintainers" option.
The maintainers may rebase, push, and merge contributors' branches.
This may necessitate doing a git reset <remote>/<branch>
to update a local branch.
Conventions
Git commits should be formatted like "subject: summary" and avoid going over 80 characters per line.
The "subject" is usually the package affected by the commit (like jar
or rhel
-- the relative path isn't needed) but sometimes a broader category (like docs
, all
, or cicd
) is OK.
All the helper scripts should handle the "normal" convention (origin
is quay/claircore
and fork
is one's personal fork) and the "British" convention (origin
is one's personal fork and upstream
is quay/claircore
).
More detailed contributor documentation can be found in the project documentation.
The claircore project has switched from a git log
based changelog to a
git notes
based changelog.
This has the benefit of making the changelog more human-friendly, as it can have
prose describing changes now, but makes adding entries a bit more involved. A
full understanding of git notes
is helpful for working with the changelog, but
not required. If the reader has worked with the notes
feature before, the
changelog entries are stored under the changelog
ref. For other users, there
are some helper scripts in .github/scripts
.
Basics of git notes
Git notes
is a mechanism for storing additional information alongside commits
without modifying the commits. It does this by creating a ref full of files
named after the commits, with their contents being the notes. This scheme
requires some special care and tooling -- see the documentation for
more information.
Helper scripts
The primary helper script is changelog-edit
. It allows a user to sync down
notes, edit an entry, or both. See the output of the h
flag for more
information.
The other script of interest is changelog-render
, which can be used to render
out the changelog on demand, assuming the changelog notes have been pulled
locally.
The changelog-update
script uses changelog-render
to add to the
CHANGELOG.md
file in the repository root.
Formatting
Broadly, changelog entries should be formatted like commit messages without any
trailers. Entries are turned into list items, with the subject being the bullet
point and the body of the entry being the "body" of the item, or hidden behind
details
elements when using HTML-enabled output.
The entries are almost always rendered as markdown, so using minimal markdown is OK. Anything requiring excessive markdown is probably better served as documentation proper, rather than a changelog entry.
Commit Style
The Claircore project utilizes well structured commits to keep the history useful and help with release automation. We suggest signing off on your commits as well.
A typical commit will take on the following structure:
<scope>: <subject>
<body>
Fixes #1
Pull Request #2
Signed-Off By: <email>
The header of the commit is regexp checked before commit and your commit will be kicked back if it does not conform.
Scope
This is the section of code this commit influences.
You will often see scopes such as "notifier", "auth", "chore", "cicd".
Subject
Subject is a short and concise summary of the change the commit is introducing. It should be a sentence fragment without starting capitalization and ending punctuation and limited to about 60 characters, to allow for the scope prefix and decoration in the git log.
Body
Body should be full of detail.
Explain what this commit is doing and why it is necessary.
You may include references to issues and pull requests as well.
Local Development
A local development environment is implemented via docker-compose.
Usage
Several make targets are defined for working with the local development environment.
claircore-db-up - creates just the claircore database useful for running integration tests without test servers
claircore-db-restart - destroys and recreates a fresh database. localhost:5434
Tests
Several make targets are defined for working with tests.
integration - run the integration test suite. requires the claircore-db to be up. run `make clair-db-up` before this target
unit - run the unit test suite.
bench - runs the benchmarks
integration-v - runs the integration test suite with verbose
unit-v - runs the unit test suite with verbose
Logging
All the logging in claircore is done with zerolog via context.Context
values. The zlog
package takes OpenTelemetry labels and attaches them to
zerolog
events.
This allows for claircore's logging to be used consistently throughout all the packages without having unintended prints to stderr.
How to Log
Adding Context
In a function, use zlog
to add key-value pairs of any relevant context:
ctx = zlog.ContextWithValues(ctx,
"component", "Example.Logger")
Alternatively, the go.opentelemetry.io/otel/baggage
package can be used for
more explicit control around the baggage values.
Logging style
Constant Messages
Zerolog emits lines when the Msg
or Msgf
methods are called. Project style
is to not use Msgf
. Any variable data should be set as key-value pairs on
the Event object.
For example, don't do this:
zlog.Info(ctx).Msgf("done at: %v", time.Now())
Do this instead:
zlog.Info(ctx).
Time("time", time.Now()).
Msgf("done")
Grammar
When noting the change during a chunk of work, make sure that the log messages scan as visually similar. Usually, this means formatting messages into "${process} ${event}". For example:
frob start
frob initialized
frob ready
frob success
frob done
Is much easier to scan than:
starting to frob
initialized frobber
ready for frobbing
did frob
done with frobing
Don't log and return
When handling an error, code should only log it if it does not propagate it. The code that ultimately handles the error is responsible for deciding what to do with it. Logging and returning ends up with the same message repeated multiple times in the logs.
Levels
Claircore attempts to have consistent leveled logging. The rules for figuring out what level to use is:
-
Panic
There's some occurrence that means the process won't work correctly.
-
Fatal
Unused, because it prevents defers from running.
-
Error
Something unexpected occurred and the process can continue, but a human needs to be notified. An error will be returned for this request.
-
Warn
Something unexpected occurred and the process can continue. An error will be returned for this request.
-
Info
Some information that may be useful to an operator. Examples include a timer-based process starting and ending, a user request starting and ending, or a summary of work done.
-
Debug
Some information that may be useful to a developer. Examples include entering and exiting functions, stepping through a function, or specific file paths used while work is being done.
Here's various codebase conventions that don't have dedicated pages:
- URLs
URLs in code should be annotated with a
//doc:url
directive comment. See the theinternal/cmd/mdbook-injecturls
command for documentation on how the preprocessor works. The list of keywords isn't an allowlist, so an invocation like the following should list the ones actually used in the documentation using a command likegit grep injecturls -- :/docs/*.md
.
Tests
Tests in the claircore module may use various helpers underneath the test
directory.
Using these packages outside of testing code is disallowed.
Assert packages are disallowed;
the go-cmp
package is the only external package helper allowed.
Tests that use external resources or generate test fixtures should be annotated according to the integration
package.
Caching
Tests using the integration
package cache generated and downloaded assets into a directory named clair-testing
inside the directory reported by os.UserCacheDir
.
For example, on a Linux system, the cache directory will be (in sh
notation) ${XDG_CACHE_HOME-$HOME/.cache}/clair-testing
.
Go Version
Claircore endeavors to declare a go
version in the go.mod directive that is the minimum required for all the language features to work as expected.
The go
version will also be updated if there is deemed to be a security fix that could directly affect claircore's codebase.