Libindex Usage

Libindex is the Go package responsible for fetching container image layers, identifying packages, distributions, and repositories within these layers, and computing a final coalesced Index Report.

An Index Report is primarily used as input to Libvuln's vulnerability matching process.

Usage

Libindex is runtime constructed via the libindex.New method. New requires an libindex.Opts struct.

Options

package libindex // import "github.com/quay/claircore/libindex"

type Options struct {
	// Store is the interface used to persist and retrieve results of indexing.
	Store indexer.Store
	// Locker provides system-wide locks. If the indexing work is distributed the
	// lock should be backed by a distributed store.
	Locker LockSource
	// FetchArena is an interface tied to the lifecycle of LibIndex to enable management
	// of the filesystem while separate processes are dealing with layers, for example:
	// you can reference count downloaded layer files to avoid racing.
	FetchArena indexer.FetchArena
	// ScanLockRetry specifies how often we should try to acquire a lock for scanning a
	// given manifest if lock is taken.
	ScanLockRetry time.Duration
	// LayerScanConcurrency specifies the number of layers to be scanned in parallel.
	LayerScanConcurrency int
	// LayerFetchOpt is unused and kept here for backwards compatibility.
	LayerFetchOpt interface{}
	// NoLayerValidation controls whether layers are checked to actually be
	// content-addressed. With this option toggled off, callers can trigger
	// layers to be indexed repeatedly by changing the identifier in the
	// manifest.
	NoLayerValidation bool
	// ControllerFactory provides an alternative method for creating a scanner during libindex runtime
	// if nil the default factory will be used. useful for testing purposes
	ControllerFactory ControllerFactory
	// Ecosystems a list of ecosystems to use which define which package databases and coalescing methods we use
	Ecosystems []*indexer.Ecosystem
	// ScannerConfig holds functions that can be passed into configurable
	// scanners. They're broken out by kind, and only used if a scanner
	// implements the appropriate interface.
	//
	// Providing a function for a scanner that's not expecting it is not a fatal
	// error.
	ScannerConfig struct {
		Package, Dist, Repo, File map[string]func(interface{}) error
	}
	Resolvers []indexer.Resolver
}
    Options are dependencies and options for constructing an instance of
    libindex

The above outlines the relevant bits of the Options structure.

Store is required needs to be an object that satisfies the indexer.Store interface.

Locker is required and needs to satisfy the LockSource interface.

FetchArena is required and needs to satify the FetchArena interface.

Providing a nil "Ecosystems" slice will supply the default set, instructing Libindex to index for all supported content in a layer, and is typically desired.

Construction

Constructing Libindex is straight forward.

	opts := new(libindex.Options)
	// Populate with desired settings...
	lib, err := libindex.New(ctx, opts, http.DefaultClient)
	if err != nil {
		panic(err)
	}
	defer lib.Close(ctx) // Remember to cleanup when done.

The constructing code should provide a valid Context tied to some lifetime.

Indexing

Indexing is the process of submitting a manifest to Libindex, fetching the manifest's layers, indexing their contents, and coalescing a final Index Report.

Coalescing is the act of computing a final set of contents (packages, distributions, repos) from a set of layers. Since layers maybe shared between many manifests, the final contents of a manifest must be computed.

To perform an Index you must provide a claircore.Manifest data struture to the Index method. The Manifest data structure describes an image manifest's layers and where they can be fetched from.

	m := new(claircore.Manifest)
	// Populate somehow ...
	ir, err := lib.Index(ctx, m)
	if err != nil {
		panic(err)
	}

The Index method will block until an claircore.IndexReport is returned. The context should be bound to some valid lifetime such as a request.

As the Indexer works on the manifest it will update its database throughout the process. You may view the status of an index report via the "IndexReport" method.

	ir, ok, err := lib.IndexReport(ctx, m.Hash)
	if err != nil {
		panic(err)
	}

Libindex performs its work incrementally and saves state as it goes along. If Libindex encounters an intermittent error during the index (for example, due to network failure while fetching a layer), when the manifest is resubmitted only the layers not yet indexed will be fetched and processed.

State

Libindex treats layers as content addressable. Once a layer identified by a particular hash is indexed its contents are definitively known. A request to re-index a known layer results in returning the previous successful response.

This comes in handy when dealing with base layers. The Ubuntu base layer is seen very often across container registries. Treating this layer as content addressable precludes the need to fetch and index the layer every time Libindex encounters it in a manifest.

There are times where re-indexing the same layer is necessary however. At the point where Libindex realizes a new version of a component has not indexed a layer being submitted it will perform the indexing operation.

A client must notice that Libindex has updated one of its components and subsequently resubmit Manifests. The State endpoint is implemented for this reason.

Clients may query the State endpoint to receive an opaque string acting as a cookie, identifying a unique state of Libindex. When a client sees this cookie change it should re-submit manifests to Libindex to obtain a new index report.

	state, err := lib.State(ctx)
	if err != nil {
		panic(err)
	}
	if state == prevState {
		// Nothing to do.
		return
	}
	// Otherwise, re-index manifest.

AffectedManifests

Libindex is capable of providing a client with all manifests affected by a set of vulnerabilities. This functionality is designed for use with a notification mechanism.

	var vulns []claircore.Vulnerability
	affected, err := lib.AffectedManifests(ctx, vulns)
	if err != nil {
		panic(err)
	}
	for manifest, vulns := range affected.VulnerableManifests {
		for _, vuln := range vulns {
			fmt.Printf("vuln affecting manifest %s: %+v", manifest, vuln)
		}
	}

The slice of vulnerabilities returned for each manifest hash will be sorted by claircore.NormalizedSeverity in "most severe" descending order.