Look up anything in an OpenAPI specification

The index knows where to find everything in a specification

This documentation only applies to v0.13+ of libopenapi. In all releases before then the index and resolver architecture was significantly different and the behavior and relationship of them was also different.

View the older documentation for v0.12 and below

When creating a model using libopenapi, under the hood, it creates an index of everything that was located in an OpenAPI specification. There is a considerable amount items it tracks:

If you’re interested in the full list of items that are tracked, look at the index docs and all of the GetXxx methods are laid out with docs (there’s no point duplicating it here)

An index is really useful when a map of an OpenAPI spec is needed. Knowing where all the references are and where they point, is very useful when resolving specifications, or just looking things up.

Comes for free with every model

When parsing an OpenAPI specification, the index is created for free. You don’t have to do anything to get it. To gain access to the index, it’s attached to the DocumentModel that is returned when calling BuildVXModel() method.

For example, parsing an OpenAPI specification and then printing out the number of schemas found in the spec.

import (
    "fmt"
    "github.com/pb33f/libopenapi"
    "io/ioutil"
)

func main() {
    // load an OpenAPI 3 specification from bytes
    petstore, _ := ioutil.ReadFile("petstorev3.json")

    // create a new document from specification bytes,
    // ignore the errors for the sake of brevity
    doc, _ := libopenapi.NewDocument(petstore)

    // because we know this is a v3 spec, we can build a ready to go
    // model from it - also ignore the errors.
    v3Model, _ := doc.BuildV3Model()

    // extract the index from the model.
    index := v3Model.Index
    
    // print the number of paths and schemas in the document
    fmt.Printf("There are %d paths and %d schemas in the document\n",
        len(index.GetAllPaths()), len(index.GetAllSchemas()))
}

Creating an index from scratch

If you have a specification in bytes, you can create an index from scratch. This is useful if you want to avoid creating a model and just want to use the index.

To create an index, use the index.NewSpecIndexWithConfig function and pass in a *root *yaml.Node pointer - unmarshalled from an OpenAPI specification bytes.

The configuration used to be optional in versions per 0.6.0, however now it’s something that should be passed in.

Do you need to create an index?

Probably not. An index exists for each individual file that is parsed, and the rolodex creates and holds references to the indexes.

However there may be a usecase where an individual index is needed.

Configuration

The index.SpecIndexConfig is used to define a new index configuration.

Field Type Description
BaseURL *url.URL Base URL for resolving relative references if the specification is exploded.
BasePath string Base Path for resolving relative references if the specification is exploded.
AllowRemoteLookup bool Allow remote lookups for references. Defaults to false
AllowFileLookup bool Allow file lookups for references. Defaults to false
AvoidBuildIndex bool Avoids building out the index when its created. Defaults to false
AvoidCircularReferenceCheck bool Used by the rolodex, avoids invoking the resolver to check for circular references. Defaults to false
Logger *slog.Logger Provide a custom structured logger
Rolodex *index.Rolodex Provide a custom rolodex
IgnorePolymorphicCircularReferences bool Ignore circular references that are polymorphic. Defaults to false
IgnoreArrayCircularReferences bool Ignore array based circular references. Defaults to false

Allowing remote and local file references

As of version v0.13, the index no-longer performs any file lookups, or remote lookups. This is to allow the index to be used in a wider variety of use cases and be more focused on individual files, rather than trying to track everything.

To use remote references, or file references, use the rolodex.

The rolodex uses the SpecIndexConfig to configure its self, which is why values for configuring remote references are still present, however the index alone no-longer uses these properties. Please see the older index documentation for more details.

Setting a Base URL

If the OpenAPI specification contains relative references, then a Base URL or a Base Path is needed to resolve them. Perhaps the most egregious example of this is the DigitalOcean OpenAPI Specification, which has is composed of over 1300 files. Omfg.

The BaseURL field in the index.SpecIndexConfig is used to resolve relative references, if all the files are still kept on the remote server and not locally.

// Digital Ocean needs a baseURL to be set, so we can resolve relative references.
baseURL, _ := url.Parse("https://raw.githubusercontent.com/digitalocean/openapi/main/specification")

config := index.SpecIndexConfig{
    AllowRemoteLookup: true, // must be set or the rolodex will complain
    BaseURL: baseURL,
}

Setting a Base Path

If all the files are local to the file system, then a Base Path can be used to resolve relative references.

config := index.SpecIndexConfig{
    AllowFileLookup: true, // must be set or the rolodex will complain
    BasePath: "../some/location/on/disk",
}

The rolodex actually performs all the indexing, and it uses the BaseURL to determine how to lookup remote references, and the BasePath to determine how to lookup local references.

Digital Ocean Example

To suck in the entire DigitalOcean OpenAPI Specification, and create indexes for all of them, we will need the rolodex to do the heavy lifting.

// read the Digital Ocean OpenAPI Specification from Github.
func read() []byte {
  res, err := http.Get("https://raw.githubusercontent.com/digitalocean" +
      "/openapi/main/specification/DigitalOcean-public.v2.yaml")
  if err != nil {
      panic(err)
  }
  defer res.Body.Close()
  data, err := io.ReadAll(res.Body)
  if err != nil {
      panic(err) 
  } 
  return data
}

// Index the digital ocean OpenAPI specification
func digitalOcean() {

  // create a root node to unmarshal the spec into.
  var rootNode yaml.Node
  _ = yaml.Unmarshal(read(), &rootNode)

  // create a new config that allows remote lookups.
  indexConfig := index.CreateOpenAPIIndexConfig()

  // we're going to check for circular references later, so we don't want to do it now.
  indexConfig.AvoidCircularReferenceCheck = true

  // create a custom logger (optional)
  indexConfig.Logger = slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
      Level: slog.LevelError,
  }))

  // define the base URL for the remote filesystem.
  location := "https://raw.githubusercontent.com/digitalocean/openapi/main/specification"
  baseURL, _ := url.Parse(location)

  // set the base URL for the remote filesystem in the config.
  indexConfig.BaseURL = baseURL

  // create a new rolodex
  rolodex := index.NewRolodex(indexConfig)

  // set the rolodex root node to the root node of the spec. 
  rolodex.SetRootNode(&rootNode)

  // create a new remote fs and set the config for indexing.
  remoteFS, _ := index.NewRemoteFSWithConfig(indexConfig)

  // add remote filesystem
  rolodex.AddRemoteFS(location, remoteFS)

  // index the rolodex
  indexedErr := rolodex.IndexTheRolodex() 
  if indexedErr != nil {
      panic(indexedErr) 
  }

  // get all the files!
  files := remoteFS.GetFiles()
  fileLen := len(files)

  // check for circular references across the entire document.
  rolodex.CheckForCircularReferences()

  fmt.Printf("%d files found and %d errors reported. There were %d circular references found.\n",
      fileLen, len(remoteFS.GetErrors()), len(rolodex.GetCaughtErrors()))

  // extrtact the resolver from the root index.
  resolver := rolodex.GetRootIndex().GetResolver()

  // print out some interesting information discovered when visiting all the references.
  fmt.Printf("%d references visited\n", resolver.GetReferenceVisited())
  fmt.Printf("%d journeys taken\n", resolver.GetJourneysTaken())
  fmt.Printf("%d index visits\n", resolver.GetIndexesVisited())
  fmt.Printf("%d relatives seen\n", resolver.GetRelativesSeen())
}

This operation takes a few seconds, libopenapi is making thousands of HTTP requests to Github to get all the files.

After a few seconds, tt will spit out something like:

1677 files found and 0 errors reported. There were 0 circular references found. 4633 references visited 299 journeys taken 1544 index visits 12916 relatives seen

All the things available in the index

The list is considerable, so instead of re-listing everything here, head over to the go docs and see them there.