The rolodex of OpenAPI references

If you have ever used OpenAPI with golang, you’re probably aware of using $ref to reference other objects like Schemas or Parameters that we want to re-use across a specification, or across multiple specifications.

libopenapi needs a way to look up these references and resolve them.

What are references in OpenAPI?

A $ref is a JSON Pointer that points to a location in the specification. The location could be local to the current document, or it could be in another document on the local file system, or out there in cyberspace somewhere.

What is the rolodex?

A rolodex is a rotating file device used to store business contact information. Its name is a portmanteau of the words rolling and index.

The rolodex inside libopenapi is a data structure that holds all the references in an OpenAPI specification and dependent files, and knows how to look them up.

How does the rolodex work?

The rolodex is a directory system that holds every index created for every file or remote based reference.

The rolodex contains a local and remote virtual filesystem that are used to locate files and remote references, and then keep track of them.

Thr rolodex starts by indexing the root document. The indexing process will first locate all $ref nodes in the document and then try to map those references by first trying to open the referenced document (either via the local or remote file system), and then once opened, that referenced document will also in turn be indexed.

Once there is nothing left to lookup, the rolodex will have a complete index of the entire specification, and know the location of every referring document. The rolodex stores references as ‘absolute’ references, meaning that they are always recorded as a full URL, or a full file path, combined with the JSON Pointer to the object being referenced.

Rolodex architecture

libopenapi starts with creating a new rolodex for a root OpenAPI document and then indexes it, recursively looking up each $ref as it comes across it.

Local references

The index will locate every $ref and determine what type of lookup to perform, if the reference is local, there is no prefix attached to the reference path, so it looks something like this:

  $ref: "#/components/schemas/Burger"

The # (hash) symbol indicates the root of the document.

The index looks up the component from within the same document and keeps a record of the found object. It does not search the rolodex for the object, because it’s already in the same document.

File references

If the reference is a file reference, it will look something like this:

  $ref: "some/path/to/openapi-schemas.yaml#/components/schemas/Burger"

The prefix (before the hash) is the path to the file that contains the lookup. libopenapi will attempt to locate the file on the local operating system and then load it into the index. If the path is relative, the rolodex will look from where it is being run as the working directory.

When a file reference is used, a whole new index is created for the entire file. The rolodex, maintains indexes for every file pulled in.

The rolodex contains a built-in local file system called LocalFS that works in two ways. By default it operates as a recursive lookup, which means it opens files as it scans through each OpenAPI file looking for references.

In another mode, it can be set to scan everything from the working directory down, regardless if the file is referenced or not, if it’s a YAML or JSON file, it will be indexed.

Remote references

If the reference is a remote reference, it will look something like this:

  $ref: "https://pb33f.io/openapi.yaml#/components/schemas/Burger"

Like file references, the prefix (before the hash) is the URI to the file that contains the lookup. libopenapi will pull down this remote document and then treat it as a local reference internally.

When a remote reference is used, a whole new index is created for the entire remote file (like file references).

The rolodex contains a built-in remote file system called RemoteFS that only works in recursive mode, which means it it can only lookup files that are referenced. It does not support scanning the entire remote file system.

Building a rolodex

If you’re building documents using the model, then you don’t need to worry about the one will be created for you based on the document configuration.

If a BaseURL or BasePath is provided or AllowFileReferences or AllowRemoteReferences for a document configuration are set, the a rolodex with local and/or remote file systems will be set up.

Adding remote and local file systems

The rolodex and the index(es) work symbiotically. You can create an index without a rolodex, but you cannot create a rolodex without an index, it has no value.

If your spec does NOT have any kind of remote/file based references, then you don’t need a rolodex, you can just create an index as everything will be looked up locally.

Below is a complete example of how to use the rolodex with both local and remote file systems.

import (
  "github.com/pb33f/libopenapi/index"
  "gopkg.in/yaml.v3"
  "os"
)

func main() {

  // load in the spec
  actualYaml, _ := os.ReadFile("spec.yaml")

  // set our base directory
  basePath := "../some/basePath/where/files/live"

  // create an index config
  config := index.CreateOpenAPIIndexConfig()

  // the rolodex will automatically try and check for circular references, you don't want to do this
  // if you're resolving the spec, as the node tree is marked as 'seen' and you won't be able to resolve
  // correctly.
  config.AvoidCircularReferenceCheck = true

  // new in 0.13+ is the ability to add remote and local file systems to the index
  // requires a new part, the rolodex. It holds all the indexes and knows where to find
  // every reference across local and remote files.
  rolodex := index.NewRolodex(config)

  // create a local file system config, tell it where to look from and the index config to pay attention to.
  fsCfg := &index.LocalFSConfig{
      BaseDirectory: basePath,
      IndexConfig:   config,
  }

  // create a local file system using config.
  fileFS, err := index.NewLocalFSWithConfig(fsCfg)
  if err != nil {
      panic(err)
  }

  // unmarshal the spec into a yaml node
  var rootNode yaml.Node
  _ = yaml.Unmarshal([]byte(actualYaml), &rootNode)

  // set the root node of the rolodex (this is the root of the spec)
  rolodex.SetRootNode(&rootNode)

  // add local file system to rolodex
  rolodex.AddLocalFS(basePath, fileFS)

  // add a new remote file system.
  remoteFS, _ := index.NewRemoteFSWithConfig(config)

  // add the remote file system to the rolodex
  rolodex.AddRemoteFS("", remoteFS)

  // set the root node of the rolodex, this is your spec.
  rolodex.SetRootNode(&rootNode)

  // index the rolodex
  indexingError := rolodex.IndexTheRolodex()
  if indexingError != nil {
      panic(indexingError)
  }

  // resolve the rolodex (if you want to)
  rolodex.Resolve()

  // there should be no errors at this point
  resolvingErrors := rolodex.GetCaughtErrors()
  if resolvingErrors != nil {
      panic(resolvingErrors) 
  }
}

Index everything

When you’re ready to index everything, you can call IndexTheRolodex() on the rolodex. This will recursively index everything that can be found.

Any resolving or lookup errors will be returned as a single error, however the rolodex has joined multiple errors into a single error, so you can use utils.UnwrapError() to get the original errors returned as a slice.

Resolving

Resolving is a destructive operation, meaning that the node tree is permanently changed, references will be permanently shifted in that model.

Use the Resolve() method on the rolodex to resolve the entire specification.

Any errors that were found during resolution can be retrieved by calling GetCaughtErrors() on the rolodex. This will return a slice of errors.

Check for circular references

The rolodex will automatically check for circular references, but you can turn this off by setting the AvoidCircularReferenceCheck on the index config to true.

To manually check for circular references, you can call CheckForCircularReferences() on the rolodex. This will return a slice of errors, each error will be a circular reference error.

Any caught errors can be retrieved by calling GetCaughtErrors() on the rolodex. This will return a slice of errors.

Add a custom HTTP handler

If you need to customize how HTTP references are looked up by the rolodex, you can add a custom HTTP handler to the remote file system by setting SetRemoteHandlerFunc() and passing in any function that matches the signature func(url string) (*http.Response, error)

This is useful if your specifications are held in a private repository and require authentication to access.

import (
  "fmt"
  "github.com/pb33f/libopenapi/index"
  "net/http"
  "os"
  "time"
)

func customHttpHandler() {

  // create an index config
  config := index.CreateOpenAPIIndexConfig()

  // create a new rolodex
  rolodex := index.NewRolodex(config)

  // create a new remote fs and set the config for indexing.
  remoteFS, _ := index.NewRemoteFSWithConfig(config)

  // custom http client.  
  client := &http.Client{
      Timeout: time.Second * 120,
  }

  // custom handler func
  customHandler := func(url string) (*http.Response, error) {

    // create a new request
    request, _ := http.NewRequest(http.MethodGet, url, nil)
    request.Header.Set("Authorization", fmt.Sprintf("Bearer %s", os.Getenv("GITHUB_TOKEN")))
    return client.Do(request)

  }

  // if a token is set, then use the custom handler.
  if os.Getenv("GITHUB_TOKEN") != "" {
    remoteFS.SetRemoteHandlerFunc(customHandler)
  }

  // add remote filesystem
  rolodex.AddRemoteFS("", remoteFS)
}

Add a custom file system

One of the design choices of libopenapi is to allow for custom file systems to be added to the rolodex. Some use-cases might include the need to look up references from databases, or other sources other than local filer or via http.

The rolodex uses fs.FS as the interface for file systems. This is a built-in interface in golang.

To add a custom file system, you need to implement the index.RolodexFS interface (which also defines the Open(name string) (fs.File, error) method)

// RolodexFS is an interface that represents a RolodexFS, is the same interface as `fs.FS`, except it
// also exposes a GetFiles() signature, to extract all files in the FS.
type RolodexFS interface {
  Open(name string) (fs.File, error)
  GetFiles() map[string]RolodexFile
}

The GetFiles() method returns a map with the key being the file path, and the value being a RolodexFile interface.

RolodexFile is an interface that exposes the fs.FileInfo and fs.File interface methods.

The rolodex understands two concepts, local files and remote files, so when deciding to add a custom file system, you should decide how you want rolodex to use it.

Anything reference that starts with http is considered remote by the rolodex, everything is else a file.

The rolodex comes with two built-in file systems, LocalFS and RemoteFS. To build custom file systems, you can use these as examples.