Look up anything in an OpenAPI specification (v0.12)

When creating a model using libopenapi, under the hood, it creates an index of everything that was located in an OpenAPI specification. There is a considerable amount items it tracks:

If you’re interested in the full list of items that are tracked, look at the index docs and all of the GetXxx methods are laid out with docs (there’s no point duplicating it here)

An index is really useful when a map of an OpenAPI spec is needed. Knowing where all the references are and where they point, is very useful when resolving specifications, or just looking things up.

Comes for free with every model

When parsing an OpenAPI specification, the index is created for free. You don’t have to do anything to get it. To gain access to the index, it’s attached to the DocumentModel that is returned when calling BuildVXModel() method.

For example, parsing an OpenAPI specification and then printing out the number of schemas found in the spec.

import (
    "fmt"
    "github.com/pb33f/libopenapi"
    "io/ioutil"
)

func main() {
    // load an OpenAPI 3 specification from bytes
    petstore, _ := ioutil.ReadFile("petstorev3.json")

    // create a new document from specification bytes,
    // ignore the errors for the sake of brevity
    doc, _ := libopenapi.NewDocument(petstore)

    // because we know this is a v3 spec, we can build a ready to go
    // model from it - also ignore the errors.
    v3Model, _ := doc.BuildV3Model()

    // extract the index from the model.
    index := v3Model.Index
    
    // print the number of paths and schemas in the document
    fmt.Printf("There are %d paths and %d schemas in the document\n",
        len(index.GetAllPaths()), len(index.GetAllSchemas()))
}

Creating an index from scratch

If you have a specification in bytes, you can create an index from scratch. This is useful if you want to avoid creating a model and just want to use the index.

To create an index, use the index.NewSpecIndexWithConfig function and pass in a *root *yaml.Node pointer - unmarshalled from an OpenAPI specification bytes.

The configuration used to be optional in versions per 0.6.0, however now it’s something that should be passed in.

Configuration

A specification index can be configured to load in remote and local references. It can also be configured with a Base URL that is used to resolve all relative references found in a specification.

Before version 0.6.0 local and remote references were followed automatically when discovered, this is a potential security risk, so now it’s up to the user to decide if they want to follow them or not.

By default, they are turned off and have to be switched on.

The index.SpecIndexConfig is used to define a new index configuration.

Allowing remote and local file references

Create a new index.SpecIndexConfig and set the AllowRemoteLookup and AllowFileLookup to true.

config := index.SpecIndexConfig{
    AllowRemoteLookup: true,
    AllowFileLookup: true,
}

You can use the index.CreateOpenAPIIndexConfig() function to do the same thing (it’s simpler)

config := index.CreateOpenAPIIndexConfig()

The opposite effect (don’t allow following) can be achieved by using index.CreateClosedAPIIndexConfig()

Setting a Base URL

If the OpenAPI specification contains relative references, then a Base URL or a Base Path is needed to resolve them. Perhaps the most egregious example of this is the DigitalOcean OpenAPI Specification, which has is composed of over 1300 files. Omfg.

The BaseURL field in the index.SpecIndexConfig is used to resolve relative references, if all the files are still kept on the remote server and not locally.

// Digital Ocean needs a baseURL to be set, so we can resolve relative references.
baseURL, _ := url.Parse("https://raw.githubusercontent.com/digitalocean/openapi/main/specification")

config := index.SpecIndexConfig{
    AllowRemoteLookup: true,
    AllowFileLookup: true,
    BaseURL: baseURL,
}

Setting a Base Path

If all the files are local to the file system, then a Base Path can be used to resolve relative references.

config := index.SpecIndexConfig{
    AllowRemoteLookup: true,
    AllowFileLookup: true,
    BasePath: "../some/location/on/disk",
}

Using a custom HTTP Resolver

When resolving remote references, the default HTTP resolver uses the net/http package to make requests. It’s easy to provide a custom HTTP handler for resolving remote references.

This is useful if your specifications are held in a private repository and require authentication to access.

To provide a custom HTTP handler, set the RemoteURLHandler field to a function that accepts a URL string and returns a *http.Response and an error.

All network requests will then use this function for fetching remote documents.

myHandler := func(url string) (*http.Response, error) {
    // do something with the url
    // return a response and an error
}

config := index.SpecIndexConfig{
    AllowRemoteLookup: true,
    AllowFileLookup: true,
    RemoteURLHandler: myHandler,
}

Using a custom fs.FS resolver

Sometimes we keep specifications in databases, or S3 buckets, or some other place that isn’t the file system or served over HTTP.

In this case, we can provide a custom fs.FS resolver to the FSHandler and all local and remote references will be resolved using this function.

var myHandler fs.FS // implements the fs.FS interface

...

config := index.SpecIndexConfig{
    AllowRemoteLookup: true,
    AllowFileLookup: true,
    FSHandler: myHandler,
}

All myHandler needs to do is implement the fs.FS interface. All calls to Open will be made with the path of the file or the URL/component path of the remote reference.

Putting it all together

Let’s read in the DigitalOcean OpenAPI Specification (all 1378 files) and create an index from it.

import (
    "fmt"
    "github.com/pb33f/libopenapi"
    "io/ioutil"
)

// read the Digital Ocean OpenAPI Specification from Github.
func read() []byte {
    res, err := http.Get("https://raw.githubusercontent.com/digitalocean" +
                    "/openapi/main/specification/DigitalOcean-public.v2.yaml")
    if err != nil {
        panic(err)
    }
    defer res.Body.Close()
    data, err := io.ReadAll(res.Body)
    if err != nil {
        panic(err)
    }
    return data
}`

func main() {
  // Digital Ocean needs a baseURL to be set, so we can resolve relative references.
  baseURL, _ := url.Parse("https://raw.githubusercontent.com/digitalocean/openapi/main/specification")

  // create a new SpecIndexConfig that allows all following and sets the baseURL correctly.
  config := index.SpecIndexConfig{
      AllowRemoteLookup: true,
      AllowFileLookup:   true,
      BaseURL:           baseURL,
  }

  // create a root yaml.Node from the specification bytes
  var rootNode yaml.Node
  _ = yaml.Unmarshal(read(), &rootNode)

  // create a new index from the specification bytes and the config.
  idx := index.NewSpecIndexWithConfig(&rootNode, &config)

  // create a new resolver from the index
  rsvlr := resolver.NewResolver(idx)

  // check for circular references
  errs := rsvlr.CheckForCircularReferences()
  if errs != nil {
      panic(errs)
  }

  // print out some interesting information discovered when resolving the references.
  fmt.Printf("%d references visited\n", rsvlr.GetReferenceVisited())
  fmt.Printf("%d journeys taken\n", rsvlr.GetJourneysTaken())
  fmt.Printf("%d index visits\n", rsvlr.GetIndexesVisited())
  fmt.Printf("%d relatives seen\n", rsvlr.GetRelativesSeen())
}

It will spit out something like:

All the things available in the index

Is considerable, so instead of re-listing everything here, head over to the go docs and see them there.