Look up anything in an OpenAPI specification
The index knows where to find everything in a specificationThis documentation only applies to v0.13+ of libopenapi
. In all releases before then the
index and resolver
architecture was significantly different and the behavior and relationship of them was also different.
When creating a model using libopenapi, under the hood, it creates an index of everything that was located in an OpenAPI specification. There is a considerable amount items it tracks:
If you’re interested in the full list of items that are tracked, look at the index docs
and all of the GetXxx
methods are laid out with docs (there’s no point duplicating it here)
An index is really useful when a map of an OpenAPI spec is needed. Knowing where all the references are and where they point, is very useful when resolving specifications, or just looking things up.
Comes for free with every model
When parsing an OpenAPI specification, the index is created for free. You don’t have to do anything to get it. To gain
access to the index, it’s attached to the DocumentModel
that is returned when calling BuildVXModel()
method.
For example, parsing an OpenAPI specification and then printing out the number of schemas found in the spec.
import (
"fmt"
"github.com/pb33f/libopenapi"
"io/ioutil"
)
func main() {
// load an OpenAPI 3 specification from bytes
petstore, _ := ioutil.ReadFile("petstorev3.json")
// create a new document from specification bytes,
// ignore the errors for the sake of brevity
doc, _ := libopenapi.NewDocument(petstore)
// because we know this is a v3 spec, we can build a ready to go
// model from it - also ignore the errors.
v3Model, _ := doc.BuildV3Model()
// extract the index from the model.
index := v3Model.Index
// print the number of paths and schemas in the document
fmt.Printf("There are %d paths and %d schemas in the document\n",
len(index.GetAllPaths()), len(index.GetAllSchemas()))
}
Creating an index from scratch
If you have a specification in bytes, you can create an index from scratch. This is useful if you want to avoid creating a model and just want to use the index.
To create an index, use the index.NewSpecIndexWithConfig
function
and pass in a *root *yaml.Node
pointer - unmarshalled from
an OpenAPI specification bytes.
The configuration used to be optional in versions per 0.6.0
, however now it’s something that should be passed in.
Do you need to create an index?
Probably not. An index exists for each individual file that is parsed, and the rolodex creates and holds references to the indexes.
However there may be a usecase where an individual index is needed.
Configuration
The index.SpecIndexConfig
is used to define a new index configuration.
Field | Type | Description |
---|---|---|
BaseURL | *url.URL |
Base URL for resolving relative references if the specification is exploded. |
BasePath | string |
Base Path for resolving relative references if the specification is exploded. |
AllowRemoteLookup | bool |
Allow remote lookups for references. Defaults to false |
AllowFileLookup | bool |
Allow file lookups for references. Defaults to false |
AvoidBuildIndex | bool |
Avoids building out the index when its created. Defaults to false |
AvoidCircularReferenceCheck | bool |
Used by the rolodex, avoids invoking the resolver to check for circular references. Defaults to false |
Logger | *slog.Logger |
Provide a custom structured logger |
Rolodex | *index.Rolodex |
Provide a custom rolodex |
IgnorePolymorphicCircularReferences | bool |
Ignore circular references that are polymorphic. Defaults to false |
IgnoreArrayCircularReferences | bool |
Ignore array based circular references. Defaults to false |
Allowing remote and local file references
As of version v0.13
, the index no-longer performs any file lookups, or remote lookups. This is to allow the index to be
used in a wider variety of use cases and be more focused on individual files, rather than trying to track everything.
To use remote references, or file references, use the rolodex.
The rolodex uses the SpecIndexConfig
to configure its self, which is why values for configuring remote references are
still present, however the index alone no-longer uses these properties. Please see
the older index documentation for more details.
Setting a Base URL
If the OpenAPI specification contains relative references, then a Base URL or a Base Path is needed to resolve them. Perhaps the most egregious example of this is the DigitalOcean OpenAPI Specification, which has is composed of over 1300 files. Omfg.
The BaseURL
field in the index.SpecIndexConfig
is used to resolve relative references, if all the files are still
kept on the remote server and not locally.
// Digital Ocean needs a baseURL to be set, so we can resolve relative references.
baseURL, _ := url.Parse("https://raw.githubusercontent.com/digitalocean/openapi/main/specification")
config := index.SpecIndexConfig{
AllowRemoteLookup: true, // must be set or the rolodex will complain
BaseURL: baseURL,
}
Setting a Base Path
If all the files are local to the file system, then a Base Path can be used to resolve relative references.
config := index.SpecIndexConfig{
AllowFileLookup: true, // must be set or the rolodex will complain
BasePath: "../some/location/on/disk",
}
The rolodex actually performs all the indexing, and it uses the BaseURL
to determine how to lookup remote references, and the BasePath
to determine how to lookup local references.
Digital Ocean Example
To suck in the entire DigitalOcean OpenAPI Specification, and create indexes for all of them, we will need the rolodex to do the heavy lifting.
// read the Digital Ocean OpenAPI Specification from Github.
func read() []byte {
res, err := http.Get("https://raw.githubusercontent.com/digitalocean" +
"/openapi/main/specification/DigitalOcean-public.v2.yaml")
if err != nil {
panic(err)
}
defer res.Body.Close()
data, err := io.ReadAll(res.Body)
if err != nil {
panic(err)
}
return data
}
// Index the digital ocean OpenAPI specification
func digitalOcean() {
// create a root node to unmarshal the spec into.
var rootNode yaml.Node
_ = yaml.Unmarshal(read(), &rootNode)
// create a new config that allows remote lookups.
indexConfig := index.CreateOpenAPIIndexConfig()
// we're going to check for circular references later, so we don't want to do it now.
indexConfig.AvoidCircularReferenceCheck = true
// create a custom logger (optional)
indexConfig.Logger = slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelError,
}))
// define the base URL for the remote filesystem.
location := "https://raw.githubusercontent.com/digitalocean/openapi/main/specification"
baseURL, _ := url.Parse(location)
// set the base URL for the remote filesystem in the config.
indexConfig.BaseURL = baseURL
// create a new rolodex
rolodex := index.NewRolodex(indexConfig)
// set the rolodex root node to the root node of the spec.
rolodex.SetRootNode(&rootNode)
// create a new remote fs and set the config for indexing.
remoteFS, _ := index.NewRemoteFSWithConfig(indexConfig)
// add remote filesystem
rolodex.AddRemoteFS(location, remoteFS)
// index the rolodex
indexedErr := rolodex.IndexTheRolodex()
if indexedErr != nil {
panic(indexedErr)
}
// get all the files!
files := remoteFS.GetFiles()
fileLen := len(files)
// check for circular references across the entire document.
rolodex.CheckForCircularReferences()
fmt.Printf("%d files found and %d errors reported. There were %d circular references found.\n",
fileLen, len(remoteFS.GetErrors()), len(rolodex.GetCaughtErrors()))
// extrtact the resolver from the root index.
resolver := rolodex.GetRootIndex().GetResolver()
// print out some interesting information discovered when visiting all the references.
fmt.Printf("%d references visited\n", resolver.GetReferenceVisited())
fmt.Printf("%d journeys taken\n", resolver.GetJourneysTaken())
fmt.Printf("%d index visits\n", resolver.GetIndexesVisited())
fmt.Printf("%d relatives seen\n", resolver.GetRelativesSeen())
}
This operation takes a few seconds,
libopenapi
is making thousands of HTTP requests to Github to get all the files.
After a few seconds, tt will spit out something like:
All the things available in the index
The list is considerable, so instead of re-listing everything here, head over to the go docs and see them there.