goten-bootstrap

What is goten-bootstrap executable?

Utility goten-bootstrap is a tool generating proto files from the specification file, also known as api-skeleton. In the goten repository, you can find the following files for the api-skeleton schema:

  • annotations/bootstrap.proto with JSON generated schema from it in schemas/api-skeleton.schema.json. This is the place you can modify input to bootstrap.

Runtime entry (main.go) can be found in the cmd/goten-bootstrap directory. It imports package in compiler/bootstrap directory, which pretty much contains the whole code for the goten-bootstrap utility. This is the place to explore if you want to modify generated protobuf files.

In main.go you can see two primary steps:

  1. Initialize the Service package object and pass it to the generator.

    During initialization, we validate input, populate defaults, and deduce all values.

  2. It then attaching all implicit API groups per each resource.

First look at the Generator object initialized with NewGenerator.

The relevant file is compiler/bootstrap/generate.go, which contains the ServiceGenerator struct with a single public method Generate. It takes parsed, validated, and initialized Service object (as described in the service.go file), then just generates all relevant files, with API groups and resources using regular for loops. Template protobuf files are all in tmpl subdirectory.

See the initTmpls function of ServiceGenerator: It collects all template files as giant strings (because those are strings…), parses them, and adds some set of functions that can be used within {{ }}. Those big strings are “render-able” objects, see https://pkg.go.dev/text/template for more details, but normally I find them self-explanatory. In those template strings, you see often:

  • {{functionName <ARGS>}}

    The word is some function. It may be built-in like define, range, if, or it may be a function we provided. From initTmpls you may see functions like uniqueResources, formatActionReplacement etc. Those are our functions. They may take arguments.

  • {{$variable}}

    This variable must be initialized somewhere using := operator. Those are Golang objects under the hood! You can access even sub-fields with dots ., or even call functions (but without arguments).

  • {{.}}

    This is a special kind of “current” active variable. In a given moment only one variable may be active. You may access its properties from regular variables like {{ $otherVar := .Field1.Field2 }}.

  • With {{ or }}

    you may see dashes: {{- or -}}. Their purpose is to remove whitespace (typically newline) behind or after them. It makes output nicer, but may occasionally render code non-compilable.

In generate.go, see svcgen.tmpl.ExecuteTemplate(file, tmplName, data). The first argument is the file writer object where the protobuf file will be generated. The second argument is a string, for example, resourceSchemaFile. The third argument is an active variable that can be accessed as {{.}}, which we mentioned. For example, you should see the following piece of code there:

if err := svcgen.genFile(
  "resourceSchemaFile",
  resource.Service.Proto.Package.CurrentVersion,
  fileName,
  resource,
  svcgen.override,
); err != nil {
    return fmt.Errorf("error generating resource file %s: %s", fileName, err)
}

The function genFile passes resourceSchemaFile as the second argument to tmpl.ExecuteTemplate, and object resource is passed as the last argument to tmpl.ExecuteTemplate. This resource object is of type Resource which you can see in the file resource.go.

How Golang templates are executed: Runtime will try to find the following piece of the template:

{{ define "resourceSchemaFile" }} ... {{ end }}

In this instance, you can find it in file tmpl/resource.tmpl.go, it starts with:

package tmpl

// language=gohtml
const ResourceTmplString = `
{{- define "resourceSchemaFile" -}}
{{- /*gotype: github.com/cloudwan/goten/annotations/bootstrap.Resource*/ -}}
{{- $resource := . }}

... stuff here....

{{ end }}

By convention, we try to provide what kind of object was passed as dot . under define, at least for main templates. Since range loops override dot value, to avoid losing resource reference (and for clarity), we often save current dot into a named variable.

Golang generates from the beginning of define till it reaches relevant {{ end }}. When it sees {{ template "..." <ARG> }}, it calls another define, and passes arg as the next “dot”. To pass multiple arguments, we often provide a dictionary using the dict function: {{ template "... name ..." dict <KEY1> <VALUE1> ... }}. Dict accepts N arguments and just makes a single object. You can see that we implemented this function in initTmpls! The generated final string is outputted to the specified file writer. This is how all protobuf files are generated.

Note that ServiceGenerator skips certain templates depending on the overrideFile argument. This is why resources and custom files for API groups are generated only once, to avoid overriding developer code. Perhaps in the future, we should be able to do some merging. That’s all regarding the generation part.

Also, very important is parsing the api-skeleton service package schema and wrapping it with the Service object as defined in the service.go file. Note that YAML in api-skeleton contains the definition of a Service not in the compiler/bootstrap/service.go, but annotations/bootstrap/bootstrap.pb.go file. See function ParseServiceSkeletonFiles in compiler/bootstrap/utils.go. It loads base bootstrap objects from yaml, and parses to according to the protobuf definition, but then we wrap them with the proper Service object. After we load all Service objects (including the next version and imported ones), we are calling the Init function of a Service. This is where we validate all input properly, and where we put default values missing in api-skeleton. The largest example is the function InitMainApi, which is called from service.go for each resource owned by the service. It adds our implicit APIs with full CRUD methods, it should be visible how all those “implicit” features play out there. We try also to validate as much input as possible. Any error messages must be wrapped with another error, so we return the full message at the top.