How the PMDT and DocFX work in Unity package docs
This document describes what the Package Manager Doc Tool (PMDT) does and how we use DocFX to generate package docs.
Roles
The roles the PMDT and DocFX play in doc generation are:
- PMDT -- prepares the package files for doc generation, setting up the files and configuration in a standardized way for DocFX. It also provides context needed to resolve links to other packages and Unity core docs. Finally, it provides a UI and options for generating the docs.
- DocFX -- Process the markdown files and C# source to create html files with a standard navigation frame, table of contents, and search. Uses a hierarchical page template system to allow customization of html.
PMDT
The PMDT is, itself, a package that runs in the context of the Unity Editor. Its most basic job is to take the markdown files, images, C# code (and various other files) from the file structure of a given package and create the build folder and configuration files needed to run DocFX. It then runs DocFX as a separate process, copying the generated files to an output folder when DocFX has finished.
Without the PMDT, every package would need to set up the DocFX file structure independently, copy the correct template files into their package repo, and create their own DocFX config and project files. This also means that any changes to the templates, config, or features we use would need to be integrated by every package before they would be effective.
The PMDT adds UI controls to the Package Manager window (using the Package Manager API) so that a user can generate docs for any installed package with no other setup required. In addition, the PMDT defines a batch API (using standard Unity API methods) so that package docs can be generated by invoking the Unity Editor from the command line (as long as the PMDT is already installed in the invoked project).
The PMDT expects the following files to exist, relative to the root of the package (the folder that contains the package's manifest file, package.json):
- Documentation~ (folder) -- the markdown and images used to create the manual section, as well as config files to set per-project options for DocFX.
- TableOfContents.md -- A markdown list used to create the manual ToC. (This file, based on the format used by the core Unity manual, is translated into the different format used by DocFX. It is also possible to use the DocFX "native" format directly.) Technically, this file is optional; it isn't used if the manual doesn't contain a file named index.md.
- index.md -- the "landing" page for the package. Always the first page in the ToC. Technically, a file with this name is optional. If it doesn't exist the first markdown file returned by the file system is used and no ToC is displayed on the page.
- api_index.md -- (optional) the landing page for the Script Reference section. A minimal default page is used if this file does not exist.
- additional markdown files -- (optional) Any number of markdown files used in the manual. All files are converted to html even if they do not appear in the ToC.
- projectMetadata.json -- (optional) defines a limited set of options for the current package, allowing a package to override default behavior. See PMDT documentation.
- config.json -- (optional) define C# preprocessor symbols for the package. Any symbols in this file are defined when compiling and generating the docs. (This file should be consolidated with projectMetadata.json, there's no need for two config files.)
- images (subfolder) -- images and other files in this folder are copied to the manual/images folder in the output. Required if the package docs use images.
- snippets (subfolder) -- (optional) contains files that are included in other markdown files. Any files in this folder can be referenced using the file inclusion mechanism, but are not converted into html.
- filter.yml -- (optional) a YAML-format file defining C# types to exclude from the script reference. (By default, all publicly accessible types and members are included. This file is also used by the Package Validation Find Missing Docs tool.)
- other files -- other files in this folder can have undefined behavior.
- C# source code -- All C# source files in the package are found and used unless they are inside a folder ignored by Unity rules (such as in folders ending with a tilde character).
- CHANGELOG.md -- Used to create the html file for the package change log.
- LICENSE.md -- Used to create the html for the package license.
When the generate docs function is invoked (via the UI or batch API), the PMDT does the following:
- Cleanup old build and output folders
- Create a config object
- Create a ToC object that defines the links in the top nav bar
- Install a local copy of Mono (if needed on Mac or Linux)
- Copy DocFX template files from the PMDT package to the build folder
- Copy package files:
- Copy the project's Library/PackageCache folder to the build folder
- Copy the project's compiled assemblies from Library/ScriptAssemblies to the build folder
- Copy packages that aren't in the project's package cache to the build folder
- Add a manual section, manual files, and images folder to config
- Handle ToC logic, add to config if used
- Move optional files from Documentation~ to correct locations in build folder
- Handle changelog and license sections
- Create a C# project file referencing all the package C# files and referenced assemblies (including the Unity assemblies from the running version of the Unity Editor)
- Add xrefmaps field to the DocFX config in order to resolve links to other packages and the Unity core docs
- Apply strings in ui.json to the web template variables. (This supports localization for UI strings used on the doc site.)
- Enable optional DocFX plug-ins, of which there is currently one, memberpages.
- Write the DocFX config and top-level navbar toc to disk
- If a cached copy of DocFX doesn't exist, copy it from the PMDT folder to the parent of the build folder (only one copy is created on a computer)
- Launch DocFX passing in the newly created DocFX config file
- Wait for completion
- Copy generated site files to output (or report error on failure)
- Delete build folder (unless Debug Doc Build option was enabled).
A few things not discussed in this list: the version selector, the language selector, and the @latest page which determines the "current" version of a package.
The less trivial aspects of the above process include:
- Creating the C# project needed by DocFX
- Resolving the correct xrefmap files
- Localization strings via ui.json and template variables
- Referencing manual files from inside the Packages directory
Create a C# project file
DocFX requires a "proper" C# project file that references the package source code files and the correct compiled code libraries in order to generate the Script Reference with links to types defined outside of the current package. This allows, for example, DocFX to create a link from a graphics package API using Texture to the core Unity Script Reference. Without a proper project file, DocFX, just puts in the bare type name with no link and no namespace. Readers have no way to know where a given type came from and no easy way to navigate to more docs about that type.
The PMDT relies on the running Unity Editor to provide the context for defining which version of the Unity libraries to reference in the project. For example, if the running Editor is version 2020.3, then only types available in 2020.3 can be linked. The destination of the links is determined by the xrefmap resolution step.
Currently the logic used to create the project file simply gathers all the available C# source files and includes them in the file. The drawback to this is that the compilation environment outside of Unity is not necessarily the same as it is inside Unity, which can lead to mismatches and other problems. Some values, like the.NET framework are currently hardcoded and might eventually not be the correct values to successfully generate docs from otherwise legal Unity code.
An alternative way to set up the project for building the script reference might be to generate XML doc files with the Unity compiler. (Generating an XML doc file is a C# compiler feature that copies all the XML doc files into a separate XML file that it places alongside the library DLL files.) Currently, there isn't an easy way to generate these files within Unity without altering the files with a Project's Asset folder, but in upcoming versions of Unity, this might be possible. If so, DocFX can generate the script reference directly from the DLLs and XML files rather than needing to parse the C# source itself. This might be the best option if we need to update this area of the PMDT in the future.
Resolve xrefmaps
Xrefmaps are DocFX-generated files that map UIDs to URLs. Whenever DocFX generates a set of docs, it also creates an xrefmap.yml file and places it in the root of the output directory. Assuming that doc set is copied to a server and hosted on the web, you can add the URL to that xrefmap file in the config file of another DocFX project. When generating the docs for that other project, DocFX will look up the UIDs for any API types or xref-style links in that xrefmap file. If the UID exists, DocFX calculates the final URL for the link from the URL in the xrefmap file along with the base URL of the file itself. For example, if a UID exists in "https://www.unity.com/foo/xrefmap.yml" with a relative URL, "bar.html", then the computed URL is: ""https://www.unity.com/foo/bar.html". If the xrefmap contains absolute URLs, then the absolute URL is used without modification.
The PMDT adds xrefmaps to the config file before running DocFX. The map files it adds
- Dependent packages, using the version of the dependent package installed in the current project
- The core Unity docs, using the version of the running Editor
- Additional packages identified in the projectMetadata.config file (which can be either a specific version or the latest version at the time the docs are generated)
- Universal xrefmaps, of which there is one, langwordMapping.yml, which maps C# language keywords to their documentation on Microsoft's site.
DocFX supports using an xref service as an alternate way to resolve xref UIDs. Microsoft provides one for C# types, which we use for the base C# API (like System.String).
A large benefit of using xrefmaps in our highly-versioned package doc sets is that the writers never have to update version numbers used in the links.
There are a few challenges with using xrefmaps, not all of which we have solved. These include:
- The external docs to be referenced must already be publicly accessible on our web site. This fact can be a problem when you have to generate and release a set of related packages. Today, each package would need to be generated and published sequentially and in the correct order for the cross-references to resolve to the correct version. (Currently we fall back to the latest published version on the website if the expected version isn't available.) This problem could be fixed by post processing the local generated xrefmap files to contain the expected absolute URLs.
- Links to non-dependent packages. For dependent packages, the PMDT relies on the dependency information to choose which version to link to. For non-dependent packages, version information doesn't exist. You have the choice of specifying the version in a config file or using whatever happens to be the latest version at the time the docs are generated. (In general, this process is still better than using absolute URLs to a versioned web site because you can update the version for a non-dependent package in one place rather than scattered throughout the text and code base. It also works in cases where an external package is referenced without being a dependency through the use of conditional compilation.
UI localization
The UI localization feature relies on the DocFX template system and variables. The PMDT reads the UI.json file and applies the values to the correct template variables, allowing strings to be localized.
Referencing manual files from inside the Packages directory
Currently, we reference manual files from inside the Packages folder that is copied to the build directory rather than copying them into the manual folder off the root of the build folder. While this has the benefit of maintaining the relative path between the package source files and the manual files, it also has some drawbacks. In some cases, DocFX falsely reports broken links because of this set up. However, changing the set up would also require some packages to rewrite different relative links for things like file and code example inclusion.
This is an issue that should be considered for future rewrites of the PMDT or might affect migration away from the PMDT.
DocFX
The role of DocFX is to generate the doc html files from markdown files and C# XML comments. (It can be configured for other programming languages and REST APIs as well.)
The main parts of DocFX that we rely on include:
- A markdown engine
- A code parser (that extracts XML comments and type relationships)
- An html page template system
- A UID to URL resolution system
- A filter mechanism for excluding types and members that would normally appear in the docs
- A file inclusion mechanism for both markdown and XML doc
- The ability to use markdown syntax inside XML doc comments
Markdown engine
DocFX provides two choices for markdown engine. The first is called "DocFX-flavored" markdown (DFM), which is essentially the same as Github-flavored markdown with a couple of additions. The other choice is the Markdig engine. Recent versions of DocFX have changed to use Markdig as the default, but Unity has not switched.
DocFX also supports customizing DFM or even using a custom markdown engine. These options require a plug-in or forking DocFX.
Code parser
The code parser relies on MSBuild. DocFX extracts type information and the XML doc comments from the source code and writes them to a set of YAML-formatted intermediate files. These files are processed in a second pass to create HTM files and a table of contents document.
We use the C# files added to a C# project as the input to DocFX. DocFX does support alternate inputs, such as the XML doc files that can be produced by the C# compiler.
HTML template system
DocFX uses a template system to determine HTML layout. The system is hierarchical in that there is a default set of template files that you can selectively override. For example, you can specify multiple template sets in the configuration file and DocFX applies them in order. The files in each successive set overwrite any files in the earlier template.
Part of this template system is the ability to define variables that you can set at doc generation time. The UI localization system uses such variables. DocFX also uses it for a couple of purposes like enabling JavaScript-level features.
Some of our package doc customizations take place at the template level. For example, the version selector is added to the project as a referenced JavaScript file.
UID to URL resolution system
A key feature of DocFX is its use of UIDs as a mechanism for resolving cross references. As discussed above, this feature solves some (but not all) of our problems with making links between the docs of different packages and the Unity core manual.
UIDs for API types and members are created automatically and are based on the comment ID string produced by the C# compiler for XML docs. UIDs for markdown files must be assigned manually using a yaml header at the top of the file.
In order for DocFX to create a valid UID for APIs, its parser must be able to resolve the type. This means that the C# project we create must either contain the source file defining the type or a proper reference to the external library defining the type. Early versions of the PMDT did not do this and links between types in different packages or to the UnityEngine or Editor types were impossible.
In XML docs, types used as return types, parameter types, inheritance, etc, are resolved using this system as are cross references using <see cref="type"/> elements.
In addition to explicit xrefmap files, which must be downloaded and searched by DocFX (in most cases), DocFX also supports the use of xref services. When using such a service, DocFX makes a REST call to the service using the UID and the service returns the URL, if the service has a reference to the UID. We use Microsoft's service for .NET types and language classes. It is possible that we might be able to solve some of our cross reference problems using such a service, but that would need some investigation.
Xrefmaps for the core Unity docs
Note that this system can be used with the Unity core manual and script reference because we generate our own xrefmap files for them. Currently, this is done with a Python script that operates on the UnityEngine.xml, UnityEditor.xml files shipped with the Unity Editor and on the online ToC JSON file on the public web site. The xrefmap files created with Python are part of the PMDT package and the generation code selects the one matching the current Editor when docs are generated.
The xrefmaps files are updated irregularly and manually and drift out of date fairly quickly. In Unity 2023.1+, these files are generated by the core manual and script ref generation code rather than a separate Python script.
API filtering mechanism
When APIs must be public but are not intended for public use, often the best way to handle this is by not including them in the generated docs. DocFX uses a YAML format as a way to define what should be excluded. This same format was adopted by the FindMissingDocs tool used by the Package Validation Suite. Both tools read the same filter.yml file that defines the exclusions.
File inclusion
File inclusion isn't specific to DocFX, but the syntax probably is unique, especially for the code example file inclusion. We use file inclusion for two main reasons: a limited form of "single-sourcing" in which the same block of content is used in multiple documents by way of file inclusion; we also use file inclusion for code samples, where a C# file or a region within one are included in either a markdown document or an API example element. A large benefit of including code files in this way is that we can place the example files in a place where Unity compiles them. If the API changes, then any compile errors in the code samples can be discovered immediately and fixed.
Markdown in XML doc
Normally, you must use full XML syntax when writing XML doc comments, including para
tags around every paragraph. This is very verbose and error prone. DocFX relaxes that by supporting markdown inside the doc comments. You must still use the top-level block tags like summary
and remarks
but everything within them can be written in a more natural, markdown style.
Other DocFX features
Override files
DocFX allows you to override the docs of an API member using a markdown file. We don't currently use this feature, but it might be useful in some cases. For example, if we wanted to consolidate the core Script Reference generation, it might be easiest to convert the mem.xml files into override files.
Plug-ins
We use the Memberpages plug-in, which puts the documentation for the members of an API type on separate pages. By default, all of the docs for a type appear on a single page, but a package can opt into using member pages.
Extensibility
DocFX supports various levels of extensibility. You can write and use plug-ins, create a parser for other programming languages beyond C#, add a post-processor (for example, the search indexer used by DocFX uses a post processor to create its index file).
We don't currently employ this feature beyond the use of plug-ins. There are things we might want to do in the future, for example, if we were to use DocFX for the core script reference, we could access the mem.xml files directly, rather than translating them to a different format.
Developing PMDT features
This section provides some information and tips about how to add features to the PMDT.
Add per project metadata
The PMDT supports adding arbitrary variables to the DocFX config, which can be used in the page layout templates.
To add a new value:
In DocFXConfig.cs, add a statement that adding the new variable to the global metadata section with the defaultConfig.build.globalMetadata.Add("name", "default_value")
function.
For example:
defaultConfig.build.globalMetadata.Add("_appLogoPath", "logo.svg");
defaultConfig.build.globalMetadata.Add("_disableToc", false);
defaultConfig.build.globalMetadata.Add("_imageZoomThreshold", 1200);
This establishes adds the property to the JSON object and sets a default value. Use the appropriate JSON data type.
Package developers can add the property name to the projectMetadata.json
file in their package to set a different value.
In templates, you can refer to these variables using the name parameter and moustache template syntax. For example:
{{#_imageZoomThreshold}} let imageZoomThreshold = {{_imageZoomThreshold}}; {{/_imageZoomThreshold}}
This example template statement assigns the value of _imageZoomThreshold
to a global JavaScript variable, allowing the value to be accessed by JavaScript on any page the template is used for.
Tip
The {{#name}}
and {{/name}}
parts are simple conditionals. The bit in the middle is only added to the template if name
exists. Refer to moustache template docs on the web for more information.
Another way is to add a meta property to the head.tmpl.partial
template file:
{{#_imageZoomThreshold}}<meta property="unity:imageZoomThreshold" content="{{_imageZoomThreshold}}">{{/_imageZoomThreshold}}
[TIP] Add
unity:
as a prefix to the property name to indicate it was added by the PMDT.
You can the access the assigned value in any JavaScript with a JQuery statement like:
const threshold = $("meta[property='unity:image-zoom-threshold']").attr("content");
Any scripts you add to templates should go in styles/PMDT.js
unless you have a good reason to put them elsewhere. This makes it easier to update the templates in the future by keeping PMDT features and scripts separate from DocFX ones.