1
0
mirror of https://github.com/lvgl/lvgl.git synced 2025-01-14 06:42:58 +08:00
lvgl/docs/build.py
2024-12-19 13:47:46 -05:00

666 lines
28 KiB
Python
Executable File

#!/usr/bin/env python3
""" Generate LVGL documentation using Doxygen and Sphinx.
The first version of this file (Apr 2021) discovered the name of
the current branch (e.g. 'master', 'release/v8.4', etc.) to support
different versions of the documentation by establishing the base URL
(used in `conf.py` and in [Edit on GitHub] links), and then ran:
- Doxygen (to generate LVGL API XML), then
- Sphinx
to generate the LVGL document tree. Internally, Sphinx uses `breathe`
(a Sphinx extension) to provide a bridge between Doxygen XML output and
Sphinx documentation. It also supported a command-line option `clean`
to remove generated files before starting (eliminates orphan files,
for docs that have moved or changed).
Since then its duties have grown to include:
- Using environment variables to convey branch names to several more
places where they are used in the docs-generating process (instead
of re-writing writing `conf.py` and a `header.rst` each time docs
were generated). These are documented where they generated below.
- Supporting additional command-line options.
- Generating a temporary `./docs/lv_conf.h` for Doxygen to use
(config_builder.py).
- Supporting multiple execution platforms (which then required tokenizing
Doxygen's INPUT path in `Doxyfile` and re-writing portions that used
`sed` to generate input or modify files).
- Adding translation and API links (requiring generating docs in a
temporary directory so that the links could be programmatically
added to each document before Sphinx was run).
- Generating EXAMPLES page + sub-examples where applicable to individual
documents, e.g. to widget-, style-, layout-pages, etc.
- Building PDF via latex (when working).
Command-Line Arguments
----------------------
Command-line arguments have been broken down to give the user the
ability to control each individual major variation in behavior of
this script. These were added to speed up long doc-development
tasks by shortening the turn-around time between doc modification
and seeing the final .html results in a local development environment.
Finally, this script can now be used in a way such that Sphinx will
only modify changed documents, and reduce an average ~22-minute
run time to a run time that is workable for rapidly repeating doc
generation to see Sphinx formatting results quickly.
Normal Usage
------------
This is the way this script is used for normal (full) docs generation.
$ python build.py skip_latex
Docs-Dev Initial Docs Generation Usage
--------------------------------------
1. Set `LVGL_FIXED_TEMP_DIR` environment variable to path to
the temporary directory you will use over and over during
document editing, without trailing directory separator.
Initially directory should not exist.
2. $ python build.py skip_latex preserve fixed_tmp_dir
This takes typically ~22 minutes.
Docs-Dev Update-Only Generation Usage
-------------------------------------
After the above has been run through once, you can thereafter
run the following until docs-development task is complete.
$ python build.py skip_latex docs_dev update
Generation time depends on the number of `.rst` files that
have been updated:
+--------------+------------+---------------------------------+
| Docs Changed | Time | Typical Time to Browser Refresh |
+==============+============+=================================+
| 0 | 6 seconds | n/a |
+--------------+------------+---------------------------------+
| 1 | 19 seconds | 12 seconds |
+--------------+------------+---------------------------------+
| 5 | 28 seconds | 21 seconds |
+--------------+------------+---------------------------------+
| 20 | 59 seconds | 52 seconds |
+--------------+------------+---------------------------------+
Sphinx Doc-Regeneration Criteria
--------------------------------
Sphinx uses the following to determine what documents get updated:
- source-doc modification date
- Change the modification date and `sphinx-build` will re-build it.
- full (absolute) path to the source document, including its file name
- Change the path or filename and `sphinx-build` will re-build it.
- whether the -E option is on the `sphinx-build` command line
- -E forces `sphinx-build` to do a full re-build.
Argument Descriptions
---------------------
- skip_latex
The meaning of this argument has not changed: it simply skips
attempting to generate Latex and subsequent PDF generation.
- skip_api
Skips generating API pages (this saves about 70% of build time).
This is intended to be used only during doc development to speed up
turn-around time between doc modifications and seeing final results.
- no_fresh_env
Excludes -E command-line argument to `sphinx-build`, which, when present,
forces it to generate a whole new environment (memory of what was built
previously, forcing a full rebuild). "no_fresh_env" enables a rebuild
of only docs that got updated -- Sphinx's default behavior.
- preserve (previously "develop")
Leaves temporary directory intact for docs development purposes.
- fixed_tmp_dir
If (fixed_tmp_dir and 'LVGL_FIXED_TEMP_DIR' in `os.environ`),
then this script uses the value of that that environment variable
to populate `temp_directory` instead of the normal (randomly-named)
temporary directory. This is important when getting `sphinx-build`
to ONLY rebuild updated documents, since changing the directory
from which they are generated (normally the randomly-named temp
dir) will force Sphinx to do a full-rebuild because it remembers
the doc paths from which the build was last performed.
- skip_trans
Skips adding translation links. This allows direct copying of
`.rst` files to `temp_directory` when they are updated to save time
during re-build. Final build must not include this option so that
the translation links are added at the top of each intended page.
- no_copy
Skips copying ./docs/ directory tree to `temp_directory`.
This is only honored if:
- fixed_tmp_dir == True, and
- the doc files were previously copied to the temporary directory
and thus are already present there.
- docs_dev
This is a command-line shortcut to combining these command-line args:
- no_fresh_env
- preserve
- fixed_tmp_dir
- no_copy
- update
When no_copy is active, check modification dates on `.rst` files
and re-copy the updated `./docs/` files to the temporary directory
that have later modification dates, thus updating what Sphinx uses
as input.
Warning: this wipes out translation links and API-page links that
were added in the first pass, so should only be used for doc
development -- not for final doc generation.
"""
# ****************************************************************************
# IMPORTANT: If you are getting a PDF-lexer error for an example, check
# for extra lines at the end of the file. Only a single empty line
# is allowed!!! Ask me how long it took me to figure this out.
# ****************************************************************************
def run():
# Python Library Imports
import sys
import os
import re
import subprocess
import shutil
import tempfile
import dirsync
from datetime import datetime
# LVGL Custom Imports
import example_list as ex
import doc_builder
import config_builder
import add_translation
# ---------------------------------------------------------------------
# Start.
# ---------------------------------------------------------------------
t1 = datetime.now()
print('Current time: ' + str(t1))
# ---------------------------------------------------------------------
# Process args.
#
# With argument `docs_dev`, Sphinx will generate docs from a fixed
# temporary directory that can be then used later using the same
# command line to get Sphinx to ONLY rebuild changed documents.
# This saves a huge amount of time during long document projects.
# ---------------------------------------------------------------------
# Set defaults.
clean = False
skip_latex = False
skip_api = False
fresh_sphinx_env = True
preserve = False
fixed_tmp_dir = False
skip_trans = False
no_copy = False
docs_dev = False
update = False
args = sys.argv[1:]
for arg in args:
# We use chained `if-elif-else` instead of `match` for those on Linux
# systems that will not have the required version 3.10 of Python yet.
if arg == "clean":
clean = True
elif arg == "skip_latex":
skip_latex = True
elif arg == 'skip_api':
skip_api = True
elif arg == 'no_fresh_env':
fresh_sphinx_env = False
elif arg == 'preserve':
preserve = True
elif arg == 'fixed_tmp_dir':
fixed_tmp_dir = True
elif arg == 'skip_trans':
skip_trans = True
elif arg == 'no_copy':
no_copy = True
elif arg == 'docs_dev':
docs_dev = True
elif arg == 'update':
update = True
else:
print(f'Argument [{arg}] not recognized.')
exit(1)
# Arg ramifications:
# docs_dev implies no_fresh_env, preserve, fixed_tmp_dir, and no_copy.
if docs_dev:
fresh_sphinx_env = False
preserve = True
fixed_tmp_dir = True
no_copy = True
# ---------------------------------------------------------------------
# Due to the modifications that take place to the documentation files
# when the documentation builds it is better to copy the source files to a
# temporary folder and modify the copies. Not setting it up this way makes it
# a real headache when making alterations that need to be committed as the
# alterations trigger the files as changed. Also, this keeps maintenance
# effort to a minimum as adding a new language translation only needs to be
# done in 2 places (add_translation.py and ./docs/_ext/link_roles.py) rather
# than once for each .rst file.
#
# The html and PDF output locations are going to remain the same as they were.
# it's just the source documentation files that are going to be copied.
# ---------------------------------------------------------------------
if fixed_tmp_dir and 'LVGL_FIXED_TEMP_DIR' in os.environ:
temp_directory = os.environ['LVGL_FIXED_TEMP_DIR']
else:
temp_directory = tempfile.mkdtemp(suffix='.lvgl_docs')
print(f'Using temp directory: [{temp_directory}]')
# ---------------------------------------------------------------------
# Set up paths.
# ---------------------------------------------------------------------
base_path = os.path.abspath(os.path.dirname(__file__))
project_path = os.path.abspath(os.path.join(base_path, '..'))
examples_path = os.path.join(project_path, 'examples')
lvgl_src_path = os.path.join(project_path, 'src')
latex_output_path = os.path.join(temp_directory, 'out_latex')
pdf_src_file = os.path.join(latex_output_path, 'LVGL.pdf')
pdf_dst_file = os.path.join(temp_directory, 'LVGL.pdf')
html_src_path = temp_directory
html_dst_path = os.path.join(project_path, 'out_html')
# ---------------------------------------------------------------------
# Change to script directory for consistency.
# ---------------------------------------------------------------------
os.chdir(base_path)
# ---------------------------------------------------------------------
# Provide a way to run an external command and abort build on error.
# ---------------------------------------------------------------------
def cmd(s, start_dir=None):
if start_dir is None:
start_dir = os.getcwd()
saved_dir = os.getcwd()
os.chdir(start_dir)
print("")
print(s)
print("-------------------------------------")
result = os.system(s)
os.chdir(saved_dir)
if result != 0:
print("Exiting build due to previous error.")
sys.exit(result)
# ---------------------------------------------------------------------
# Populate LVGL_URLPATH and LVGL_GITCOMMIT environment variables:
# - LVGL_URLPATH <= 'master' or '8.4' '9.2' etc.
# - LVGL_GITCOMMIT <= same (see 03-Oct-2024 note below).
#
# These supply input later in the doc-generation process as follows:
#
# LVGL_URLPATH is used by:
# - `conf.py` to build `html_baseurl` for Sphinx for
# - generated index
# - generated search window
# - establishing canonical page for search engines
# - `link_roles.py` to generate translation links
# - `doc_builder.py` to generate links to API pages
#
# LVGL_GITCOMMIT is used by:
# - `conf.py` => html_context['github_version'] for
# Sphinx Read-the-Docs theme to add to [Edit on GitHub] links
# - `conf.py` => repo_commit_hash for generated EXAMPLES pages for:
# - [View on GitHub] buttons (view C code examples)
# - [View on GitHub] buttons (view Python code examples)
# ---------------------------------------------------------------------
# 03-Oct-2024: Gabor requested LVGL_GITCOMMIT be changed to a branch
# name since that will always be current, and it will fix a large
# number of broken links on the docs website, since commits that
# generated docs can sometimes go away. This gets used in:
# - [Edit on GitHub] links in doc pages (via Sphinx theme), and
# - [View on GitHub] links in example pages (via `example_list.py`
# and `lv_example.py`).
# Original code:
# status, br = subprocess.getstatusoutput("git branch --show-current")
# _, gitcommit = subprocess.getstatusoutput("git rev-parse HEAD")
# br = re.sub(r'\* ', '', br)
# 're' was previously used to remove leading '* ' from current branch
# string when we were parsing output from bare `git branch` output.
# This is no longer needed with `--show-current` option now used.
# ---------------------------------------------------------------------
status, branch = subprocess.getstatusoutput("git branch --show-current")
# If above failed (i.e. `branch` not valid), default to 'master'.
if status != 0:
branch = 'master'
elif branch == 'master':
# Expected in most cases. Nothing to change.
pass
else:
# `branch` is valid. Capture release version if in a 'release/' branch.
if branch.startswith('release/'):
branch = branch[8:]
else:
# Default to 'master'.
branch = 'master'
os.environ['LVGL_URLPATH'] = branch
os.environ['LVGL_GITCOMMIT'] = branch
# ---------------------------------------------------------------------
# Start doc-build process.
# ---------------------------------------------------------------------
print("")
print("****************")
print("Building")
print("****************")
# Remove all previous output files if 'clean' on command line.
if clean:
print('Removing previous output files...')
# The below commented-out code below is being preserved
# for docs-generation development purposes.
# api_path = os.path.join(temp_directory, 'API')
# xml_path = os.path.join(temp_directory, 'xml')
# doxy_path = os.path.join(temp_directory, 'doxygen_html')
# if os.path.exists(api_path):
# shutil.rmtree(api_path)
# lang = 'en'
# if os.path.exists(lang):
# shutil.rmtree(lang)
if os.path.exists(html_dst_path):
shutil.rmtree(html_dst_path)
# if os.path.exists(xml_path):
# shutil.rmtree(xml_path)
#
# if os.path.exists(doxy_path):
# shutil.rmtree(doxy_path)
# os.mkdir(api_path)
# os.mkdir(lang)
# ---------------------------------------------------------------------
# Build local lv_conf.h from lv_conf_template.h for this build only.
# ---------------------------------------------------------------------
config_builder.run()
# ---------------------------------------------------------------------
# Provide answer to question: Can we have reasonable confidence that
# the contents of `temp_directory` already exists?
# ---------------------------------------------------------------------
def temp_dir_contents_exists():
result = False
c1 = os.path.exists(temp_directory)
if c1:
temp_path = os.path.join(temp_directory, 'CHANGELOG.rst')
c2 = os.path.exists(temp_path)
temp_path = os.path.join(temp_directory, 'CODING_STYLE.rst')
c3 = os.path.exists(temp_path)
temp_path = os.path.join(temp_directory, 'CONTRIBUTING.rst')
c4 = os.path.exists(temp_path)
temp_path = os.path.join(temp_directory, '_ext')
c5 = os.path.exists(temp_path)
temp_path = os.path.join(temp_directory, '_static')
c6 = os.path.exists(temp_path)
temp_path = os.path.join(temp_directory, 'details')
c7 = os.path.exists(temp_path)
temp_path = os.path.join(temp_directory, 'intro')
c8 = os.path.exists(temp_path)
temp_path = os.path.join(temp_directory, 'examples')
c9 = os.path.exists(temp_path)
result = c2 and c3 and c4 and c5 and c6 and c7 and c8 and c9
return result
# ---------------------------------------------------------------------
# Copy files to 'temp_directory' where they will be edited (translation
# link and API links) before being used to generate new docs.
# ---------------------------------------------------------------------
doc_files_copied = False
if no_copy and fixed_tmp_dir and temp_dir_contents_exists():
if update:
exclude_list = ['lv_conf.h']
options = {
'verbose': True,
'create': True,
'exclude': exclude_list
}
dirsync.sync('.', temp_directory, 'update', **options)
else:
print("Skipping copying ./docs/ directory as requested.")
else:
shutil.copytree('.', temp_directory, dirs_exist_ok=True)
shutil.copytree(examples_path, os.path.join(temp_directory, 'examples'), dirs_exist_ok=True)
doc_files_copied = True
# ---------------------------------------------------------------------
# Replace tokens in Doxyfile in 'temp_directory' with data from this run.
# ---------------------------------------------------------------------
if doc_files_copied:
with open(os.path.join(temp_directory, 'Doxyfile'), 'rb') as f:
data = f.read().decode('utf-8')
data = data.replace('#*#*LV_CONF_PATH*#*#', os.path.join(base_path, 'lv_conf.h'))
data = data.replace('*#*#SRC#*#*', '"{0}"'.format(lvgl_src_path))
with open(os.path.join(temp_directory, 'Doxyfile'), 'wb') as f:
f.write(data.encode('utf-8'))
# -----------------------------------------------------------------
# Generate examples pages. Include sub-pages pages that get included
# in individual documents where applicable.
# -----------------------------------------------------------------
print("Generating examples...")
ex.exec(temp_directory)
# -----------------------------------------------------------------
# Add translation links.
# -----------------------------------------------------------------
if skip_trans:
print("Skipping translation links as requested.")
else:
print("Adding translation links...")
add_translation.exec(temp_directory)
# ---------------------------------------------------------------------
# Generate API pages and links thereto.
# ---------------------------------------------------------------------
if skip_api:
print("Skipping API generation as requested.")
else:
print("Running Doxygen...")
cmd('doxygen Doxyfile', temp_directory)
doc_builder.EMIT_WARNINGS = False
# Create .RST files for API pages.
doc_builder.run(
project_path,
temp_directory,
os.path.join(temp_directory, 'intro'),
os.path.join(temp_directory, 'intro', 'add-lvgl-to-your-project'),
os.path.join(temp_directory, 'details'),
os.path.join(temp_directory, 'details', 'base-widget'),
os.path.join(temp_directory, 'details', 'base-widget', 'layouts'),
os.path.join(temp_directory, 'details', 'base-widget', 'styles'),
os.path.join(temp_directory, 'details', 'debugging'),
os.path.join(temp_directory, 'details', 'integration'),
os.path.join(temp_directory, 'details', 'integration', 'bindings'),
os.path.join(temp_directory, 'details', 'integration', 'building'),
os.path.join(temp_directory, 'details', 'integration', 'chip'),
os.path.join(temp_directory, 'details', 'integration', 'driver'),
os.path.join(temp_directory, 'details', 'integration', 'driver', 'display'),
os.path.join(temp_directory, 'details', 'integration', 'driver', 'touchpad'),
os.path.join(temp_directory, 'details', 'integration', 'framework'),
os.path.join(temp_directory, 'details', 'integration', 'ide'),
os.path.join(temp_directory, 'details', 'integration', 'os'),
os.path.join(temp_directory, 'details', 'integration', 'os', 'yocto'),
os.path.join(temp_directory, 'details', 'integration', 'renderers'),
os.path.join(temp_directory, 'details', 'libs'),
os.path.join(temp_directory, 'details', 'main-components'),
os.path.join(temp_directory, 'details', 'other-components'),
os.path.join(temp_directory, 'details', 'widgets')
)
print('Reading Doxygen output...')
# ---------------------------------------------------------------------
# BUILD PDF
# ---------------------------------------------------------------------
if skip_latex:
print("Skipping latex build as requested.")
else:
# Remove PDF link so PDF does not have a link to itself.
index_path = os.path.join(temp_directory, 'index.rst')
with open(index_path, 'rb') as f:
index_data = f.read().decode('utf-8')
# Support both Windows and Linux platforms with `os.linesep`.
pdf_link_ref_str = 'PDF version: :download:`LVGL.pdf <LVGL.pdf>`' + os.linesep
if pdf_link_ref_str in index_data:
index_data = index_data.replace(pdf_link_ref_str, '')
with open(index_path, 'wb') as f:
f.write(index_data.encode('utf-8'))
# Silly workaround to include the more or less correct
# PDF download link in the PDF
# cmd("cp -f " + lang +"/latex/LVGL.pdf LVGL.pdf | true")
src = temp_directory
dst = latex_output_path
cpu = os.cpu_count()
cmd_line = f'sphinx-build -b latex "{src}" "{dst}" -j {cpu}'
cmd(cmd_line)
# Generate PDF.
cmd_line = 'latexmk -pdf "LVGL.tex"'
cmd(cmd_line, latex_output_path)
# Copy the result PDF to the main directory to make
# it available for the HTML build.
shutil.copyfile(pdf_src_file, pdf_dst_file)
# Add PDF link back in so HTML build will have it.
index_data = pdf_link_ref_str + index_data
with open(index_path, 'wb') as f:
f.write(index_data.encode('utf-8'))
# ---------------------------------------------------------------------
# BUILD HTML
# ---------------------------------------------------------------------
# This version of get_version() works correctly under both Linux and Windows.
# Updated to be resilient to changes in `lv_version.h` compliant with C macro syntax.
def get_version():
path = os.path.join(project_path, 'lv_version.h')
major = ''
minor = ''
with open(path, 'r') as file:
major_re = re.compile(r'define\s+LVGL_VERSION_MAJOR\s+(\d+)')
minor_re = re.compile(r'define\s+LVGL_VERSION_MINOR\s+(\d+)')
for line in file.readlines():
# Skip if line not long enough to match.
if len(line) < 28:
continue
match = major_re.search(line)
if match is not None:
major = match[1]
else:
match = minor_re.search(line)
if match is not None:
minor = match[1]
# Exit early if we have both values.
if len(major) > 0 and len(minor) > 0:
break
return f'{major}.{minor}'
# Note: While it can be done (e.g. if one needs to set a stop point
# in Sphinx code for development purposes), it is NOT a good idea to
# run Sphinx from script as
# from sphinx.cmd.build import main as sphinx_build
# sphinx_args = [...]
# sphinx_build(sphinx_args)
# because it takes ~10X longer to run than `sphinx_build` executable.
# Literally > 3 hours.
# '-E' option forces Sphinx to rebuild its environment so all docs are
# fully regenerated, even if not changed.
# Note: Sphinx runs in ./docs/, but uses `temp_directory` for input.
if fresh_sphinx_env:
print("Regenerating all files...")
env_opt = '-E'
else:
print("Regenerating only updated files...")
env_opt = ''
ver = get_version()
src = html_src_path
dst = html_dst_path
cpu = os.cpu_count()
cmd_line = f'sphinx-build -b html "{src}" "{dst}" -D version="{ver}" {env_opt} -j {cpu}'
t2 = datetime.now()
print('Current time: ' + str(t2))
cmd(cmd_line)
t3 = datetime.now()
print('Current time: ' + str(t3))
print('Sphinx run time: ' + str(t3 - t2))
# ---------------------------------------------------------------------
# Cleanup.
# ---------------------------------------------------------------------
if preserve:
print('Temp directory: ', temp_directory)
else:
print('Removing temporary files...', temp_directory)
if os.path.exists(temp_directory):
shutil.rmtree(temp_directory)
# ---------------------------------------------------------------------
# Remove temporary `lv_conf.h` created for this build.
# ---------------------------------------------------------------------
config_builder.cleanup()
# ---------------------------------------------------------------------
# Indicate results.
# ---------------------------------------------------------------------
t4 = datetime.now()
print('Total run time: ' + str(t4 - t1))
print('Output path: ', html_dst_path)
print()
print('Note: warnings about `/details/index.rst` and `/intro/index.rst`')
print(' "not being in any toctree" are expected and intentional.')
print()
print('Finished.')
# -------------------------------------------------------------------------
# Make module importable as well as run-able.
# -------------------------------------------------------------------------
if __name__ == '__main__':
run()