Architecture documentation for Configurator and API (#13935)

* Architecture documentation for the configurator and api

* Apply suggestions from code review

Co-authored-by: James Young <18669334+noroadsleft@users.noreply.github.com>

Co-authored-by: James Young <18669334+noroadsleft@users.noreply.github.com>
This commit is contained in:
Zach White 2021-08-10 07:47:53 -07:00 committed by GitHub
parent aeff347a07
commit 9a0118c603
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 66 additions and 0 deletions

View File

@ -19,6 +19,7 @@
* [Overview](newbs_building_firmware_configurator.md) * [Overview](newbs_building_firmware_configurator.md)
* [Step by Step](configurator_step_by_step.md) * [Step by Step](configurator_step_by_step.md)
* [Troubleshooting](configurator_troubleshooting.md) * [Troubleshooting](configurator_troubleshooting.md)
* [Architecture](configurator_architecture.md)
* QMK API * QMK API
* [Overview](api_overview.md) * [Overview](api_overview.md)
* [API Documentation](api_docs.md) * [API Documentation](api_docs.md)

View File

@ -0,0 +1,61 @@
# QMK Configurator Architecture
This page describes the web architecture behind QMK Configurator at a high level. If you are interested in the architecture of the QMK Configurator code itself you should start at the [qmk_configurator](https://github.com/qmk/qmk_configurator) repository.
# Overview
![QMK Configurator Architecture Diagram](configurator_diagram.svg)
# Detailed Description
QMK Configurator is a [Single Page Application](https://en.wikipedia.org/wiki/Single-page_application) that allows users to create custom keymaps for their QMK-compatible keyboard. They can export JSON representation of their keymaps and compile firmware binaries that can be flashed to their keyboard using a tool like [QMK Toolbox](https://github.com/qmk/qmk_toolbox).
Configurator gets metadata about keyboards from the Keyboard Metadata store and submits compile requests to the QMK API. The results of those compile requests will be made available on [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/), an S3-compatible data store.
## Configurator Frontend
Address: <https://config.qmk.fm>
The [Configurator Frontend](https://config.qmk.fm) is compiled into a set of static files that are served by Github Pages. This action happens every time a commit is pushed to the [qmk_configurator `master`](https://github.com/qmk/qmk_configurator) branch. You can view the status of these jobs on the [qmk_configurator actions tab](https://github.com/qmk/qmk_configurator/actions/workflows/build.yml).
## Keyboard Metadata
Address: <https://keyboards.qmk.fm>
The Keyboard Metadata is generated every time a keyboard in [qmk_firmware](https://github.com/qmk/qmk_firmware) changes. The resulting JSON files are uploaded to Spaces and used by Configurator to generate UI for each keyboard. You can view the status of this job on the [qmk_firmware actions tab](https://github.com/qmk/qmk_firmware/actions/workflows/api.yml). If you are a QMK Collaborator you can manually run this job using the `workflow_dispatch` event trigger.
## QMK API
Address: <http://api.qmk.fm>
The QMK API accepts `keymap.json` files for compilation. These are the same files you can use directly with `qmk compile` and `qmk flash`. When a `keymap.json` is submitted the browser will poll the status of the job periodically (every 2 seconds or longer, preferably) until the job has completed. The final status JSON will contain pointers to source and binary downloads for the keymap.
QMK API always presents the source and binary downloads side-by-side to comply with the GPL.
There are 3 non-error status responses from the API-
1. Compile Job Queued
2. Compile Job Running
3. Compile Job Finished
### Compile Job Queued
This status indicates that the job has not yet been picked up by a [QMK Compiler](#qmk-compiler) node. Configurator shows this status as "Waiting for an oven".
### Compile Job Running
This status indicates that the job has started compiling. Configurator shows this status as "Baking".
### Compile Job Finished
This status indicates that the job has completed. There will be keys in the status JSON for source and binary downloads.
## Redis/RQ
QMK API uses RQ to distribute jobs to the available [QMK Compiler](#qmk-compiler) nodes. When a `keymap.json` is received it's put into the RQ queue, where a `qmk_compiler` node will pick it up from.
## QMK Compiler
[QMK Compiler](https://github.com/qmk/qmk_compiler) is what actually performs the compilation of the `keymap.json`. It does so by checking out the requested `qmk_firmware` branch, running `qmk compile keymap.json`, and then uploading the resulting source and binary to Digital Ocean Spaces.
When users download their source/binary, API will redirect them to the authenticated Spaces download URL.

View File

@ -0,0 +1 @@
<mxfile host="Electron" modified="2021-08-09T19:46:29.036Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36" etag="PQ2r34UrZa0TfW4Fw0EV" version="14.6.13" type="device"><diagram id="NEtccoSKIy4HskWlhJpu" name="Page-1">5VvbcqM4EP2a1O4+hOLqy2Ni5zKX1CTxzszOU0oG2dZEIBZEYu/XbwuEDQg7csZ2vFlXjQca0RLdR+eohXPiDML5VYLi2Q0LMD2xzWB+4gxPbNvqezb8JywLaTFtr7BMExJI28owIv/gsqG0ZiTAaa0hZ4xyEteNPosi7POaDSUJe643mzBa7zVGU6wYRj6iqvU7CfissPY8c2W/xmQ6K3u2THklRGVjaUhnKGDPFZNzceIMEsZ4cRTOB5iK6JVxKe67XHN1ObAER1znhgfT76Zzxxp3Z49j5+7zjw/z+NSS+XlCNJNPLEfLF2UIcAARkacs4TM2ZRGiFyvrecKyKMCiHxPOVm0+MxaD0QLjT8z5QqYXZZyBacZDKq/iKDgTyYLTiEW4sFwSSqXLAKWz3L9orD64jEXKssTHG57WlQBCyRTzTVGRDsWTV3qQcb3CLMQ8WUCDBFPEyVMdK0hCbrpst8oKHMjEbJEk9+UckTDHcTWm0jQk4RR6pWQM3z4l8QNKuDhkYZxxnKRw/J0ljymHJ2HRg2X35vDPiKPp2kA/4YTj+cbIyKunjmv0zMpHeljSQXH6XJla0jSrzKrStvPIei9Hdgm9HNoZpyTCgyXZmPWYI0qmERz7EDGcgIGiMaa3LCUittULIoQEaOZzo8GYcc7CSoMz6ZKLqXQOJBKLgYXzqSBc4xmPKcy11JgSPsvG0GLCIn6JQkJFiK8xfcLCjbwgJyDM+uJ8wChL8ud0ivSILnjCHnHblQlMyIp9kn/ADkMJCK55G14OL4YXmybrFhhyOzXQtKEGdMVwVeBUzDvHTjmKjdS54jWfojQlfpP3YCR/CRwZXnn6o3ptOJcgK84W8gwma8JVz7lZ0qZVI9Ff401PkzddTdqsJM5rme6lTZtdZQ+3jMCTLXFjlRkqyaZJI8Vzy7uqytl01Gk4chqOisAojnJYLR/7F5BWTpkV0gZUTDjgbnOUxTEorqCosxO7QyGG5+MEjqbiaDBLoPsTGzo2L0mCJ2yuNhri9FHQi21+iehCAfHzjHA8ilEOkWegnTqGm0QVkiDIVwU5950j/3Garw9Kdsjd7YYZlmurkhlUYrDNFoB19qUnltPCCUW40xhFZcBnnIvl65noTChxNCFT4+/w0ZiEZXOwV+9Qk9bmd0SiKYTeNm+F9AM2RKTX3tno4iN6QiM/IbFYHXz7eqHeOJI8AeM2YbTQDL4fiuFnCeIs0RvmNUt54eWqUK1iwOnaZ28Asr7cfC0894E/q2PYehDs7w2CGqtFdbndqklVRaoI1BpN2tEaXVdryql2JGKzTKiEgvdasXHchqNDi01XAz/vpiK0tOHW0YTbYUpC6/1ULj9TuH0XlWZ93riOuhjotsz/rmf09pWkjpKkuvI/4sWYoSRIG+K/ks9PsgV4ucEcBYgjXV2ekCR8hoiq7a+uoeGZX+RN3PQ1BsdiwXB2+wG+h629rFR7SKDaRPSLj4VIm7n2ptsuG2voPKRIu7Zt9N9apHtbiXRr7VirASVHVgtAc4cFoDZNekclypZnWK5p9Tp9y+v1up1y73nRQMbWBWHdz3LP90ASbav14HuW6L4m9spsHotE95Us3d18Kkm2mS5KSZziik76lGXBy+XNLpTTq8O5Y6rKadkts3Rv27JlZ2uFE8VkrWTeX4z+nGS0DLSeWILDTYp3jyJ/Bqsf2xRrFXP4Bb6+3fy2tegdUuc6pqenct29pVFjkfryDulhtzDLWXskCnbarCtd75WadaoUlpqqBelBi0qzWDRINwzZanTkOObmkTVv8Pq1G+CgGMNuRVSl53csonZXV0R7RyWitrobIasPMMoCRGF4WY807b+PnD+OS3idXh347fuHrdrbt8qWu4+5WpzU1RcE81QocLTwHcMPIiMoUsJEPtI8+obPWqRZRlNK7+kLoqskejut3UWCeg1msh3L6PYrn54qsf2WbO3thYOjrpTeTGK1Oea4ikSQ/EaSX7t1qwiZfeC9W6ft9dO71TSn/D3YS3hz7KPSNEct3+/vlDyVciS2/FLOEnwYSWpskXS7ai3YOSjB2WqwcEDEm+9cjEwRuqaGfGTineJdhrOW3c/2AjApnO6/BNxBkpqVXrfz1pWes91Lx73KkPZ+UTkP31qHyiw26zFt2fEacGiWiPtWHXWNXvwuIIwpETNmDbPhuZhhAcvG+RWrJj3Wm2x99Vt+QXLYrS9XXdAVu4YDCCehIpztbHefRVH+a07dX22sc/efoLy+/daU5+pswh+I8vRXQt3joDyZVbeW015ztaxLgHU3/eZbnT3Tn6uuULYGwoF/z+lo7wYdF16AHes00GuuO/V3Qxui2bP0ULP1bqjTYK5yS2Gvm5vu/6sQ1IVzKa67KwTXEJK3GVu6KLWd14H0ZRjB6eovj4rmqz/gci7+BQ==</diagram></mxfile>

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 39 KiB