| 1 | # Interactive Smartlog |
| 2 | |
| 3 | Interactive Smartlog (ISL) is an embeddable, web-based GUI for Sapling. |
| 4 | [See user documentation here](https://sapling-scm.com/docs/addons/isl). |
| 5 | |
| 6 | The code for ISL lives in the addons folder: |
| 7 | |
| 8 | | folder | use | |
| 9 | | ---------------- | ---------------------------------------------------------- | |
| 10 | | isl | Front end UI written with React and Jotai | |
| 11 | | isl-server | Back end, which runs sl commands / interacts with the repo | |
| 12 | | isl-server/proxy | `sl web` CLI and server management | |
| 13 | | shared | Utils shared by other projects | |
| 14 | | components | Shareable component library | |
| 15 | | vscode | VS Code extension for Sapling, including ISL as a webview | |
| 16 | |
| 17 | ## Development |
| 18 | |
| 19 | First run `yarn` to make sure all of the Node dependencies are installed. |
| 20 | |
| 21 | Use this command from the `addons/` folder to start ISL in development mode: |
| 22 | |
| 23 | ``` |
| 24 | yarn dev browser --launch . |
| 25 | ``` |
| 26 | |
| 27 | This does 3 things: |
| 28 | |
| 29 | - Build the client, and watch for changes (equivalent to `yarn start` in `isl/`) |
| 30 | - Build the server, and watch for changes (equivalent to `yarn watch` in `isl-server/`) |
| 31 | - Spawn a local server instance, which opens ISL in your browser (equivalent to `yarn serve` in `isl-server`, with some args). The server will open with `.` as the cwd. Use `--launch /path/to/my/repo` to use a different repository. |
| 32 | |
| 33 | The `yarn dev` command is a shorthand to running each of these in their own terminal. |
| 34 | |
| 35 | Note: the client and server build jobs will watch for changes. The webpage will hot reload as changes are made. The server must be restarted to pick up changes. |
| 36 | Press `R` when running `yarn dev browser --launch CWD` to restart the server while leaving the build running. |
| 37 | |
| 38 | ### Launching an ISL Server |
| 39 | |
| 40 | To see more server output, you may sometimes want to use `yarn dev browser` WITHOUT `--launch` to build the client and server, and then launch the server yourself with `yarn serve`. This launches the local ISL server. |
| 41 | |
| 42 | **In the `isl-server/` folder, run `yarn serve --dev` to start the server and open the browser**. |
| 43 | You will have to manually restart it in order to pick up server changes. |
| 44 | This is the development mode equivalent of running `sl web`. |
| 45 | |
| 46 | This launches a WebSocket Server to proxy requests between the server and the |
| 47 | client. The entry point code lives in the `isl-server/proxy/` folder and is a |
| 48 | simple HTTP server that processes `upgrade` requests and forwards |
| 49 | them to the WebSocket Server that expects connections at `/ws`. |
| 50 | |
| 51 | Note: When the server is started, it creates a token to prevent unwanted access. |
| 52 | `--dev` opens the browser on the port used by vite in `yarn start` |
| 53 | to ensure the client connects with the right token. |
| 54 | |
| 55 | **When developing, it's useful to add a few extra arguments to `yarn serve`:** |
| 56 | |
| 57 | ``` |
| 58 | yarn serve --dev --force --foreground --stdout |
| 59 | ``` |
| 60 | |
| 61 | - `--dev`: Connect to the vite dev build's hot-reloading front-end server (defaulting to 3000), even though this server will spawn on 3001. |
| 62 | - `--force`: Kill any other active ISL server running on this port, which makes sure it's the latest version of the code. |
| 63 | - `--foreground`: instead of spawning the server in the background, run it in the foreground. `ctrl-c`-ing the `yarn serve` process will kill this server. |
| 64 | - `--stdout`: when combined with `--foreground`, prints the server logs to stdout so you can read them directly in the `yarn serve` terminal output. |
| 65 | - `--command sl`: override the command to use for `sl`, for example you might use `./sl`, or an alias to your local build like `lsl`, or `hg` for Meta-internal uses |
| 66 | |
| 67 | ## Production builds |
| 68 | |
| 69 | `build-tar.py` is a script to build production bundles and |
| 70 | package them into a single self-contained `tar.xz` that can be distributed |
| 71 | along with `sl`. It can be launched by the `sl web` command. |
| 72 | |
| 73 | `yarn build` lets you build production bundles without watching for changes, in either |
| 74 | `isl/` or `isl-server/`. |
| 75 | |
| 76 | You can also use `yarn dev --production` to run both client & server `yarn build`. |
| 77 | |
| 78 | ## VS Code build |
| 79 | |
| 80 | Similarly to developing in the browser, you can use this command: |
| 81 | |
| 82 | ``` |
| 83 | yarn dev vscode --launch . |
| 84 | ``` |
| 85 | |
| 86 | This again does 3 things: |
| 87 | |
| 88 | - Build the webview (client), and watch for changes (equivalent to `yarn watch-webview` in `vscode/`) |
| 89 | - Build the extension (server), and watch for changes (equivalent to `yarn watch-extension` in `vscode/`) |
| 90 | - Start VS Code in extension development mode with the given directory. |
| 91 | |
| 92 | As with the server, you may want to launch vscode yourself. Just use `yarn dev vscode` to build without launching vscode. |
| 93 | |
| 94 | See also `../vscode/CONTRIBUTING.md`. |
| 95 | |
| 96 | ## Testing |
| 97 | |
| 98 | Run `yarn test` in the `isl` server to run client-side tests. These generally use |
| 99 | [React Testing Library](https://testing-library.com/docs/react-testing-library/intro/) to "render" the UI in a node process, and check that the fake in-memory DOM is correct. |
| 100 | |
| 101 | Sometimes, this can spit out very long errors, showing the entire DOM when some element is not found. |
| 102 | You can disable this by passing HIDE_RTL_DOM_ERRORS as an env var: |
| 103 | `HIDE_RTL_DOM_ERRORS=1 yarn test` |
| 104 | |
| 105 | # Goals |
| 106 | |
| 107 | ISL is designed to be an opinionated UI. It does not implement every single feature or argument that the CLI supports. |
| 108 | Rather, it implements an intuitive UI by leveraging a subset of features of the `sl` CLI. |
| 109 | |
| 110 | ISL aims to optimize common workflows and provide an intuitive UX around some advanced workflows. |
| 111 | |
| 112 | - **Opinionated**: ISL is opinionated about the "right" way to work. |
| 113 | This includes using stacks, amending commits, using one-commit-per-PR, rebasing to merge. |
| 114 | - **Simple**: ISL hides unnecessary details and aims to be beginner-friendly. |
| 115 | Each new button added to the UI makes it more intimidating to new users. |
| 116 | - **User concepts, not machine concepts**: |
| 117 | ISL hides implementation details to present source control in a way a human would understand it. |
| 118 | The salient example of this is not showing commit hashes in the UI by default. |
| 119 | Hashes are needed to refer to commits when typing in a CLI, but |
| 120 | ISL prefers being able to just click directly on commits, thus we don't need to show the hash by default. |
| 121 | Other examples of this include drag & drop to rebase, and showing PR info directly under a commit by leaning on one-PR-per-commit. |
| 122 | - **Previews & Smoothness**: The UI should let you preview what action you'll take. It shows an optimistic |
| 123 | version of the result of each command so the UI feels instant. We aim to avoid the UI _jumping_ between |
| 124 | states as a result of async data fetches |
| 125 | - **Documentation & Transparency**: The UI uses tooltips and other signals to show you what every button will do. |
| 126 | It always confirms before running dangerous commands. It shows exactly what CLI command is being run, so you |
| 127 | could do it yourself and trust what it's doing. |
| 128 | |
| 129 | # Internals |
| 130 | |
| 131 | The following sections describe how ISL is implemented. |
| 132 | |
| 133 | ## Build / Bundling |
| 134 | |
| 135 | - All parts of ISL (client, server, vscode extension) are built with vite/rollup, which produces javascript/css bundles. |
| 136 | This includes node_modules inside the bundle, which means we don't need to worry about including node_modules in builds. |
| 137 | - `sl web` is a normal `sl` python command, which invokes the latest ISL built CLI. |
| 138 | `isl-server/proxy/run-proxy.ts` is the typescript entry point which is spawned by Python via `node`. |
| 139 | In development mode, you interact directly with `run-proxy` rather than dealing with `sl web`. |
| 140 | Note: there are slightly differences between the python `sl web` CLI args and the `run-proxy` CLI args. |
| 141 | In general, `run-proxy` exposes more options, most of which aren't needed by normal `sl web` users. |
| 142 | |
| 143 | ## Architecture |
| 144 | |
| 145 | ISL uses an embeddable Client / Server architecture. |
| 146 | |
| 147 | - The Client runs in a browser-like context (web browser, VS Code webview, Electron renderer) |
| 148 | - The Server runs in a node-like context (node server from `sl web`, VS Code extension host, Electron main) |
| 149 | |
| 150 | The server serves the client's static (html/js/css) files via HTTP. |
| 151 | The client JavaScript then connects back to the server via WebSocket, |
| 152 | where both sides can send and receive messages to communicate. |
| 153 | |
| 154 | ### Client |
| 155 | |
| 156 | The client renders the UI and asks the server to actually do stuff. The client has no direct access |
| 157 | to the filesystem or repository. The client can make normal web requests, but does not have access tokens |
| 158 | to make authenticated requests to GitHub. |
| 159 | |
| 160 | The client uses React (for rendering the UI) and [Jotai](https://jotai.org/) (for state management). |
| 161 | We use a combination of regular CSS and [StyleX](https://stylexjs.com/) for styling. |
| 162 | |
| 163 | ### Server |
| 164 | |
| 165 | The server is able to interact with the file system, spawn processes, run `sl commands`, |
| 166 | and make authenticated network requests to GitHub. |
| 167 | The server is also responsible for watching the repository for changes. |
| 168 | This will optionally use Watchman if it's installed. |
| 169 | If not, the server falls back to a polling mechanism, which polls on a variable frequency |
| 170 | which depends on if the UI is focused and visible. |
| 171 | |
| 172 | The server shells out to the `gh` CLI to make authenticated requests to GitHub. |
| 173 | |
| 174 | Most of the server's work is done by the `Repository` object, which represents a single Sapling repository. |
| 175 | This object also delegates to manage Watchman subscriptions and GitHub fetching. |
| 176 | |
| 177 | ### Server reuse and sharing |
| 178 | |
| 179 | To support running `sl web` in multiple repos / cwds at the same time, ISL supports reusing server instances. |
| 180 | When spawning an ISL server, if the port is already in use by an ISL server, that server will be reused. |
| 181 | |
| 182 | Since the server acts like a normal http web server, it supports multiple clients connecting at the same time, |
| 183 | both the static resources and WebSocket connections. |
| 184 | |
| 185 | `Repository` instances inside the server are cached per repo root. |
| 186 | `RepositoryCache` manages Repositories by reference counting. |
| 187 | A `Repository` does not have its own cwd set. Rather, each reference to a `Repository` |
| 188 | via `RepositoryCache` has an associated cwd. This way, A single `Repository` instance is reused |
| 189 | even if accessed from multiple cwds within the same repo. |
| 190 | We treat each WebSocket connection as its own cwd, and each WebSocket connections has one reference |
| 191 | to a shared Repository via RepositoryCache. |
| 192 | |
| 193 | Connecting multiple clients to the same sever at the same cwd is also supported. |
| 194 | Server-side fetched data is sent to all relevant (same repo) clients, not just the one that made a request. |
| 195 | Note that client-side cached data is not shared, which means optimistic state may not work as well |
| 196 | in a second window for operations triggered in a different window. |
| 197 | |
| 198 | After all clients are disconnected, the server auto-shutdowns after one minute with no remaining repositories |
| 199 | which helps ensure that old ISL servers aren't reused. |
| 200 | |
| 201 | Note that ISL exposes `--kill` and `--force` options to kill old servers and force a fresh server, to make |
| 202 | it easy to work around unexpectedly reusing old ISL servers. |
| 203 | |
| 204 | ### Security |
| 205 | |
| 206 | The client sends messages to the server to run `sl` commands. |
| 207 | We must authenticate clients to ensure arbitrary websites or XSS attacks can't connect on localhost:3011 to run commands. |
| 208 | The approach we take is to generate a cryptographic token when a server is started. |
| 209 | Connecting via WebSocket to the server requires this token. |
| 210 | The token is included in the url generated by `sl web`, which allows URLs from `sl web` to connect successfully. |
| 211 | |
| 212 | Because of this token, restarting the ISL server requires clicking a fresh link to use the new token. |
| 213 | Once an ISL server stops running, its token is no longer valid. |
| 214 | |
| 215 | In order to support reusing ISL servers, we must persist the server's token to disk, |
| 216 | so that later `sl web` invocations can find the right token to use. |
| 217 | This persisted data includes the token but also some other metadata about the server, |
| 218 | which is written to a permission-restricted file. |
| 219 | |
| 220 | Detail: we have a second token we use to verify that a server running on a port |
| 221 | is actually an ISL server, to prevent misleading/phishing "reuses" of a server. |
| 222 | |
| 223 | ## Embedding |
| 224 | |
| 225 | ISL is designed to be embedded in multiple contexts. `sl web` is the default, |
| 226 | which is also the most complicated due to server reuse and managing tokens. |
| 227 | |
| 228 | The Sapling VS Code extension's ISL webview is another example of an embedding. |
| 229 | Other embeddings are possible, such as an Electron / Tauri standalone app, or |
| 230 | other IDE extensions such as Android Studio. |
| 231 | |
| 232 | ### Platform |
| 233 | |
| 234 | To support running in multiple contexts, ISL has the notion of a Platform, |
| 235 | on both the client and server, which contains embedding-specific implementations |
| 236 | of a common API. |
| 237 | |
| 238 | This includes things like opening a file. In the browser, the best we can do is use the OS default. |
| 239 | Inside the VS Code extension, we always want to open with VS Code. |
| 240 | Each platform can implement this to match their UX best. |
| 241 | The Client's platform is where platform-specific code first runs. Some embeddings |
| 242 | have their client platform send platform-specific messages to the server platform. |
| 243 | |
| 244 | The "default" platform is the BrowserPlatform, used by `sl web`. |
| 245 | |
| 246 | Custom platforms can be implemented either by: |
| 247 | |
| 248 | - including platform code in the build process (the VS Code extension does this) |
| 249 | - adding a new platform to isl-server for use by `run-proxy`'s `--platform` option (Android Studio does this) |
| 250 | |
| 251 | ## Syncing repository state |
| 252 | |
| 253 | ISL started as a way to automatically re-run `sl status` and `sl smartlog` in a loop. |
| 254 | The UI should always feel up-to-date, even though it needs to run these commands |
| 255 | to actually fetch the data. |
| 256 | The client subscribes to this data, which the server is in charge of fetching automatically. |
| 257 | The server uses Watchman (if installed) to detect when: |
| 258 | |
| 259 | - the `.sl/dirstate` has changed to indicate the list of commits has changed, so we should re-run `sl log`. |
| 260 | - any normal file in the repository has changed, so we should re-run `sl status` to look for uncommitted changes. |
| 261 | If Watchman is not installed, `sl log` and `sl status` are polled on an interval by `WatchForChanges` and based on window focus. |
| 262 | |
| 263 | Similarly, the server fetches new data from GitHub when the list of PRs changes, and refreshes by polling. |
| 264 | |
| 265 | ## Running Operations |
| 266 | |
| 267 | ISL defines an "Operation" as any mutating `sl` command, such as `sl pull`, `sl rebase`, `sl goto`, `sl amend`, `sl add`, etc. Non-examples include `sl status`, `sl log`, `sl cat`, `sl diff`. |
| 268 | |
| 269 | The lifecycle of an operation looks like this: |
| 270 | |
| 271 | ``` |
| 272 | Ready to run -> Preview -> Queued -> Running -> Optimistic state -> Completed |
| 273 | ``` |
| 274 | |
| 275 | ### Preview Appliers |
| 276 | |
| 277 | Critically, fetching data via `sl log` and `sl status` is separate from running operations. |
| 278 | We only get the "new" state of the world after _both_ the operation has completed _AND_ |
| 279 | `sl log` / `sl status` has run to provide us with the latest data. |
| 280 | |
| 281 | This would cause the UI to appear laggy and out of date. |
| 282 | Thus, we support using previews and optimistic to update the UI immediately. |
| 283 | |
| 284 | To support this, ISL defines a "`preview applier`" function for every operation. |
| 285 | The preview applier function describes how the DAG of commits and uncommitted changes |
| 286 | would change as a result of running this operation. |
| 287 | (Detail: there's actually a separate preview applier function for uncommitted changes and the commit DAG |
| 288 | to ensure UI smoothness if `sl log` and `sl status` return data at different times) |
| 289 | |
| 290 | This supports both: |
| 291 | |
| 292 | - **previews**: What would the DAG look like if I ran this command? |
| 293 | - e.g. Drag & drop rebase preview before clicking "run rebase" |
| 294 | - **optimistic state**: How should we pretend the DAG looks while this command is running? |
| 295 | - e.g. showing result of a rebase while rebase command is running |
| 296 | |
| 297 | Because `sl log` and `sl status` are run separately from an operation running, |
| 298 | the optimistic state preview applier must be used not just while the operation is running, |
| 299 | but also _after_ it finishes up until we get new data from `sl log` / `sl status`. |
| 300 | |
| 301 | ### Queued commands |
| 302 | |
| 303 | Preview Appliers are functions which take a commit DAG and return a new commit DAG. |
| 304 | This allows us to stack the result of preview appliers on top of each other. |
| 305 | This trivially enables _Queued Commands_, which work like `&&` on the CLI. |
| 306 | |
| 307 | If an operation is ongoing, and we click a button to run another, |
| 308 | it is queued up by the server to run next. |
| 309 | The client then renders the DAG resulting from first running Operation 1's preview applier, |
| 310 | then running Operation 2's preview applier. |
| 311 | |
| 312 | Important detail here: if an operation references a commit hash, the queued version |
| 313 | of that operation will not yet know the new hash after the previous operation finishes. |
| 314 | For example, `sl amend` in the middle of a stack, then `sl goto` the top of the stack. |
| 315 | Thus, when telling the server to run an Operation we tag which args are revsets, |
| 316 | so they are replaced with `max(successors(${revset}))` so the hash is replaced |
| 317 | with the latest successor hash. If you intentionally target an obsolete commit, then the hash is used directly. |
| 318 | |
| 319 | ## Internationalization |
| 320 | |
| 321 | ISL has a built-in i18n system, however the only language currently implemented is `en-US` English. |
| 322 | `t()` and `<T>` functions convert English strings or keys into values for other languages in the `isl/i18n/${languageCode}` folders. To add support for a new language, add a new `isl/i18n/${languageCode}/common.js` |
| 323 | and provide translations for all the strings found by grepping for `t()` and `<T>` in `isl`. |
| 324 | This system can be improved later as new languages are supported. |
| 325 | |
| 326 | # Debugging |
| 327 | |
| 328 | ## ✅ Attaching ISL server to VS Code debugger |
| 329 | |
| 330 | There's a "Run & Debug isl-server" vscode build action which runs `yarn serve --dev` for you with a few additional arguments. When spawned from here, you can use breakpoints in VS Code to step through your server-side code. |
| 331 | |
| 332 | Note that you should have the client & server rollup compilation jobs (described above) running before doing this (it currently won't compile for you, just launch `yarn serve`). |
| 333 | |
| 334 | ## ❓ Attaching ISL client to a debugger |
| 335 | |
| 336 | Attaching the client to VS Code debugger does not work as well as the server side. |
| 337 | There is currently no launch task to launch the browser and connect to the debugger. |
| 338 | You can try using "Debug: Open Link" from the command palette, and paste in the ISL server link |
| 339 | (with the token included), but I found breakpoint line numbers don't match up correctly. |
| 340 | |
| 341 | You can open the chrome devtools, go to sources, search for files, and set breakpoints in there, |
| 342 | which will mostly work. `debugger;` statements also work in the dev tools. |
| 343 | |
| 344 | ## Stack traces |
| 345 | |
| 346 | If you encounter a stack trace in production, it will be referencing minified line numbers like: |
| 347 | |
| 348 | ```txt |
| 349 | Error: something went wrong |
| 350 | at t (/some/production/path/to/isl-server/dist/run-proxy.js:1:4152) |
| 351 | ``` |
| 352 | |
| 353 | We build/ship with source maps that sit next to source files, like `isl-server/dist/run-proxy.js.map`. |
| 354 | |
| 355 | You can use these source maps to recover the real stack trace, using a tool like [stacktracify](https://github.com/mifi/stacktracify). |
| 356 | |
| 357 | ```sh |
| 358 | $ npm install -g stacktracify |
| 359 | # copy minified stack trace to clipboard, then give the path to the source map: |
| 360 | $ stacktracify /path/to/isl-server/dist/run-proxy.js.map |
| 361 | Error: something went wrong |
| 362 | at from (webpack://isl-server/proxy/proxyUtils.ts:14:22) |
| 363 | ``` |
| 364 | |
| 365 | Note that the source map you use must match the version in the original stack trace. |
| 366 | Usually, you can tell the version by the path in the stack trace. |
| 367 | |
| 368 | ## Profiling bundle sizes and dependencies |
| 369 | |
| 370 | **Client:** |
| 371 | To analyze the client bundle size (code splitting and dependencies, etc): |
| 372 | |
| 373 | - `cd isl` |
| 374 | - `npx vite-bundle-visualizer` |
| 375 | |
| 376 | Should also work in `vscode/` for the webview code. |
| 377 | |
| 378 | **Server:** |
| 379 | Install [rollup-plugin-visualizer](https://www.npmjs.com/package/rollup-plugin-visualizer) |
| 380 | and add it to the server's rollup.config.mjs, then `yarn build` and inspect the stats.html file. |
| 381 | |
| 382 | Should also work for the vscode extension config. |
| 383 | |