Blame · README.md

b69ab31	1	# Interactive Smartlog
b69ab31	2
b69ab31	3	Interactive Smartlog (ISL) is an embeddable, web-based GUI for Sapling.
b69ab31	4	[See user documentation here](https://sapling-scm.com/docs/addons/isl).
b69ab31	5
b69ab31	6	The code for ISL lives in the addons folder:
b69ab31	7
b69ab31	8	\| folder \| use \|
b69ab31	9	\| ---------------- \| ---------------------------------------------------------- \|
b69ab31	10	\| isl \| Front end UI written with React and Jotai \|
b69ab31	11	\| isl-server \| Back end, which runs sl commands / interacts with the repo \|
b69ab31	12	\| isl-server/proxy \| `sl web` CLI and server management \|
b69ab31	13	\| shared \| Utils shared by other projects \|
b69ab31	14	\| components \| Shareable component library \|
b69ab31	15	\| vscode \| VS Code extension for Sapling, including ISL as a webview \|
b69ab31	16
b69ab31	17	## Development
b69ab31	18
b69ab31	19	First run `yarn` to make sure all of the Node dependencies are installed.
b69ab31	20
b69ab31	21	Use this command from the `addons/` folder to start ISL in development mode:
b69ab31	22
b69ab31	23	```
b69ab31	24	yarn dev browser --launch .
b69ab31	25	```
b69ab31	26
b69ab31	27	This does 3 things:
b69ab31	28
b69ab31	29	- Build the client, and watch for changes (equivalent to `yarn start` in `isl/`)
b69ab31	30	- Build the server, and watch for changes (equivalent to `yarn watch` in `isl-server/`)
b69ab31	31	- Spawn a local server instance, which opens ISL in your browser (equivalent to `yarn serve` in `isl-server`, with some args). The server will open with `.` as the cwd. Use `--launch /path/to/my/repo` to use a different repository.
b69ab31	32
b69ab31	33	The `yarn dev` command is a shorthand to running each of these in their own terminal.
b69ab31	34
b69ab31	35	Note: the client and server build jobs will watch for changes. The webpage will hot reload as changes are made. The server must be restarted to pick up changes.
b69ab31	36	Press `R` when running `yarn dev browser --launch CWD` to restart the server while leaving the build running.
b69ab31	37
b69ab31	38	### Launching an ISL Server
b69ab31	39
b69ab31	40	To see more server output, you may sometimes want to use `yarn dev browser` WITHOUT `--launch` to build the client and server, and then launch the server yourself with `yarn serve`. This launches the local ISL server.
b69ab31	41
b69ab31	42	In the `isl-server/` folder, run `yarn serve --dev` to start the server and open the browser.
b69ab31	43	You will have to manually restart it in order to pick up server changes.
b69ab31	44	This is the development mode equivalent of running `sl web`.
b69ab31	45
b69ab31	46	This launches a WebSocket Server to proxy requests between the server and the
b69ab31	47	client. The entry point code lives in the `isl-server/proxy/` folder and is a
b69ab31	48	simple HTTP server that processes `upgrade` requests and forwards
b69ab31	49	them to the WebSocket Server that expects connections at `/ws`.
b69ab31	50
b69ab31	51	Note: When the server is started, it creates a token to prevent unwanted access.
b69ab31	52	`--dev` opens the browser on the port used by vite in `yarn start`
b69ab31	53	to ensure the client connects with the right token.
b69ab31	54
b69ab31	55	When developing, it's useful to add a few extra arguments to `yarn serve`:
b69ab31	56
b69ab31	57	```
b69ab31	58	yarn serve --dev --force --foreground --stdout
b69ab31	59	```
b69ab31	60
b69ab31	61	- `--dev`: Connect to the vite dev build's hot-reloading front-end server (defaulting to 3000), even though this server will spawn on 3001.
b69ab31	62	- `--force`: Kill any other active ISL server running on this port, which makes sure it's the latest version of the code.
b69ab31	63	- `--foreground`: instead of spawning the server in the background, run it in the foreground. `ctrl-c`-ing the `yarn serve` process will kill this server.
b69ab31	64	- `--stdout`: when combined with `--foreground`, prints the server logs to stdout so you can read them directly in the `yarn serve` terminal output.
b69ab31	65	- `--command sl`: override the command to use for `sl`, for example you might use `./sl`, or an alias to your local build like `lsl`, or `hg` for Meta-internal uses
b69ab31	66
b69ab31	67	## Production builds
b69ab31	68
b69ab31	69	`build-tar.py` is a script to build production bundles and
b69ab31	70	package them into a single self-contained `tar.xz` that can be distributed
b69ab31	71	along with `sl`. It can be launched by the `sl web` command.
b69ab31	72
b69ab31	73	`yarn build` lets you build production bundles without watching for changes, in either
b69ab31	74	`isl/` or `isl-server/`.
b69ab31	75
b69ab31	76	You can also use `yarn dev --production` to run both client & server `yarn build`.
b69ab31	77
b69ab31	78	## VS Code build
b69ab31	79
b69ab31	80	Similarly to developing in the browser, you can use this command:
b69ab31	81
b69ab31	82	```
b69ab31	83	yarn dev vscode --launch .
b69ab31	84	```
b69ab31	85
b69ab31	86	This again does 3 things:
b69ab31	87
b69ab31	88	- Build the webview (client), and watch for changes (equivalent to `yarn watch-webview` in `vscode/`)
b69ab31	89	- Build the extension (server), and watch for changes (equivalent to `yarn watch-extension` in `vscode/`)
b69ab31	90	- Start VS Code in extension development mode with the given directory.
b69ab31	91
b69ab31	92	As with the server, you may want to launch vscode yourself. Just use `yarn dev vscode` to build without launching vscode.
b69ab31	93
b69ab31	94	See also `../vscode/CONTRIBUTING.md`.
b69ab31	95
b69ab31	96	## Testing
b69ab31	97
b69ab31	98	Run `yarn test` in the `isl` server to run client-side tests. These generally use
b69ab31	99	[React Testing Library](https://testing-library.com/docs/react-testing-library/intro/) to "render" the UI in a node process, and check that the fake in-memory DOM is correct.
b69ab31	100
b69ab31	101	Sometimes, this can spit out very long errors, showing the entire DOM when some element is not found.
b69ab31	102	You can disable this by passing HIDE_RTL_DOM_ERRORS as an env var:
b69ab31	103	`HIDE_RTL_DOM_ERRORS=1 yarn test`
b69ab31	104
b69ab31	105	# Goals
b69ab31	106
b69ab31	107	ISL is designed to be an opinionated UI. It does not implement every single feature or argument that the CLI supports.
b69ab31	108	Rather, it implements an intuitive UI by leveraging a subset of features of the `sl` CLI.
b69ab31	109
b69ab31	110	ISL aims to optimize common workflows and provide an intuitive UX around some advanced workflows.
b69ab31	111
b69ab31	112	- Opinionated: ISL is opinionated about the "right" way to work.
b69ab31	113	This includes using stacks, amending commits, using one-commit-per-PR, rebasing to merge.
b69ab31	114	- Simple: ISL hides unnecessary details and aims to be beginner-friendly.
b69ab31	115	Each new button added to the UI makes it more intimidating to new users.
b69ab31	116	- User concepts, not machine concepts:
b69ab31	117	ISL hides implementation details to present source control in a way a human would understand it.
b69ab31	118	The salient example of this is not showing commit hashes in the UI by default.
b69ab31	119	Hashes are needed to refer to commits when typing in a CLI, but
b69ab31	120	ISL prefers being able to just click directly on commits, thus we don't need to show the hash by default.
b69ab31	121	Other examples of this include drag & drop to rebase, and showing PR info directly under a commit by leaning on one-PR-per-commit.
b69ab31	122	- Previews & Smoothness: The UI should let you preview what action you'll take. It shows an optimistic
b69ab31	123	version of the result of each command so the UI feels instant. We aim to avoid the UI _jumping_ between
b69ab31	124	states as a result of async data fetches
b69ab31	125	- Documentation & Transparency: The UI uses tooltips and other signals to show you what every button will do.
b69ab31	126	It always confirms before running dangerous commands. It shows exactly what CLI command is being run, so you
b69ab31	127	could do it yourself and trust what it's doing.
b69ab31	128
b69ab31	129	# Internals
b69ab31	130
b69ab31	131	The following sections describe how ISL is implemented.
b69ab31	132
b69ab31	133	## Build / Bundling
b69ab31	134
b69ab31	135	- All parts of ISL (client, server, vscode extension) are built with vite/rollup, which produces javascript/css bundles.
b69ab31	136	This includes node_modules inside the bundle, which means we don't need to worry about including node_modules in builds.
b69ab31	137	- `sl web` is a normal `sl` python command, which invokes the latest ISL built CLI.
b69ab31	138	`isl-server/proxy/run-proxy.ts` is the typescript entry point which is spawned by Python via `node`.
b69ab31	139	In development mode, you interact directly with `run-proxy` rather than dealing with `sl web`.
b69ab31	140	Note: there are slightly differences between the python `sl web` CLI args and the `run-proxy` CLI args.
b69ab31	141	In general, `run-proxy` exposes more options, most of which aren't needed by normal `sl web` users.
b69ab31	142
b69ab31	143	## Architecture
b69ab31	144
b69ab31	145	ISL uses an embeddable Client / Server architecture.
b69ab31	146
b69ab31	147	- The Client runs in a browser-like context (web browser, VS Code webview, Electron renderer)
b69ab31	148	- The Server runs in a node-like context (node server from `sl web`, VS Code extension host, Electron main)
b69ab31	149
b69ab31	150	The server serves the client's static (html/js/css) files via HTTP.
b69ab31	151	The client JavaScript then connects back to the server via WebSocket,
b69ab31	152	where both sides can send and receive messages to communicate.
b69ab31	153
b69ab31	154	### Client
b69ab31	155
b69ab31	156	The client renders the UI and asks the server to actually do stuff. The client has no direct access
b69ab31	157	to the filesystem or repository. The client can make normal web requests, but does not have access tokens
b69ab31	158	to make authenticated requests to GitHub.
b69ab31	159
b69ab31	160	The client uses React (for rendering the UI) and [Jotai](https://jotai.org/) (for state management).
b69ab31	161	We use a combination of regular CSS and [StyleX](https://stylexjs.com/) for styling.
b69ab31	162
b69ab31	163	### Server
b69ab31	164
b69ab31	165	The server is able to interact with the file system, spawn processes, run `sl commands`,
b69ab31	166	and make authenticated network requests to GitHub.
b69ab31	167	The server is also responsible for watching the repository for changes.
b69ab31	168	This will optionally use Watchman if it's installed.
b69ab31	169	If not, the server falls back to a polling mechanism, which polls on a variable frequency
b69ab31	170	which depends on if the UI is focused and visible.
b69ab31	171
b69ab31	172	The server shells out to the `gh` CLI to make authenticated requests to GitHub.
b69ab31	173
b69ab31	174	Most of the server's work is done by the `Repository` object, which represents a single Sapling repository.
b69ab31	175	This object also delegates to manage Watchman subscriptions and GitHub fetching.
b69ab31	176
b69ab31	177	### Server reuse and sharing
b69ab31	178
b69ab31	179	To support running `sl web` in multiple repos / cwds at the same time, ISL supports reusing server instances.
b69ab31	180	When spawning an ISL server, if the port is already in use by an ISL server, that server will be reused.
b69ab31	181
b69ab31	182	Since the server acts like a normal http web server, it supports multiple clients connecting at the same time,
b69ab31	183	both the static resources and WebSocket connections.
b69ab31	184
b69ab31	185	`Repository` instances inside the server are cached per repo root.
b69ab31	186	`RepositoryCache` manages Repositories by reference counting.
b69ab31	187	A `Repository` does not have its own cwd set. Rather, each reference to a `Repository`
b69ab31	188	via `RepositoryCache` has an associated cwd. This way, A single `Repository` instance is reused
b69ab31	189	even if accessed from multiple cwds within the same repo.
b69ab31	190	We treat each WebSocket connection as its own cwd, and each WebSocket connections has one reference
b69ab31	191	to a shared Repository via RepositoryCache.
b69ab31	192
b69ab31	193	Connecting multiple clients to the same sever at the same cwd is also supported.
b69ab31	194	Server-side fetched data is sent to all relevant (same repo) clients, not just the one that made a request.
b69ab31	195	Note that client-side cached data is not shared, which means optimistic state may not work as well
b69ab31	196	in a second window for operations triggered in a different window.
b69ab31	197
b69ab31	198	After all clients are disconnected, the server auto-shutdowns after one minute with no remaining repositories
b69ab31	199	which helps ensure that old ISL servers aren't reused.
b69ab31	200
b69ab31	201	Note that ISL exposes `--kill` and `--force` options to kill old servers and force a fresh server, to make
b69ab31	202	it easy to work around unexpectedly reusing old ISL servers.
b69ab31	203
b69ab31	204	### Security
b69ab31	205
b69ab31	206	The client sends messages to the server to run `sl` commands.
b69ab31	207	We must authenticate clients to ensure arbitrary websites or XSS attacks can't connect on localhost:3011 to run commands.
b69ab31	208	The approach we take is to generate a cryptographic token when a server is started.
b69ab31	209	Connecting via WebSocket to the server requires this token.
b69ab31	210	The token is included in the url generated by `sl web`, which allows URLs from `sl web` to connect successfully.
b69ab31	211
b69ab31	212	Because of this token, restarting the ISL server requires clicking a fresh link to use the new token.
b69ab31	213	Once an ISL server stops running, its token is no longer valid.
b69ab31	214
b69ab31	215	In order to support reusing ISL servers, we must persist the server's token to disk,
b69ab31	216	so that later `sl web` invocations can find the right token to use.
b69ab31	217	This persisted data includes the token but also some other metadata about the server,
b69ab31	218	which is written to a permission-restricted file.
b69ab31	219
b69ab31	220	Detail: we have a second token we use to verify that a server running on a port
b69ab31	221	is actually an ISL server, to prevent misleading/phishing "reuses" of a server.
b69ab31	222
b69ab31	223	## Embedding
b69ab31	224
b69ab31	225	ISL is designed to be embedded in multiple contexts. `sl web` is the default,
b69ab31	226	which is also the most complicated due to server reuse and managing tokens.
b69ab31	227
b69ab31	228	The Sapling VS Code extension's ISL webview is another example of an embedding.
b69ab31	229	Other embeddings are possible, such as an Electron / Tauri standalone app, or
b69ab31	230	other IDE extensions such as Android Studio.
b69ab31	231
b69ab31	232	### Platform
b69ab31	233
b69ab31	234	To support running in multiple contexts, ISL has the notion of a Platform,
b69ab31	235	on both the client and server, which contains embedding-specific implementations
b69ab31	236	of a common API.
b69ab31	237
b69ab31	238	This includes things like opening a file. In the browser, the best we can do is use the OS default.
b69ab31	239	Inside the VS Code extension, we always want to open with VS Code.
b69ab31	240	Each platform can implement this to match their UX best.
b69ab31	241	The Client's platform is where platform-specific code first runs. Some embeddings
b69ab31	242	have their client platform send platform-specific messages to the server platform.
b69ab31	243
b69ab31	244	The "default" platform is the BrowserPlatform, used by `sl web`.
b69ab31	245
b69ab31	246	Custom platforms can be implemented either by:
b69ab31	247
b69ab31	248	- including platform code in the build process (the VS Code extension does this)
b69ab31	249	- adding a new platform to isl-server for use by `run-proxy`'s `--platform` option (Android Studio does this)
b69ab31	250
b69ab31	251	## Syncing repository state
b69ab31	252
b69ab31	253	ISL started as a way to automatically re-run `sl status` and `sl smartlog` in a loop.
b69ab31	254	The UI should always feel up-to-date, even though it needs to run these commands
b69ab31	255	to actually fetch the data.
b69ab31	256	The client subscribes to this data, which the server is in charge of fetching automatically.
b69ab31	257	The server uses Watchman (if installed) to detect when:
b69ab31	258
b69ab31	259	- the `.sl/dirstate` has changed to indicate the list of commits has changed, so we should re-run `sl log`.
b69ab31	260	- any normal file in the repository has changed, so we should re-run `sl status` to look for uncommitted changes.
b69ab31	261	If Watchman is not installed, `sl log` and `sl status` are polled on an interval by `WatchForChanges` and based on window focus.
b69ab31	262
b69ab31	263	Similarly, the server fetches new data from GitHub when the list of PRs changes, and refreshes by polling.
b69ab31	264
b69ab31	265	## Running Operations
b69ab31	266
b69ab31	267	ISL defines an "Operation" as any mutating `sl` command, such as `sl pull`, `sl rebase`, `sl goto`, `sl amend`, `sl add`, etc. Non-examples include `sl status`, `sl log`, `sl cat`, `sl diff`.
b69ab31	268
b69ab31	269	The lifecycle of an operation looks like this:
b69ab31	270
b69ab31	271	```
b69ab31	272	Ready to run -> Preview -> Queued -> Running -> Optimistic state -> Completed
b69ab31	273	```
b69ab31	274
b69ab31	275	### Preview Appliers
b69ab31	276
b69ab31	277	Critically, fetching data via `sl log` and `sl status` is separate from running operations.
b69ab31	278	We only get the "new" state of the world after _both_ the operation has completed _AND_
b69ab31	279	`sl log` / `sl status` has run to provide us with the latest data.
b69ab31	280
b69ab31	281	This would cause the UI to appear laggy and out of date.
b69ab31	282	Thus, we support using previews and optimistic to update the UI immediately.
b69ab31	283
b69ab31	284	To support this, ISL defines a "`preview applier`" function for every operation.
b69ab31	285	The preview applier function describes how the DAG of commits and uncommitted changes
b69ab31	286	would change as a result of running this operation.
b69ab31	287	(Detail: there's actually a separate preview applier function for uncommitted changes and the commit DAG
b69ab31	288	to ensure UI smoothness if `sl log` and `sl status` return data at different times)
b69ab31	289
b69ab31	290	This supports both:
b69ab31	291
b69ab31	292	- previews: What would the DAG look like if I ran this command?
b69ab31	293	- e.g. Drag & drop rebase preview before clicking "run rebase"
b69ab31	294	- optimistic state: How should we pretend the DAG looks while this command is running?
b69ab31	295	- e.g. showing result of a rebase while rebase command is running
b69ab31	296
b69ab31	297	Because `sl log` and `sl status` are run separately from an operation running,
b69ab31	298	the optimistic state preview applier must be used not just while the operation is running,
b69ab31	299	but also _after_ it finishes up until we get new data from `sl log` / `sl status`.
b69ab31	300
b69ab31	301	### Queued commands
b69ab31	302
b69ab31	303	Preview Appliers are functions which take a commit DAG and return a new commit DAG.
b69ab31	304	This allows us to stack the result of preview appliers on top of each other.
b69ab31	305	This trivially enables _Queued Commands_, which work like `&&` on the CLI.
b69ab31	306
b69ab31	307	If an operation is ongoing, and we click a button to run another,
b69ab31	308	it is queued up by the server to run next.
b69ab31	309	The client then renders the DAG resulting from first running Operation 1's preview applier,
b69ab31	310	then running Operation 2's preview applier.
b69ab31	311
b69ab31	312	Important detail here: if an operation references a commit hash, the queued version
b69ab31	313	of that operation will not yet know the new hash after the previous operation finishes.
b69ab31	314	For example, `sl amend` in the middle of a stack, then `sl goto` the top of the stack.
b69ab31	315	Thus, when telling the server to run an Operation we tag which args are revsets,
b69ab31	316	so they are replaced with `max(successors(${revset}))` so the hash is replaced
b69ab31	317	with the latest successor hash. If you intentionally target an obsolete commit, then the hash is used directly.
b69ab31	318
b69ab31	319	## Internationalization
b69ab31	320
b69ab31	321	ISL has a built-in i18n system, however the only language currently implemented is `en-US` English.
b69ab31	322	`t()` and `<T>` functions convert English strings or keys into values for other languages in the `isl/i18n/${languageCode}` folders. To add support for a new language, add a new `isl/i18n/${languageCode}/common.js`
b69ab31	323	and provide translations for all the strings found by grepping for `t()` and `<T>` in `isl`.
b69ab31	324	This system can be improved later as new languages are supported.
b69ab31	325
b69ab31	326	# Debugging
b69ab31	327
b69ab31	328	## ✅ Attaching ISL server to VS Code debugger
b69ab31	329
b69ab31	330	There's a "Run & Debug isl-server" vscode build action which runs `yarn serve --dev` for you with a few additional arguments. When spawned from here, you can use breakpoints in VS Code to step through your server-side code.
b69ab31	331
b69ab31	332	Note that you should have the client & server rollup compilation jobs (described above) running before doing this (it currently won't compile for you, just launch `yarn serve`).
b69ab31	333
b69ab31	334	## ❓ Attaching ISL client to a debugger
b69ab31	335
b69ab31	336	Attaching the client to VS Code debugger does not work as well as the server side.
b69ab31	337	There is currently no launch task to launch the browser and connect to the debugger.
b69ab31	338	You can try using "Debug: Open Link" from the command palette, and paste in the ISL server link
b69ab31	339	(with the token included), but I found breakpoint line numbers don't match up correctly.
b69ab31	340
b69ab31	341	You can open the chrome devtools, go to sources, search for files, and set breakpoints in there,
b69ab31	342	which will mostly work. `debugger;` statements also work in the dev tools.
b69ab31	343
b69ab31	344	## Stack traces
b69ab31	345
b69ab31	346	If you encounter a stack trace in production, it will be referencing minified line numbers like:
b69ab31	347
b69ab31	348	```txt
b69ab31	349	Error: something went wrong
b69ab31	350	at t (/some/production/path/to/isl-server/dist/run-proxy.js:1:4152)
b69ab31	351	```
b69ab31	352
b69ab31	353	We build/ship with source maps that sit next to source files, like `isl-server/dist/run-proxy.js.map`.
b69ab31	354
b69ab31	355	You can use these source maps to recover the real stack trace, using a tool like [stacktracify](https://github.com/mifi/stacktracify).
b69ab31	356
b69ab31	357	```sh
b69ab31	358	$ npm install -g stacktracify
b69ab31	359	# copy minified stack trace to clipboard, then give the path to the source map:
b69ab31	360	$ stacktracify /path/to/isl-server/dist/run-proxy.js.map
b69ab31	361	Error: something went wrong
b69ab31	362	at from (webpack://isl-server/proxy/proxyUtils.ts:14:22)
b69ab31	363	```
b69ab31	364
b69ab31	365	Note that the source map you use must match the version in the original stack trace.
b69ab31	366	Usually, you can tell the version by the path in the stack trace.
b69ab31	367
b69ab31	368	## Profiling bundle sizes and dependencies
b69ab31	369
b69ab31	370	Client:
b69ab31	371	To analyze the client bundle size (code splitting and dependencies, etc):
b69ab31	372
b69ab31	373	- `cd isl`
b69ab31	374	- `npx vite-bundle-visualizer`
b69ab31	375
b69ab31	376	Should also work in `vscode/` for the webview code.
b69ab31	377
b69ab31	378	Server:
b69ab31	379	Install [rollup-plugin-visualizer](https://www.npmjs.com/package/rollup-plugin-visualizer)
b69ab31	380	and add it to the server's rollup.config.mjs, then `yarn build` and inspect the stats.html file.
b69ab31	381
b69ab31	382	Should also work for the vscode extension config.