Team DevLog Holzchopf - Update 2021/07/06

[Holzchopf]

Wow, already a month went by. Here is a short update on what I am doing. @Phil7 and @Holzchopf can do similar if they want to.

Well, of course! Here we are. I start a new thread, to keep your dev log tidy

As you might have read, I am currently working on a VS Code extension for Cerberus X. What I have so far:
- syntax highlighting
- document outline
- rudimentary in-editor help panel

See for yourself:

It's not yet usable, as there's no build option yet. But you know, the whole thing started as an experiment and my primary focus was on whether an extension could provide an outline and in-editor help panel. Latter isn't nice to look at yet (and turned out to be a massive pain in the rear thanks to (valid) security restrictions), but you can tell that a proper VS Code extension for Cerberus X is in its making.

 

[MikeHart]

I am already in love. Can't wait to get my hands on it. But please don't rush.

 

[Holzchopf]

In-Editor Help

This is everything but straight-forward. The vscode.Webview is quite restrictive; html+css is rendered correctly except for images, external links open in a new (standard) browser window, internal links to #anchors don't work at all. Also, there's no navigation panel. For links to work (i.e. open a new page), they have to call a JavaScript function which tells the extension to load a new page, which then again does that work and provides the new html source. Great. I saw myself confronted with two options:

1. Try parsing the existing generated .html files, replace links with those function calls, replace image resources with the correct VSCode-scheme-URI, replace the existing header with one that provides a navigation.
2. Let Makedocs spit out a more easily readable format - a decl.json containing all DocDecls - and build the contents from that.

Both methods have their pros and cons. I went down path 2 because I was too scared of parsing .html which might change and then break the parsing. Of course this means I have to transform Markdown in docs again, which wouldn't be too much of a hassle if all the Markdown was standard Markdown (I guess I'd find some existing Markdown to HTML transformer).... Also this method will break syntax-highlighting in code blocks. It could be implemented again, but for now I see things that have higher priority. Lastly this method will ignore custom global style rules like the multicolumn class, but if there's only one of them, I might even just copy that into my template. Speaking of styles, this method allows to style everything according to the currently set VSCode theme, which is nice.

So where am I now?

1. A navigation panel (not fully functional now)
2. The main menu is part of the .html documentation templates - There would be ways of extracting it from there, but for now I just hard-coded it. I'll adapt it to what @Phil7 will present us with
3. Declaration-targeting links work. They're resolved by Makedocs directly as these resolving rules are soooo complicated*. External links work too, naturally. Anchor-links don't work yet

* just to give you a hint of what Makedocs has to deal with when resolving a link: [[Create]] might be targeting a document or any declared identifier (function, method, class, module...). How should it now? It doesn't. It first looks in the current scope (module or class depending on where the decl containing that link sits), then it looks in all imported modules, then it looks in the global scope, which possibly even finds it in documents.

 

[MikeHart]

I am all for moving to a standard format. Rich provided a conversion tool, so we could settle for .md for all the official stuff.

 

[Holzchopf]

I just had the following idea: Instead of passing raw .cerberusdoc content to the extension, I could parse the markdown directly within Makedocs and put the result in the output .json - then we'd only have one tool that has to deal with this kind of markdown, namely Makedocs. This way the generated html does not change when templates change and also if we ever change the markdown format it will only be Makedocs that has to be updated.

I'll try that now and report back.

 

[Holzchopf]

Uuuhh I promised to report back, but I never did

Makedocs now generates that one big .json file containing all docs. Works "flawlessly". I put that in quotation marks because there's one nasty flaw: brl.markdown doesn't always generate valid html and different user agents seem to render the generated, broken html differently. No game stopping thing, it mostly happens in lists (seems they're not terminated correctly) - I only noticed, because the landing page looked different.

(notice the margins between some list points)

 

[MikeHart]

HOME HOME HOME

But it is great to see the progress. Thank you

 

[Holzchopf]

Just a quick update: I'm still working on it. Some weeks ago I gave few people access to the repo so they could test it. Apparently it's not running right now. I will look into that and I will definitely continue my work, with focus on building so it could already be somewhat used for productive work. But (!) right now, work for my company is plenty as a big project's deadline is approaching. So for the last 4 weeks I couldn't work on it and I think I won't be able to pick up work before February 2021.

 

[Holzchopf]

These times... So I managed to pack an early alpha. It's attached. Many things don't run yet, unfortunately. But I ask you to test the following:

1. Install the vsix (Extensions > ... > Install from VSIX), open a .cxs file of your choice.
2. A popup will appear telling you the Cerberus X Path hasn't been set. Click 'Set Path' and select your Cerberus X installation path.
3. Try the commands (from the command prompt CTRL+SHIFT+P) 'Cerberus X: Build as HTML5 application' and 'Cerberus X: Run as HTML5 application'

Please report any Errors that occur along the way. Or even if all these steps worked. You can also glimpse at the Cerberus X documentation, but they're not fully functional yet.

Cheers!

 

[MikeHart]

Good job so far. What I noticed:

1. You need to build the app first. Run doesn't include building.
2. After building, I have to click inside the source code, or I will get an error message:
   
   
   
3. Yes, the docs display is rather not pretty.
4. No F1 help.

 

[Pierrou]

It builds and runs when testing with Bouncy Aliens

One of my own files looks like this :

 

[MikeHart]

One of my own files looks like this :

I noticed that too, but I think it is because of VSCode still analysing the Decls file. Not sure. After a restart of VSC, it looked fine.
Or do you have, by any chance a different Cerberus extension installed too?

 

[Martin]

This is so cool! Thanks for making that!

 

[Holzchopf]

Thanks for your reports! @MikeHart concerning the docs display, since there are multiple aspects to consider:

• The docs are themed with the current vscode theme colors. I tried to always take pairs of --XXX-front and --XXX-background colors, hoping all would be legible. If that's somewhere not the case, can you post a screenshot of how they look?
• Some features simply don't work in the webview vscode provides (for security reasons), breaking some of the pages (simply meaning: there's stuff that won't be loaded and the page might look relatively blank). Are there sites/docs that are completely messed up?
• Yes, the "one page per declaration" thing is new and might irritate at first. It was the quickest to implement. But I plan to also provide a nice overview page (something a bit nicer than this plain table). Is it maybe that what you consider "rather not pretty"? =)
• All in all, I am experimenting a lot with the docs layout there. Wishes are welcome, although I can't promise they'll be respected in full

The F1 thing is definitely on my to do list!

@Pierrou that seems to be an error in my syntax definitions... I haven't considered preprocessor directives with string values

 

[Holzchopf]

Tiny update (although the file size increased by an unholy amount compared to the actual changes )

Changelog:

Fixed

• Syntax highlighting for precompiler directives

Changed

• Run as HTML5 now also builds

 

[Holzchopf]

One quite old but still open issue on GitHub about VSCode is that there's no way to find the TextMate scope at a given position programmatically. It should be possible, as the language definition is complete in that manner. Also, there are dev-tools that do exactly that.

Now, why does this matter?

I implemented an autocapitalise feature for keywords. Works, that's great BUT! It also triggers when in block comments or within strings. The easiest way to counteract this would simply be to ask TextMate "Hey, what scope am I currently in?" and it would respond "You're in '
comment.block.cxs'" and the extension would say "Thanks mate!" and wouldn't bother to capitalise in that case. But NOOOOO the devs made it clear there won't ever be a way to interact with TextMate programmatically in that way. Sucks!

So, what does mister I-cannot-live-with-half-assed-solutions Holzchopf do? Simple! Just kidding, it's everything else but simple

First, I wrote a CX tokenizer that is actually clever enough to only re-tokenize what's necessary. This works and to my surprise wasn't that much of a pain in the rear. Why that? Well, there are multiple occasions where a readily tokenized complete file beats having the source just as text and going through the whole text to do whatever task. Over and over again.

One of these tasks is to create the document outline (a thing that already worked). When VSCode asked for the outline, the document was parsed as a whole and the outline generated from that.

I'm now rewriting so it asks the tokenizer to update the document's tokens and then builds the outline from those tokens. That's what I'm currently working on. Of course, this means completely rewriting the documentOutlineProvider. But it will actually be faster when I'm done.

Then come the other features that rely on a tokenized document. Like the autocapitalise feature. Or the F1-to-open-help. They will all - when called - just trigger the tokenizer, which will update the tokens where necessary.


Oh and also: I refactored the whole thing. Have I stated that the whole thing started as a test, whether the CX Docs were displayable in a VSCode webview? This meant the extension was kinda like built around a documentation presenter. I now properly moved the documentation presenter to what one could call a sub-module hosted by something that's worth bearing the name extension.

 

[MikeHart]

Thanks for the inside informations. Sounds quite like a task. The other day I found a site showing a tree view. If you clicked on an item , it would navigate to the position inside the file. Like what we have now. Is that of interest for you?

 

[Holzchopf]

This will be possible =)

 

[Jimmy]

Looking good. Nice progress everywhere!

 

[MikeHart]

Looking great @Holzchopf