Extension:VisualEditor/Plugin for source code editing (SyntaxHighlight GeSHi support)

Google Summer of Code 2013

Introduction

Me

I mainly work with JavaScript and Java.

Expectation about GSoC project

The project I will be working on will add source code editing feature to the existing VisualEditor. It aims to provide some basic features that are mostly seen in IDEs. Proposed features include beautifier, auto-indent, and a basic grammar checker that checks generic coding practices. In the future it might be possible to add a linter mechanism but it's out of the scope of the current project. I hope that the project, once finished, will indeed help those who needs to work with code pieces in article pages. The project will free the editor from formatting the codes and help them concentrate more on the content itself.

VisualEditor plugin for source code editing (SyntaxHighlight GeSHi support)

Name and contact information

Name: Tongbo S.
Location: Beijing, China; GMT+8
Typical working hours: 9 am - 5 pm

Synopsis

Tracked in Phabricator
Task T49742

The plugin will include: A separate input interface dedicated for code editing. To prevent it from stealing focus from the main content, the interface will be an overlay layer on the editor. The use of separate interface will help the author focus more on solely the code piece, without worrying about changing other parts of the article by accident.

In this interface:

• A simple grammar checking mechanism. Checks some fundamental grammatical errors such as having a { without closing it, having a () with a trailing period and nothing else, having solely a pair of brackets, etc.
• An indentation correction tool. The user can click on the button to evoke this tool, which will perform indentation formatting on the codes.
• A beautify code tool. This is different from indentation correction since it will reorganize code formatting, including indentation. Sometimes this tool can be bothering since it changes the codes into a single format, which could be inconvenient for the author, although being good for readers.

Anyone who needs to put source codes in their article will benefit from this plugin.

Deliverables

An iterative and cooperative approach is carried out in this project. In the beginning phase, I should discuss with mentor about the project’s logical structure and form of presentation (UI, feedback, etc.). Then it is necessary for me to discuss and settle down with mentor on what coding practices, if any, should be enforced; whether it is a particular API, or avoiding use of certain codes. After that I shall proceed to coding phase. Then the code will be evaluated. The project will be maintained by me afterwards since I’m the author of it.

Tentative Timeline and deliverables:

Community bonding period:

• Week 1:

Get to know the community and with mentor

Study of existing code/API, reading documentations

• Week 2:

Agreement with mentor on project’s logical structure, form of presentation

Study of existing code/API, reading documentations

• Week 3:

Agreement with mentor on coding practices

Study of existing code/API, reading documentations

Deliverable: document of logical structure (flowchart), UI mockup, details on coding practices; this concludes works of the first 3 weeks.

Starting from Week 4 (June 17): Coding period

• Week 4

Create basic UI for testing purpose. Create a functional prototype includes the grammar checker

• Week 5

Create a functional prototype of indentation corrector.

Deliverable: Code; a working prototype of grammar checker and indentation corrector.

• Week 6, 7

Work on the code beautifier. This is based on the codes of the other two tools already prototyped; however this requires a lot more work to make the beautifier work just right. Code up a working prototype and test it. Discuss with mentor about possible additional features and implement those; also do more testing on the existing prototype in search for serious bugs.

Deliverable: Code; a working prototype added with beautifier.

• Week 8, 9

Finalize features code. All additional features, if any, should also be added in and tested. Eliminate significant bugs when testing in stand-alone.

Build the complete UI.

Deliverable: Code of everything.

Mid-term evaluation

• Week 10

Merging test. See if the plugin can cooperate correctly with other parts of the trunk.

Deliverable: Commits of fixes.

• Week 11

Usability test. See if the plugin UI is user-friendly, ease-to-use, and the UI itself behaves correctly. Open to major UI changes(layouts etc.) if applicable.

Deliverable: Commits of fixes.

• Week 12 - 16

Code review continued. Code testing and bugfix. Should not open to major UI changes (they should be set during week 11).

• Starting from Week 13

Documentation writing.

Deliverable: Commits of fixes. Documentation.

• Sep 16 (Week 17)

Pencils down. Taking a week to scrub code, write tests, improve documentation, etc.

• Sep 23 (Week 18)

Submit final evaluations

Conclusion of deliverables & documentation:

gerrit:68615
Documentation of related JSON files

About you

Motivation: "Focus on the content, forget formatting"

I've always run into the problem in text editors where I have to spend a lot of time fixing the format of my code to make it looks more organized and human-readable. Codes without correct formatting just look like a piece of random, scrambled text that makes no sense. Having to spend time correct formatting has become a major distraction for me, and thus dividing my energy from actually writing codes.

I can imagine all other people who share the same pain as I do, and am sure the population is not small. Editors like Eclipse and its offshoots have very neat features to format codes automatically, but they are too big to fit into a webpage. Therefore it is essential to implement a lightweight and yet fully featured plugin to achieve similar functionalities. This plugin will release those who needs to edit codes in their articles from tedious and boring process of correcting code formats, and enable them to focus more on the content and quality of the codes.

Technical skills

Operating Systems: Windows, Linux (Ubuntu)
Object-oriented programming: Java, JavaScript, C++
Web development: HTML, CSS, JavaScript, jQuery
Scripting: Python, Ruby
Mobile: Android
IDE: Eclipse, Aptana

Participation

Communication will be carried via email, IRC, Google Hangout, and/or Skype if needed. Source code will be managed under Git and reviewed on Gerrit. Problem solving is more self-education oriented. I like to search for answers before ask questions. Most of the time I can find what I need; but if I don’t, I will reach out to anybody available for help.

Any other info

Thoughts on the implementation of the plugin

A possible implementation of the plugin is to develop base on the idea of lexical analysis. Using regular expressions in JavaScript to check syntax errors, and parse all places that need re-formatting.

Internal mechanism of the plugin

Some confusion might arise about how this plugin will work when VisualEditor is almost restricted to editing plaintext.

I understand that the rich text is just a presentation and I can see in article source that the formats are defined using a set of markups. What I'm proposing is a plugin/feature which will take care of write the markup(plaintext) for the user when editing code snippets, just like what VisualEditor is intended to do (in my opinion). In addition source code needs special formatting that is different from natural written human language, so that's why I think the feature should include automatic indentation and such. For editing just a part of the text, I'm thinking of an interface similar to a pop-up window, obscuring parts of the editing window. Just like when you insert an image the editor will have a new window pop up asking image url, alt text, target, etc. When you click insert the editor will convert your choice in that pop-up window to the plaintext with tags and other markups, which will be visually reflected in the main WYSIWYG editor.

(As mentioned in comment #6 of Bugzilla page of this proposal.)

Relation to CodeEditor

As Matthew Flaschen pointed out, it's necessary to consider the relationship between this plugin and CodeEditor. In some aspects the plugin implements some of the features that already exist in CodeEditor. However, the plugin focuses on providing these features to those who need to insert codes snippets into a general article, as opposed to file editing in CodeEditor.

Moreover, CodeEditor specializes on syntax validity check for JavaScript. This plugin provides syntax validity check that will be language-independent. As mentioned previously in the proposal, the plugin's validity checker focuses on lexical checking. For example a piece of code opened by "{" should be closed later by a "}", and similar rules for "(". The checker is not intended to check whether if the code can actually run, but intended to check whether the code is complete to not cause confusion to readers. Therefore the validity checker here will enforce rules that are commonly applied to coding practices, and it will work even for pseudo-codes. Since it doesn't check everything, it would be much more lightweight than a standard code editor.

It is, however, possible to consult the codes of CodeEditor to build this plugin.

Relation to SyntaxHighlight

The plugin page on Bugzilla has been marked as a dependency of SyntaxHighlight. This editing is accurate as SyntaxHighlight insert codes as-is, and the plugin could make use of it in its graphical user interface, and add its own features.

It is reasonable to maintain the plaintext (for backend) within a ‎<syntaxhighlight> tag, and it is possible to consult the code of SyntaxHighlight.

Other related extensions

Scribunto Embeds Lua in MediaWiki (rather than a code editor?)

Monthly reports

June

• Created basic user interface and basic editing features
• Created mechanisms of highlighter and syntax checker. Created a ruleset for JavaScript as a demo.
• Created auto indenter.
• Various code tweaks for better performance.

July

• Finalized editor features.
  • Auto indenter.
  • Reformatter.
  • Language selection dropdown.
  • Basic editing.
  • Clipboard & edit history support.
  • Search & replace.
• Completed UI components.
• Code fixes.
• Created its own resource loader module.

August

• Some major bug fixes regarding to auto indenter and reformatter.
• Code improvements based on coding practices.
• Documentation and comment.
• Code review.

September

• Code review
• Documentations
• Code cleanup and rebases

Project wrap-up

During the GSoC 2013 that concluded a few days ago I worked particularly on the support of ‎<syntaxhighlight> nodes for VisualEditor. Such a node would be based on parsed context by Parsoid from SyntaxHighlight GeSHi extension ("the extension" mentioned afterwards). As the support being established, right now ‎<syntaxhighlight> nodes can be edited with a light-weight code editor, which acts as one of the tools of VisualEditor. As of the end of September, the editor only reads lang=?? parameter of the extension, which is the only essential parameter that it needs.

Just as a normal light-weight editor, the editor supports basic editing features, clipboard, and edit history. You can also search and replace certain phrases in your code. These functions can be accessed with common shortcut keys. It also has an auto indenter and a reformatter that allows you to format the indentation and whitespaces.

Currently the editor has an internal highlighter that provides highlighting of the code during editing, and a simple syntax checker that provide a basic level of validation of your code. These mechanisms are based on predefined JSON files that corresponds to languages supported by the extension. Since this plugin is still in its early phases, right now only text and JavaScript has predefined files. More files will be added in gradually to expand and eventually match the support to that of the extension. JSON file documentation can be seen here.

Some of the functions are missing in this initial version, and they will be added in future patches. A non-exhaustive list would be:

• Support for adding new ‎<syntaxhighlight> node
• Support for other parameters such as line highlighting and line numbers
• Language supports

Feedbacks, bug reports, suggestions are warmly welcome.

Many thanks to Trevor Parscal, James Forrester, and Roan Kattouw, and many others who helped me along the way on IRC and emails. It has been a great pleasure working with such a team.