Syntax highlighting nightmare
Two days ago I got, what is probably the most serious bug report so far for Padre. As it turns out while we are trying to build the best Perl IDE on Earth or whatever big title, we cannot even parse perl 5.10 correctly. Specifically it does not know what to do with the new defined-or // operator.
By chance just a few days earlier I wrote down the version numbers of our major non-perl dependencies, that is wxWidgets and Scintilla. This information now helped me to locate the problem and try to evaluate how can we solve it.
The latest version of [dist://Alien::wxWidgets] on CPAN is 0.40 released on Oct 15, 2008. Based on this file I understand that it is using version 2.8.8 of wxWidgets. Checking the contrib/src/stc/scintilla/README.txt file in the wxWidgets 2.8.8 source code reveals that it uses Scintilla 1.70.
According to the wxWidgets web site currently the latest version of wxWidgets is 2.8.9 that was released on Sept 22, 2008. Based on file same README file we can find out that it also uses version 1.70 of Scintilla.
According to Scintilla web site the latest version of Scintilla is 1.77. It was released on Oct 18, 2008 while 1.70 was released 20 June 2006, more than 2 years ago.
Our problem is that perl 5.10 came out on Dec 18, 2007 and Scintilla included support for the new syntax only in version 1.76. (See the Scintilla History.
Solution 1
So the immediate idea was to check how can we get the latest Scintilla.
I checked the wxWidgtes development, and in their trunk, they indeed have a newer version of Scintilla. According to README in the trunk of wxWidgtes, it currently contains Scintilla 1.75. Argh.
That's still too old. In order for Padre to support perl 5.10, first Scintilla 1.77 needs to be integrated into wxWidgets. Then we have to wait for the next release of wxWidgetes and then wait some more till wxPerl catches on. By the time all this is done, a newer version of Perl will come out with yet another keyword or construct and we are again in a awful big delay.
Solution 2
We could also sidestep the wxWidgets integration of Scintilla and integrate it ourself but that means an awful lot of XS work that none of us likes and we are still dependent on Scintilla.
Solution 3 - write our own syntax highlighter
In the end no matter how good these syntax highlighters are, there are still going to be cases when they fail. After all only perl can parse Perl.
Well, except of course [dist://PPI].
Indeed, yesterday morning, just a few hours after we discussed the situation on #padre, Fayland Lam has already implemented a solution.
Of course it was slow as Adam Kennedy promised but it worked.
Some tweaking and now it is available as an experimental feature.
TODO
We still need to solve the speed issue though.
A quick analysis using [dist://Devel::NYTProf] shows that the biggest time consumption is PPI analysing the file.
Two directions where people might want to help:
- Implement more of [dist://PPI::XS] to make PPI fast.
- Change PPI so it can parse only part of the file and then restart later from that point.
Other ideas are also welcome.