vhaudiquet/deb-lite-xl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,865 Commits 1 Branch 1 Tag
c947e8a4d1a40a04fcd26b3728a5e6a4b3dfcef0
Commit Graph
21 Commits
Author SHA1 Message Date
Guldoman c947e8a4d1 Convert more byte offsets to utf-8 pos in regex tokenizer 2022-06-12 02:55:36 +02:00
Guldoman d8efb1ab53 Show error if language plugin pattern has mismatching number of groups 
The number of results from a pattern with groups must never be greater
than the number of token types for that pattern.

Also if a token type was undefined, it's now pushed as a `normal` one.
 2022-05-31 02:05:37 +02:00
Guldoman 7ac776bef6 Fix UTF-8 matches in regex group tokenizer 2022-05-31 01:59:14 +02:00
Guldoman 2a41002355 Allow using regex groups to split tokens 
Before, this was only supported by Lua patterns.

This expects the regex to use the same syntax used for patterns. That 
is, the token should be split by empty groups.
 2022-05-28 01:38:22 +02:00
jgmdev 94430bcbd2 tokenizer: fix next utf8 char retrieval bug 2022-05-13 11:21:46 -04:00
Jefferson GonzálezandGitHub e572c58f24 Add utf8 support to tokenizer (#945) 
* add utf8 support to tokenizer

* wrap utf8 functions in string table using a 'u' prefix

* document new utf8 functions
 2022-04-26 09:42:02 -04:00
Guldoman caefc9112a Force syntax patterns starting with ^ to match with the whole line 
Before, syntax patterns/regexes that started with `^` didn't have the 
desired effect of matching with the start of the line.

Now those patterns are used only when matching the whole line.
 2022-03-04 11:27:01 +01:00
Guldoman 51975472a9 Add bit32 polyfill globally 2022-01-12 00:07:53 +01:00
Jan200101 99ddf1fb92 Migrate to Lua 5.4 2021-12-31 13:53:01 +01:00
Guldoman 4faaf089ef Consume unmatched character correctly 
We must consume the whole UTF-8 character, not just a single byte.
 2021-12-11 03:43:33 +01:00
Adam Harrison 96db380c73 Manual merge of into . 2021-11-23 15:57:22 -05:00
Francesco Abbate 5cdd800910 Fix problem checking utf-8 cont at end of string 2021-10-23 15:03:09 +02:00
Guldoman 8a516d35ce Correctly identify the start of the next character in tokenizer 
When moving to the next character, we have to consider that the current 
one might be multi-byte.
 2021-10-11 22:37:31 +02:00
takase1121 30ccde896d replace unpack() with table.unpack() 
I have no idea unpack() is still used and how it still worked.
 2021-08-29 09:14:12 +08:00
AdamandGitHub 248d70a8ca Add PCRE to support regular expressions 
Use regular expressions instead of Lua patterns for find and replace editor commands.

Syntax files can now use regex or Lua patterns as before keeping backward compatibility for plugins.
 2021-06-02 21:27:00 +02:00
AdamandGitHub 949692860e Tokenizer cleanup (#198) 
* Cleaned up tokenizer to make subsyntax operations more clear.

* Explanatory comments.

* Made it so push_subsyntax could be safely called elsewhere.

* Unified terminology.

* Minor bug fix.

* State is an incredibly vaguely named variable. Changed convention to represent what it actually is.

* Also changed function name.

* Fixed bug.
 2021-05-20 21:58:27 +02:00
liquidevandGitHub 86a7037ed9 support for multiple groups in one pattern (#196) 2021-05-19 22:35:28 +02:00
lqdev ba4fbde33d fixed mixed indentation 2021-05-18 17:52:18 +02:00
adamharrisonandGitHub 3fe6665b9a Nested Syntax Highlighting (#160) 2021-05-01 11:45:30 +02:00
rxi 6525269386 Made tokenizer skip parsing process on plain-text files 
This, along with the earlier rencache changes should resolve #64
 2020-05-14 10:10:50 +01:00
rxi f5025efbb8 Moved highlighter code from DocView to Doc 
* Only one highlighter state is kept per-document as opposed
  to one per-docview
* Fixes a bug with retaining older highlighter state as a
  DocView wasn't able to detect lines changing above it's viewport
* Renames `highlighter` module to more descriptive `tokenizer`
 2020-05-07 21:14:46 +01:00