NAME
Go::Tokenize - Tokenize Go
SYNOPSIS
use Go::Tokenize;VERSION
This documents version 0.01 of Go-Tokenize corresponding to git commit 62708e18afa5676b39aff68935df240bd51d02c9 released on Wed Mar 10 19:10:53 2021 +0900.
DESCRIPTION
Turn Go code into tokens.
FUNCTIONS
tokenize
my $tokens = tokenize ($go);Given your Go program text in $go, turn it into tokens. The return value is an array reference which contains hash references. The hash references have the following keys
- contents
-
The text content of the token.
- end
-
The position of the final character of the token, in characters. See "start" for the numbering. This is the value of pos as the regex used for matching matches components.
- line
-
The line number of the start position of the token, numbered from 1.
- start
-
The position of the beginning character of the token, in characters. This is numbered from 0 for the first character of the file. If you read the Go file in using
:encoding(utf8)and the file contains non-ASCII characters, this and "end" will be different from the byte offsets. Also, because this is the position of the character, if the token is one character in length, its value will be the same as "end". In other words the length of the token is (end - start + 1). - type
-
The type of the token. Currently the following are supported:
- comment
- keyword
- operator
- string
DEPENDENCIES
- C::Tokenize
-
This is used because Go comments are identical to C comments.
- Text::LineNumber
AUTHOR
Ben Bullock, <bkb@cpan.org>
COPYRIGHT & LICENCE
This package and associated files are copyright (C) 2021 Ben Bullock.
You can use, copy, modify and redistribute this package and associated files under the Perl Artistic Licence or the GNU General Public Licence.