Package cssutils :: Module tokenize2 :: Class Tokenizer
[hide private]
[frames] | no frames]

Class Tokenizer

source code

object --+
         |
        Tokenizer

generates a list of Token tuples:
(Tokenname, value, startline, startcolumn)


Instance Methods [hide private]
 
__init__(self, macros=None, productions=None)
inits tokenizer with given macros and productions which default to cssutils own macros and productions
source code
 
_expand_macros(self, macros, productions)
returns macro expanded productions, order of productions is kept
source code
 
_compile_productions(self, expanded_productions)
compile productions into callable match objects, order is kept
source code
 
tokenize(self, text, linesep=None, fullsheet=False)
(tokenname, tokenvalue, line, col)
source code

Inherited from object: __delattr__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __str__

Class Variables [hide private]
  nl = '\r\n'
Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, macros=None, productions=None)
(Constructor)

source code 
inits tokenizer with given macros and productions which default to cssutils own macros and productions
Overrides: object.__init__

tokenize(self, text, linesep=None, fullsheet=False)

source code 
generator: tokenizes text and yiels tokens, each token is a tuple of
(tokenname, tokenvalue, line, col)
text
to be tokenized
linesep
used to detect the linenumber, defaults to os.linesep
fullsheet
if True appends EOF token as last one and completes incomplete COMMENT tokens