tokenize.generate_tokens()
tokenize.generate_tokens() is a Python function in the Tokenize Module category. Like tokenize() but works with strings (not bytes). The syntax is tokenize.generate_tokens(readline, extra_tokens=False). Returns: Generator of TokenInfo objects. A typical example: import io, tokenize
code = "x = 42\nprint(x)"
tokens = tokenize.generate_tokens(io.StringIO(code).readline)
for tok in tokens:
print(tok.string). A close sibling is tokenize.tokenize(), which yields tokens from a Python source file. A close sibling is tokenize.untokenize(), which converts tokens back into Python source code. More about this category: Python tokenizer — tokenize, generate_tokens, untokenize. Related Tokenize Module entries: tokenize.tokenize(), tokenize.untokenize(). This page is part of the free Python Reference documentation covering 978+ functions, methods, and modules with examples and parameter details. You can try tokenize.generate_tokens() immediately in the Python playground, look up unfamiliar terms in the glossary, or return to the Python Reference homepage.