tokenize.tokenize()
tokenize.tokenize() is a Python function in the Tokenize Module category. Yields tokens from a Python source file. The syntax is tokenize.tokenize(readline). Returns: Generator of TokenInfo objects. A typical example: import tokenize
with open("script.py", "rb") as f:
for tok in tokenize.tokenize(f.readline):
print(tokenize.tok_name[tok.type], tok.string). A close sibling is tokenize.generate_tokens(), which like tokenize() but works with strings (not bytes). A close sibling is tokenize.untokenize(), which converts tokens back into Python source code. More about this category: Python tokenizer — tokenize, generate_tokens, untokenize. Related Tokenize Module entries: tokenize.generate_tokens(), tokenize.untokenize(). This page is part of the free Python Reference documentation covering 978+ functions, methods, and modules with examples and parameter details. You can try tokenize.tokenize() immediately in the Python playground, look up unfamiliar terms in the glossary, or return to the Python Reference homepage.