Python Tokenize Module
Python tokenizer — tokenize, generate_tokens, untokenize. This category documents 3 Python tokenize module entries, and each page gives you the full syntax, the parameter list, the return value, and a runnable example. The entries include: tokenize.tokenize(), tokenize.generate_tokens(), tokenize.untokenize(). tokenize.tokenize() yields tokens from a Python source file. tokenize.generate_tokens() like tokenize() but works with strings (not bytes). tokenize.untokenize() converts tokens back into Python source code. Tokenize Module belongs to the Dev Tools group alongside Inspect Module, Pdb Module, Traceback Module, Timeit Module. You can also search across every category from the search page, browse the complete A–Z index, or try any example in the Python playground. Python Reference documents 978+ functions, methods, and modules across 114 categories of the standard library, and every entry in this Tokenize Module category follows the same format — syntax first, then what it does, the parameters it accepts, what it returns, and a working code example — so once you have read one page you can read them all.