Tokenization is an essential text preprocessing step in almost all large language models (LLMs), and byte-pair encoding (BPE) is a popular tokenization method used by models such as GPT, GPT-2, and RoBERTa. Since LLMs have many applications that require the fast processing of large amounts of data (e.g., real-time document summarization and analysis), offline tokenization algorithms may be unsuitable or at least impose a prohibitive memory cost. In this paper, we study BPE tokenization with a focus on the streaming model. We view BPE tokenization as a sequence-to-sequence transduction and obtain a bound on the delay of a BPE ruleset that enables streaming tokenization. We propose a novel streaming algorithm that uses a small amount of memory (independent of the input text) and has linear time complexity in the length of the input text. Our experimental evaluation shows that our algorithm performs well in comparison to existing BPE tokenizers.