Python Bytes to String: Decode Bytes Safely in Python

Python bytes-to-string conversion looks simple until the first UnicodeDecodeError, the first HTTP response with the wrong charset, or the first log line that says b'hello' instead of hello. In Python 3, bytes and str are intentionally different types. Bytes are raw binary data. Strings are Unicode text.

The short answer is data.decode("utf-8"). The safe answer is more specific: use UTF-8 only when the bytes really contain UTF-8 text. If the payload came from a legacy file, a subprocess, an HTTP response, Base64 decoding, a socket, or a binary file, first decide whether the payload is text at all.

data = b"hello"
text = data.decode("utf-8")
print(text)

# Output:
# hello

Bytes vs strings in Python 3

A bytes object is a sequence of integers from 0 to 255. A str is Unicode text. Decoding moves from bytes to string. Encoding moves from string to bytes. The direction matters because the same byte sequence can mean different text in different encodings.

Python makes you choose an encoding because bytes do not carry enough information to prove how they should become characters. UTF-8, Latin-1, Windows-1252, Shift JIS, and other encodings can interpret bytes differently. If you guess wrong, Python may crash, or worse, produce readable-looking text that is still corrupt.

decode() vs str(bytes, encoding) vs str(bytes)

Python documents str(bytes, encoding, errors) as equivalent to decoding bytes-like objects. In application code, .decode() is usually clearer because it says exactly what is happening.

data = b"ByteJSON"

print(data.decode("utf-8"))
print(str(data, "utf-8"))
print(str(data))

# Output:
# ByteJSON
# ByteJSON
# b'ByteJSON'

My default code-review rule is simple: if you want text, use .decode(). If you want a debug view of raw bytes, use repr(data) or a carefully truncated byte preview.

Reproduce and fix UnicodeDecodeError

UnicodeDecodeError usually means you decoded bytes with the wrong encoding, or tried to decode bytes that are not text at all. This example creates Latin-1 bytes and then incorrectly decodes them as UTF-8.

data = "caf\u00e9".encode("latin-1")

print(data)
print(data.decode("utf-8"))

# Typical output:
# b'caf\xe9'
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3

The fix is not to try random encodings until one looks nice. The fix is to identify the source. If the file or system is Latin-1, decode as Latin-1:

data = "caf\u00e9".encode("latin-1")
text = data.decode("latin-1")

print(text.encode("unicode_escape").decode("ascii"))

# Output:
# caf\xe9

Choose an error handler intentionally

The errors argument controls what Python does when decoding fails. This is not just a convenience flag. It is a data integrity decision.

data = b"valid text \xff invalid"

print(data.decode("utf-8", errors="replace").encode("unicode_escape").decode("ascii"))
print(data.decode("utf-8", errors="backslashreplace"))

# Output:
# valid text \ufffd invalid
# valid text \xff invalid

Decode bytes from HTTP responses

Network responses are bytes at the transport layer. A Python HTTP library may decode for you, but low-level APIs such as urllib.request.urlopen().read() return bytes. Decode only after you know the response is text and have a reasonable encoding.

from urllib.request import urlopen

with urlopen("https://example.com/") as response:
    body = response.read()
    encoding = response.headers.get_content_charset() or "utf-8"
    text = body.decode(encoding, errors="replace")

print(text[:120])

Production code needs more care. If the response is compressed, decompress it first. If the response is JSON, UTF-8 is the normal expectation for modern APIs, but check the API documentation. If the server lies about charset, treat that as an integration bug. Use the MIME Types tool when you need to inspect Content-Type and charset clues.

Files: binary mode vs text mode

Decide whether you want bytes or text at the file boundary. For normal text files, text mode with an explicit encoding is clearer. Use binary mode when you are processing real binary files, unknown payloads, hashes, uploads, or formats where only part of the file is text.

with open("message.txt", "rb") as file:
    data = file.read()

text = data.decode("utf-8")
print(text)

with open("message.txt", "r", encoding="utf-8") as file:
    text = file.read()

print(text)

If you find yourself reading every text file in rb mode and decoding later, ask whether you are solving a real boundary problem or just moving the same encoding decision deeper into the code.

Subprocess output: decode manually or ask Python to decode

Captured subprocess output is bytes by default. If the command output is known text, ask subprocess.run to decode it for you.

import subprocess

raw = subprocess.run(
    ["python", "--version"],
    capture_output=True,
)
print(raw.stdout.decode("utf-8", errors="replace"))

text = subprocess.run(
    ["python", "--version"],
    capture_output=True,
    text=True,
    encoding="utf-8",
)
print(text.stdout)

Keep binary mode for commands that return archive content, image data, checksums, protocol frames, or anything documented as bytes. Use text mode for CLI output you would normally read in a terminal.

Base64 decoding returns bytes, not always text

Base64 is a text-safe representation of binary data. Decoding Base64 gives you bytes. Those bytes may contain UTF-8 text, but they may also contain an image, a PDF, compressed data, encrypted data, or random bytes.

import base64

encoded = "SGVsbG8sIEJ5dGVKU09OIQ=="
data = base64.b64decode(encoded)

print(data)
print(data.decode("utf-8"))

# Output:
# b'Hello, ByteJSON!'
# Hello, ByteJSON!

If you are not sure whether a Base64 payload is text, inspect it first with the Base64 Decode tool or view the bytes as hex with Base64 to Hex. A failed Unicode decode may be the correct signal that the payload is not text.

Common sources of bytes and what to check

In production code, naming helps. I prefer variables such as raw_body, body_text, payload_bytes, and decoded_name. That makes it obvious whether a value is still bytes, already decoded text, or a debug representation.

When not to decode bytes

Do not decode bytes just because Python printed a b'...' prefix. Some bytes are not text and should stay bytes.

data = b"\x89PNG\r\n\x1a\n..."

print(len(data))
print(data[:16])
print(data[:16].hex())

# Output:
# 11
# b'\x89PNG\r\n\x1a\n...'
# 89504e470d0a1a0a2e2e2e

Practical debugging checklist

Frequently asked questions

Authoritative references

Use primary Python references when behavior matters:

• Python docs for bytes.decode()
• Python docs for codec error handlers
• Python Unicode HOWTO
• Python base64 module docs
• Python subprocess text and encoding arguments

Related tutorials