URL-safe Base64, hex encoding, and raw random bytes are three different output formats for the same underlying randomness — and choosing the wrong format for the context is one of the most common causes of "invalid token" errors in authentication systems
The previous articles on this site covered token length and the birthday problem, why Math.random() fails for security, UUID v4 vs ULID vs NanoID, and the test data vs security token distinction. This article addresses token output format selection — specifically which encoding format to use for which context, why the choice matters, and the specific characters that break different systems.
The three common formats for random token output
Raw bytes: the underlying random data as binary — 32 bytes of randomness is the most compact representation but not printable or URL-safe. Used in low-level APIs and when passing tokens between systems that handle binary data natively.
Hex encoding: converts each byte to two hexadecimal characters (0-9, a-f). 32 random bytes → 64 hex characters. The character set is tiny (16 characters), making hex tokens universally safe in any context — URLs, JSON, database columns, HTTP headers — without escaping. Slightly longer than Base64 for the same entropy.
Base64 and Base64url: converts every 3 bytes to 4 characters using a 64-character alphabet. 32 bytes → approximately 44 Base64 characters (with padding). More compact than hex but the standard Base64 alphabet includes +, /, and = — all of which are special characters in URLs, query strings, and some database contexts.
Base64url: a variant of Base64 that replaces + with - and / with _, and omits = padding. Safe for URLs without percent-encoding.
Where standard Base64 tokens break
The + and / characters in standard Base64 cause problems in specific contexts:
URL query strings: + means "space" in application/x-www-form-urlencoded context. A Base64 token aB+cD/eF== in a URL query parameter would have the + decoded as a space by most web frameworks, corrupting the token. The / is also a path separator in URLs — though this typically only causes problems in path parameters, not query strings.
HTTP Basic Auth: the Base64 encoded credential in Authorization: Basic ... uses standard Base64, where / and + are fine — because the entire value is treated as an opaque Base64-encoded string, not parsed as a URL.
Cookie values: some cookie implementations restrict special characters. Tokens used as cookie values should use hex or Base64url, not standard Base64.
Database VARCHAR columns with case-insensitive collation: this only affects the token's distinctness, not syntax — but ABC and abc being treated as the same value in a case-insensitive column means tokens that differ only in case could collide. Hex tokens are inherently lowercase (a-f, 0-9), making them safe for case-insensitive columns.
Choosing by context: a decision table
HTTP Authorization Bearer token: Base64url or hex — both are safe in HTTP header values. JWTs use Base64url natively (the three sections are Base64url-encoded).
URL-embedded token (password reset link, email verification): Base64url or hex — both are URL-safe. Avoid standard Base64. Example: https://example.com/verify?token=aB-cD_eF (Base64url) or https://example.com/verify?token=1a2b3c4d (hex).
Session cookie: hex or Base64url — both safe for cookie values.
Database storage: any format works, but be consistent. Hex is often preferred for readability (hex tokens are visually similar strings; Base64url tokens have more visual variation). Store as VARCHAR with a length matching the encoding's output (64 chars for 32-byte hex, 43 chars for 32-byte Base64url without padding).
API key displayed to a user once at creation: all formats are readable, but different conventions apply: GitHub uses Base64url-style tokens, AWS uses a custom Base62 alphabet (A-Z, a-z, 0-9 only, like UUID but shorter), Stripe uses prefixes (sk_live_, pk_test_) followed by random characters. The convention choice matters more for user experience than for security.
Entropy comparison across formats
32 bytes of random data has 256 bits of entropy regardless of format — hex encoding doesn't add or remove entropy; it just changes the representation. The format choice affects:
- Length: 32 bytes → 64 hex chars → 43 Base64url chars → shorter but same entropy
- Character set safety: hex is universally safe; Base64url is safe for URLs but not for all contexts; standard Base64 requires escaping in URLs
- Readability: hex is slightly easier to compare visually (two tokens differ at position 12: is it the 12th hex pair or the 12th base64 quartet?)
For a given token length in characters (not bytes), the formats differ in security:
- 40 hex characters: 20 bytes = 160 bits of entropy
- 40 Base64url characters: approximately 30 bytes = 240 bits of entropy
- 40 alphanumeric characters (0-9, a-z, A-Z): log₂(62^40) ≈ 238 bits
So for the same character count, Base64url and alphanumeric tokens carry more entropy than hex tokens. For a token of fixed display length (e.g., an API key displayed as 32 characters), Base64url is more efficient than hex.
Token prefixes: why some tokens start with a readable prefix
Prefixed tokens (like sk_live_xxxxxx for Stripe or ghp_xxxxxx for GitHub) combine a human-readable prefix with random characters:
Benefits:
- Immediately identifiable: security scanners can detect accidentally committed tokens by scanning for known prefixes
- Self-documenting: a token that starts with
sk_live_is obviously a Stripe live secret key - Revocable by prefix: revoking all tokens starting with a specific prefix is straightforward
Implementation: the prefix is just prepended to the randomly generated characters — sk_live_ + 32 random Base62 characters = sk_live_Kxa9mBPqz1.... The prefix doesn't affect the entropy of the random portion; it does increase the total token length.
How to use the Random String Generator on sadiqbd.com
- Select the encoding format for your context: hex for universal compatibility; Base64url for shorter tokens in URL contexts; alphanumeric for maximum readability
- Choose byte length, not character length, for security tokens: 16 bytes (128 bits) is the minimum for most security tokens; 32 bytes (256 bits) for high-security applications. The character output length follows from the encoding
- For test data generation: character count matters more than encoding — generate strings of the length your system expects, in the format it accepts, to test field limits and display behavior
Frequently Asked Questions
Should I store tokens in the database as-is, or hash them first? Hash them — store only the hash, never the raw token. If your database is breached, raw tokens are immediately usable by attackers. Hashing tokens before storage means a database breach doesn't expose valid tokens. Use a fast hash (SHA-256) for token storage — not bcrypt (which is designed for passwords and adds unnecessary slow-hashing overhead for tokens, which don't need brute-force resistance if they're long enough). When verifying a token, hash the provided token and compare to the stored hash.
Is the Random String Generator free? Yes — completely free, no sign-up required.
Try the Random String Generator free at sadiqbd.com — generate cryptographically secure random strings with custom length and character sets.
