tanmay/Regolith
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add repo-tokens GitHub Action with token count badge

Browse Source 
Reusable composite action that counts codebase tokens using tiktoken
and generates a shields.io-style SVG badge. Color reflects context
window usage: green (<30%), yellow-green (30-50%), yellow (50-70%),
red (70%+). Badge includes hardcoded link back to repo-tokens.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
gavrielc
2026-02-15 01:41:14 +02:00
parent 6f2e10f0c3
commit c8ab3d95e1
9 changed files with 408 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

105
repo-tokens/README.md Normal file
View File

@@ -0,0 +1,105 @@
# Repo Tokens
A GitHub Action that calculates the size of your codebase in terms of tokens and updates a badge in your README.
<p>
<img src="examples/green.svg" alt="tokens 12.4k">&nbsp;
<img src="examples/yellow-green.svg" alt="tokens 74.8k">&nbsp;
<img src="examples/yellow.svg" alt="tokens 120k">&nbsp;
<img src="examples/red.svg" alt="tokens 158k">
</p>
## Usage
```yaml
- uses: qwibitai/nanoclaw/repo-tokens@v1
with:
include: 'src/**/*.ts'
exclude: 'src/**/*.test.ts'
```
This counts tokens using [tiktoken](https://github.com/openai/tiktoken) and writes the result between HTML comment markers in your README:
The badge color reflects what percentage of an LLMs context window the codebase fills (context window size is configurable, defaults to 200k which is the size of Claude Opus). Green for under 30%, yellow-green for 30%-50%, yellow for 50%-70%, red for 70%+.
## Why
Small codebases were always a good thing. With coding agents, there's now a huge advantage to having a codebase small enough that an agent can hold the full thing in context.
This badge gives some indication of how easy it will be to work with an agent on the codebase, and will hopefully be a visual reminder to avoid bloat.
### Full workflow example
```yaml
name: Update token count
on:
push:
branches: [main]
paths: ['src/**']
permissions:
contents: write
jobs:
update-tokens:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- uses: qwibitai/nanoclaw/repo-tokens@v1
id: tokens
with:
include: 'src/**/*.ts'
exclude: 'src/**/*.test.ts'
badge-path: '.github/badges/tokens.svg'
- name: Commit if changed
run: |
git add README.md .github/badges/tokens.svg
git diff --cached --quiet && exit 0
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git commit -m "docs: update token count to ${{ steps.tokens.outputs.badge }}"
git push
```
### README setup
Add markers where you want the token count text to appear:
```html
<!-- token-count --><!-- /token-count -->
```
The action replaces everything between the markers with the token count.
## Inputs
| Input | Default | Description |
|-------|---------|-------------|
| `include` | *required* | Glob patterns for files to count (space-separated) |
| `exclude` | `''` | Glob patterns to exclude (space-separated) |
| `context-window` | `200000` | Context window size for percentage calculation |
| `readme` | `README.md` | Path to README file |
| `encoding` | `cl100k_base` | Tiktoken encoding name |
| `marker` | `token-count` | HTML comment marker name |
| `badge-path` | `''` | Path to write SVG badge (empty = no SVG) |
## Outputs
| Output | Description |
|--------|-------------|
| `tokens` | Total token count (e.g., `34940`) |
| `percentage` | Percentage of context window (e.g., `17`) |
| `badge` | The formatted text that was inserted (e.g., `34.9k tokens · 17% of context window`) |
## How it works
Composite GitHub Action. Installs tiktoken, runs ~60 lines of inline Python. Takes about 10 seconds.
The action counts tokens and updates the README but does not commit. Your workflow decides the git strategy.

186
repo-tokens/action.yml Normal file
View File

@@ -0,0 +1,186 @@
name: Repo Tokens
description: Count codebase tokens with tiktoken and update a README badge
inputs:
include:
description: 'Glob patterns for files to count (space-separated)'
required: true
exclude:
description: 'Glob patterns to exclude (space-separated)'
required: false
default: ''
context-window:
description: 'Context window size for percentage calculation'
required: false
default: '200000'
readme:
description: 'Path to README file'
required: false
default: 'README.md'
encoding:
description: 'Tiktoken encoding name'
required: false
default: 'cl100k_base'
marker:
description: 'HTML comment marker name'
required: false
default: 'token-count'
badge-path:
description: 'Path to write SVG badge (empty = no SVG)'
required: false
default: ''
outputs:
tokens:
description: 'Total token count'
value: ${{ steps.count.outputs.tokens }}
percentage:
description: 'Percentage of context window'
value: ${{ steps.count.outputs.percentage }}
badge:
description: 'Badge text that was inserted'
value: ${{ steps.count.outputs.badge }}
runs:
using: composite
steps:
- name: Install tiktoken
shell: bash
run: pip install tiktoken
- name: Count tokens and update README
id: count
shell: python
env:
INPUT_INCLUDE: ${{ inputs.include }}
INPUT_EXCLUDE: ${{ inputs.exclude }}
INPUT_CONTEXT_WINDOW: ${{ inputs.context-window }}
INPUT_README: ${{ inputs.readme }}
INPUT_ENCODING: ${{ inputs.encoding }}
INPUT_MARKER: ${{ inputs.marker }}
INPUT_BADGE_PATH: ${{ inputs.badge-path }}
run: |
import glob, os, re, tiktoken
include_patterns = os.environ["INPUT_INCLUDE"].split()
exclude_patterns = os.environ["INPUT_EXCLUDE"].split()
context_window = int(os.environ["INPUT_CONTEXT_WINDOW"])
readme_path = os.environ["INPUT_README"]
encoding_name = os.environ["INPUT_ENCODING"]
marker = os.environ["INPUT_MARKER"]
badge_path = os.environ.get("INPUT_BADGE_PATH", "").strip()
# Expand globs
included = set()
for pattern in include_patterns:
included.update(glob.glob(pattern, recursive=True))
excluded = set()
for pattern in exclude_patterns:
excluded.update(glob.glob(pattern, recursive=True))
files = sorted(included - excluded)
files = [f for f in files if os.path.isfile(f)]
# Count tokens
enc = tiktoken.get_encoding(encoding_name)
total = 0
for path in files:
try:
with open(path, "r", encoding="utf-8", errors="ignore") as f:
total += len(enc.encode(f.read()))
except Exception as e:
print(f"Skipping {path}: {e}")
# Format
if total >= 100000:
display = f"{round(total / 1000)}k"
elif total >= 1000:
display = f"{total / 1000:.1f}k"
else:
display = str(total)
pct = round(total / context_window * 100)
badge = f"{display} tokens \u00b7 {pct}% of context window"
print(f"Files: {len(files)}, Tokens: {total}, Badge: {badge}")
# Update README (text between markers)
marker_re = re.compile(
rf"(<!--\s*{re.escape(marker)}\s*-->).*?(<!--\s*/{re.escape(marker)}\s*-->)",
re.DOTALL,
)
with open(readme_path, "r", encoding="utf-8") as f:
content = f.read()
repo_tokens_url = "https://github.com/qwibitai/nanoclaw/tree/main/repo-tokens"
linked_badge = f'<a href="{repo_tokens_url}">{badge}</a>'
new_content = marker_re.sub(rf"\1{linked_badge}\2", content)
if new_content != content:
with open(readme_path, "w", encoding="utf-8") as f:
f.write(new_content)
print("README updated")
else:
print("No change to README")
# Generate SVG badge
if badge_path:
label_text = "tokens"
value_text = display
full_desc = f"{display} tokens, {pct}% of context window"
cw = 7.0
label_w = round(len(label_text) * cw) + 10
value_w = round(len(value_text) * cw) + 10
total_w = label_w + value_w
if pct < 30:
color = "#4c1"
elif pct < 50:
color = "#97ca00"
elif pct < 70:
color = "#dfb317"
else:
color = "#e05d44"
lx = label_w // 2
vx = label_w + value_w // 2
repo_tokens_url = "https://github.com/qwibitai/nanoclaw/tree/main/repo-tokens"
svg = f'''<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="{total_w}" height="20" role="img" aria-label="{full_desc}">
<title>{full_desc}</title>
<linearGradient id="s" x2="0" y2="100%">
<stop offset="0" stop-color="#bbb" stop-opacity=".1"/>
<stop offset="1" stop-opacity=".1"/>
</linearGradient>
<clipPath id="r">
<rect width="{total_w}" height="20" rx="3" fill="#fff"/>
</clipPath>
<a xlink:href="{repo_tokens_url}">
<g clip-path="url(#r)">
<rect width="{label_w}" height="20" fill="#555"/>
<rect x="{label_w}" width="{value_w}" height="20" fill="{color}"/>
<rect width="{total_w}" height="20" fill="url(#s)"/>
<g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="11">
<text aria-hidden="true" x="{lx}" y="15" fill="#010101" fill-opacity=".3">{label_text}</text>
<text x="{lx}" y="14">{label_text}</text>
<text aria-hidden="true" x="{vx}" y="15" fill="#010101" fill-opacity=".3">{value_text}</text>
<text x="{vx}" y="14">{value_text}</text>
</g>
</g>
</a>
</svg>'''
os.makedirs(os.path.dirname(badge_path) or ".", exist_ok=True)
with open(badge_path, "w", encoding="utf-8") as f:
f.write(svg)
print(f"Badge SVG written to {badge_path}")
# Set outputs
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
f.write(f"tokens={total}\n")
f.write(f"percentage={pct}\n")
f.write(f"badge={badge}\n")

23
repo-tokens/badge.svg Normal file
View File

@@ -0,0 +1,23 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="97" height="20" role="img" aria-label="34.9k tokens, 17% of context window">
<title>34.9k tokens, 17% of context window</title>
<linearGradient id="s" x2="0" y2="100%">
<stop offset="0" stop-color="#bbb" stop-opacity=".1"/>
<stop offset="1" stop-opacity=".1"/>
</linearGradient>
<clipPath id="r">
<rect width="97" height="20" rx="3" fill="#fff"/>
</clipPath>
<a xlink:href="https://github.com/qwibitai/nanoclaw/tree/main/repo-tokens">
<g clip-path="url(#r)">
<rect width="52" height="20" fill="#555"/>
<rect x="52" width="45" height="20" fill="#4c1"/>
<rect width="97" height="20" fill="url(#s)"/>
<g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="11">
<text aria-hidden="true" x="26" y="15" fill="#010101" fill-opacity=".3">tokens</text>
<text x="26" y="14">tokens</text>
<text aria-hidden="true" x="74" y="15" fill="#010101" fill-opacity=".3">34.9k</text>
<text x="74" y="14">34.9k</text>
</g>
</g>
</a>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.1 KiB

14
repo-tokens/examples/green.svg Normal file
View File

@@ -0,0 +1,14 @@
<svg xmlns="http://www.w3.org/2000/svg" width="97" height="20" role="img" aria-label="12.4k tokens">
<title>12.4k tokens</title>
<linearGradient id="s" x2="0" y2="100%"><stop offset="0" stop-color="#bbb" stop-opacity=".1"/><stop offset="1" stop-opacity=".1"/></linearGradient>
<clipPath id="r"><rect width="97" height="20" rx="3"/></clipPath>
<g clip-path="url(#r)">
<rect width="52" height="20" fill="#555"/>
<rect x="52" width="45" height="20" fill="#4c1"/>
<rect width="97" height="20" fill="url(#s)"/>
<g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="11">
<text x="26" y="15" fill="#010101" fill-opacity=".3">tokens</text><text x="26" y="14">tokens</text>
<text x="74" y="15" fill="#010101" fill-opacity=".3">12.4k</text><text x="74" y="14">12.4k</text>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 867 B

14
repo-tokens/examples/red.svg Normal file
View File

@@ -0,0 +1,14 @@
<svg xmlns="http://www.w3.org/2000/svg" width="90" height="20" role="img" aria-label="158k tokens">
<title>158k tokens</title>
<linearGradient id="s" x2="0" y2="100%"><stop offset="0" stop-color="#bbb" stop-opacity=".1"/><stop offset="1" stop-opacity=".1"/></linearGradient>
<clipPath id="r"><rect width="90" height="20" rx="3"/></clipPath>
<g clip-path="url(#r)">
<rect width="52" height="20" fill="#555"/>
<rect x="52" width="38" height="20" fill="#e05d44"/>
<rect width="90" height="20" fill="url(#s)"/>
<g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="11">
<text x="26" y="15" fill="#010101" fill-opacity=".3">tokens</text><text x="26" y="14">tokens</text>
<text x="71" y="15" fill="#010101" fill-opacity=".3">158k</text><text x="71" y="14">158k</text>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 866 B

14
repo-tokens/examples/yellow-green.svg Normal file
View File

@@ -0,0 +1,14 @@
<svg xmlns="http://www.w3.org/2000/svg" width="97" height="20" role="img" aria-label="74.8k tokens">
<title>74.8k tokens</title>
<linearGradient id="s" x2="0" y2="100%"><stop offset="0" stop-color="#bbb" stop-opacity=".1"/><stop offset="1" stop-opacity=".1"/></linearGradient>
<clipPath id="r"><rect width="97" height="20" rx="3"/></clipPath>
<g clip-path="url(#r)">
<rect width="52" height="20" fill="#555"/>
<rect x="52" width="45" height="20" fill="#97ca00"/>
<rect width="97" height="20" fill="url(#s)"/>
<g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="11">
<text x="26" y="15" fill="#010101" fill-opacity=".3">tokens</text><text x="26" y="14">tokens</text>
<text x="74" y="15" fill="#010101" fill-opacity=".3">74.8k</text><text x="74" y="14">74.8k</text>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 870 B

14
repo-tokens/examples/yellow.svg Normal file
View File

@@ -0,0 +1,14 @@
<svg xmlns="http://www.w3.org/2000/svg" width="90" height="20" role="img" aria-label="120k tokens">
<title>120k tokens</title>
<linearGradient id="s" x2="0" y2="100%"><stop offset="0" stop-color="#bbb" stop-opacity=".1"/><stop offset="1" stop-opacity=".1"/></linearGradient>
<clipPath id="r"><rect width="90" height="20" rx="3"/></clipPath>
<g clip-path="url(#r)">
<rect width="52" height="20" fill="#555"/>
<rect x="52" width="38" height="20" fill="#dfb317"/>
<rect width="90" height="20" fill="url(#s)"/>
<g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="11">
<text x="26" y="15" fill="#010101" fill-opacity=".3">tokens</text><text x="26" y="14">tokens</text>
<text x="71" y="15" fill="#010101" fill-opacity=".3">120k</text><text x="71" y="14">120k</text>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 866 B