
One RTX 2060 runs a 35B model: an anti-scam blogger's inference engine, a Threads post added 171 stars in a day
What it is A 3.8MB local LLM inference program for Windows that caches the experts commonly used in large MoE models on the GPU, letting an older 8GB NVIDIA card run 35B models. It claims to be twice as fast as llama.cpp and also includes a desktop agent that can do tasks and chat by voice.
Why it blew up The author, behind the anti-scam bot PenipuMY, built about 3,900 followers on Threads. The day after a major release he posted a screen recording titled "Claude-distilled 35B on RTX 2060 8GB, 70 tok/s". It got 128k views and +171 stars that day. Nobody on Reddit, HN or X picked it up.
CHow you could follow it Don't build another engine that claims to be "2x faster than llama.cpp". Copy the posting method instead: old hardware, a concrete number, and an uncut recording, posted right after a major release, in places it skipped, like r/LocalLLaMA and Show HN. If you want a product, fill the gaps it left: Linux, AMD GPUs, or a small tool that tells you what models your card can run.
- Data (updated daily) · Repo created 2026-10-27 · 243 stars · 7 days +197
GitHub · stars Data 2026-11-05 → 2026-11-20 (16 days, 16 observations). Points before 10-28 come from deeply / archive backfill, not every day has one
Malaysian developer Apiz Pudin wrote ghost-infer, a local LLM inference engine in Rust that only supports NVIDIA GPUs and only runs on Windows. 11-14 He posted a screen recording on Threads, and the repo went from 46 to 217 stars in a single day. The code has substance, but nearly all of these two hundred-odd stars came from his own social account, and they dropped back to single digits a day by the next day. Below: what it is, where the stars came from, how crowded this space is, and what you should learn if you want a project that earns stars the same way.
A screen recording titled "run Claude locally"
Nov 13 22:08 (UTC, which was already 6 a.m. on the 14th in Malaysia), ghost-infer shipped v0.4.0, and the speed in the README went from about 45 tok/s to 80. A few hours later the author posted a video on Threads (@apizpudin) with a one-line title:
Running Claude-distilled 35B on RTX 2060 8GB GPU, at 70 tok/s.
The video is a real-time screen recording of a six-year-old gaming GPU generating text locally. The post got 128k views, 1.1K likes, 145 comments and 268 shares (Threads page, 11-15). The repo link sat in the author's own first reply, which got 82 likes.
That day the repo went from 46 stars to 217, and forks went from 3 to 13. Most of the new fork accounts look like real Southeast Asian users, and the first one appeared shortly after the post (GitHub, 11-15). The v0.4.0 installer was downloaded 159 times, while the previous version v0.3.0 had only 9 downloads. 11-15 then added just 2 more.
The most common questions in the comments were of two kinds. One was "claude?? what do you mean" and "can this run Claude locally?" The author answered plainly: "Claude by looks, Qwen is the content." What actually runs is a community Qwen3.5-35B-A3B distilled from Claude Opus, and the demo uses an uncensored version. The other kind was "will it run on my 4050 6GB?" and "what about an RX 5700XT?" He replied to each one in Indonesian, Russian and Italian.
What it actually is
A 3.8MB single Rust binary that reads GGUF models. It needs no Python, no Docker and no CUDA toolkit, exposes an OpenAI-compatible API, and ships with a chat page featuring a little ghost mascot. It only supports NVIDIA GPUs, and only a Windows build exists so far.
It is fast at one specific thing. When a large MoE model (35B total parameters, roughly 3B active per token) doesn't fit in 8GB of VRAM, attention layers stay on the GPU and expert weights stay in system RAM. The "hot experts" that keep getting reused in a reply are cached on the GPU, and the rest are computed on the CPU in parallel. Each layer runs as a CUDA graph, so the GPU picks the experts itself. The README benchmark (RTX 2060 SUPER 8GB plus 24GB RAM, 11-14, with a reproduction script attached):
| Model | ghost-infer | llama.cpp | Ollama |
|---|---|---|---|
| Qwen3.6-35B-A3B | 80.5 | 39.5 | 34.7 |
| Qwen3-30B-A3B | 68.7 | 29.5 | 26.9 |
| Qwen3 8B | 75.3 | 64.7 | 65.8 |
Units are tok/s. Only the MoE case where the model is bigger than VRAM shows the two-times gap. When the 8B model fits in VRAM, it is only about 15% faster. These numbers were measured on one machine by the author, with no third-party reproduction, and it's unknown whether llama.cpp's settings were tuned as well.
On top of the engine there's a desktop app: an agent that can read and write files and run commands, a voice call with a companion character named Maya (a voice the author fine-tuned himself), and an experimental "sleep training" feature that trains the day's conversation facts into a small model each night. The UI is bilingual in English and Chinese. The Chinese was translated by a contributor.
The repo was created on 10-27. The author made 138 commits alone and shipped 10 releases in half a month. His GitHub account has only 2 followers, and this is his first public large project.
Where the stars came from
His own account, and it wasn't built on tech content. @apizpudin's bio reads "Anti-Scam Activist." His pinned post is about PenipuMY, a Telegram bot he made in Malaysia to check for scams. That post has 11.8k likes, 11-02 and he also posted that PenipuMY was covered in the newspaper. He has about 3,900 followers, mostly Malay speakers and Southeast Asian audiences. Each of the two star jumps matches one of his posts: 11-03 the first one, "Running 35B model locally on 8GB VRAM & 16GB RAM," got 386 likes, and the repo stalled at 46 stars for a week. 11-14 The second one is the post above.
The title included "Claude." The two posts covered the same thing, and the one with Claude in the title got about 3 times the likes of the earlier one. Many people who clicked assumed they could run Claude locally, and the author admitted this in his replies.
The numbers were well chosen. RTX 2060, 8GB, 35B, 70 tok/s. Each figure is something an ordinary player can relate to: an older card, small VRAM, a big model, and a fast result. That is much easier to share than the benchmark table in the README.
The post came right after a major release. He first brought the speed from 45 to 80, then posted the new numbers the next day, so the traffic landed on the new installer.
It held up once people arrived. The Windows installer doesn't need admin rights, and on first launch it downloads the model for you. Someone with a gaming laptop in Southeast Asia can click and install. 11-14 On the same day someone filed a crash issue, and another user (nullbyte61) opened an issue about a Linux port, saying they had already got the engine and server running on Arch with an RTX 3070, and asking whether he wanted it merged.
Where it didn't go says a lot too. r/LocalLLaMA had no post, Hacker News had nothing, and nobody mentioned it on X for 30 days. No media covered it, and it didn't make GitHub Trending or the Trendshift charts (Trendshift recorded 11-14 +179 and 11-15 +2). The two hundred-plus stars came from AI circles in Malaysia and Southeast Asia on Threads. The open source community itself barely knows about it.
How crowded this niche is
"Running big MoE models on small GPUs" is one of the most crowded open-source opportunities of 2026, with a long line of newer projects bigger than this one:
| Project | star | How it differs |
|---|---|---|
| Ollama | 187k | One-click and easy to use, built on llama.cpp. ghost-infer's own tests show it twice as fast. |
llama.cpp(--n-cpu-moe) | 135k | The de facto standard. Experts are pinned per layer, with no dynamic hot-expert cache. |
| colibri | 51k | Pure C, streams experts from disk, and runs even larger frontier MoE models. |
| Strata | 44k | Same pitch: one-click install, big MoE on consumer hardware. Available for Windows and Linux, and still gaining 7 2,700 in the last day. |
| ktransformers | 19.7k | Experts on CPU, attention on GPU, aimed at large-RAM workstations. |
| Edge0 | 11.4k | Streams from SSD with routing prediction. A 35B model takes only about 2.5GB of RAM, and it even runs on phones. |
| magnitude | 7,802 | "2x faster than Ollama." The pitch is almost identical. |
| MoE4All | 361 | The closest match: a Windows portable build, big MoE on gaming cards beyond their VRAM, but on Vulkan, so AMD, NVIDIA and Intel GPUs all work. |
| glm-flash-lite | 351 | The same elastic GPU expert cache plus CPU layers, aimed at 3090 |
Star counts are from GitHub, as of mid-November. The hot-expert cache isn't unique to it either: there are already llama.cpp community PRs doing the same thing. What's actually rare is the combination: an old 8GB NVIDIA card, a one-click Windows install, an engine bundled with an agent and a voice companion, and reproducible benchmarks against llama.cpp and Ollama. But the leaders, Strata and colibri, are two orders of magnitude bigger.
No copycats yet. The only fork is the Chinese translator's.
If it were you
C: worth learning from, but following it isn't a good bet. This niche is already packed with projects that have tens of thousands of stars. ghost-infer's two hundred-odd stars weren't earned by the tech. They came from an anti-scam creator's existing account and a recording with Claude in the title. Building another "8GB card runs 35B" engine will most likely not earn stars. What's worth taking is the posting method.
Who it suits: developers who already have a few thousand followers on a platform (Threads, X, Xiaohongshu, Bilibili or Douyin all work), even if those followers didn't come for the tech. You also need a project you can explain with one number: old hardware running something new, or one command replacing a whole workflow.
Who it doesn't suit: anyone with no distribution channel who plans to write another "2x faster than llama.cpp" engine (magnitude uses exactly that line, and it's dozens of times bigger). Also anyone who only targets Windows and NVIDIA and hopes a narrow hardware niche will break out. One person can't keep up with every GPU vendor's quirks, and Strata and MoE4All already hold "all platforms" and "all GPUs."
How to do it: copy three steps. ① Pick an old machine that lots of people have, run your project to a concrete, relatable number, and record a real-time screen capture without editing. ② Ship the version first, then post right after a major release so downloads concentrate on the new build, with the repo link in your first reply. ③ Keep the install short enough to be "click and install," and answer "will my card run it?" in the commenters' own language. Look for the gaps it didn't fill: a Linux build (someone already offered a port you could merge), AMD GPU and Mac support, or a small tool that helps pick a model and tells you what your card can run. That last question was the most common one in its comments.
How to get users: the channels are your own account, plus the open source venues it never went to. Its lesson is that it only posted on Threads, so each post was a single spike. The same recording with an English title, posted to r/LocalLLaMA and Show HN, would have a real chance of sustained growth. Putting a popular model's name in the title really does work, but be ready to explain in the comments that it isn't actually Claude.
When the window closes: the "small GPU, big MoE" hook still works, but it loses value every month. Leading projects keep lowering the bar (Edge0 already says a 35B model needs only 2.5GB). If you want to ride this topic, plan on a monthly timeline. The posting method itself doesn't expire.
What would show this analysis was wrong: ① within a month, someone re-runs its benchmarks on r/LocalLLaMA, the post takes off, and stars climb into the thousands. That would mean the tech alone can earn stars, and this piece gave too much weight to the author's account. ② After it merges the Linux build and gets posted to HN, it keeps gaining dozens of stars a day. That would mean changing channels is enough, and the spot isn't actually crowded. ③ In reverse, if reproduction shows llama.cpp was not tuned well and the two-times advantage shrinks to one or two tenths, then it doesn't even hold up on the "numbers are solid" point.
Research notes: competitor table, acquisition evidence, full research report
- Competitors
Name Where Launched Scale How it differs from this one llama.cpp(--n-cpu-moe) Open source 2023-03 135k stars A general standard. Experts are pinned per layer, with no dynamic hot-expert cache. ghost-infer's own tests show it about 2x faster on MoE models that exceed VRAM. Ollama Open source 2023-06 187k stars One-click and easy to use, built on llama.cpp. Its own tests show it half as fast. ghost-infer also installs in one click, and it comes with an agent and voice. LM Studio Other 2023 A large base of closed-source desktop users. Cross-platform with multiple backends. ghost-infer only does Windows and NVIDIA, tuned specifically for them. ktransformers Open source 2024-07 19.7k stars Also experts on CPU, attention on GPU, but aimed at large-RAM workstations and a Python stack. ghost-infer targets 8GB gaming cards in a single file. Niko1221/Strata Open source 2026-09 44k stars, +2,709 over 7 days Same pitch: one-click install, big MoE on consumer hardware. It's two orders of magnitude bigger. JustVugg/colibri Open source 2026-07 51k stars Pure C, streams experts from disk, and runs even larger frontier MoE models. Edge0-AI/Edge0 Open source 2026-09 11.4k stars Streams from SSD with routing prediction. Cross-platform, including phones. A 35B model takes only 2.5GB of RAM. magnitudedev/magnitude Open source 2026-06 7,802 star The same "2x faster than Ollama" pitch. It auto-tunes kernels per hardware and supports multiple platforms. Headmaster218/MoE4All Open source 2026-08 361 star The closest match: a Windows portable app, big MoE on gaming cards beyond their VRAM. It uses Vulkan, so AMD, NVIDIA and Intel GPUs all work. sybil-solutions/glm-flash-lite Open source 2026-10 351 star The same elastic GPU expert cache plus CPU layers, aimed at running 3090 GLM. neurall/llama.cpp (PR #27861, hot-expert cache) Open source 2026 Small Implements the same hot-expert cache inside llama.cpp, focused on multi-GPU. ghost-infer is a from-scratch rewrite for a single card. - Where users come from
- Verified The author's own Threads account (screen-recording video posts + repo links in replies): the 11-03 post "Running 35B model locally on 8GB VRAM" got 386 likes, matching 0→46 stars. The 11-14 post "Running Claude-distilled 35B on RTX 2060 8GB GPU, at 70 tok/s" got 128k views / 1.1K likes / 268 shares, and that day it gained +171 stars and +10 forks (threads.com/@apizpudin/post/Dfsp6_pgKKD).
- Verified Audience built earlier (anti-scam bot PenipuMY): @apizpudin's bio reads Anti-Scam Activist, with about 3,937 followers; the pinned PenipuMY post has 11.8k likes, and 11-02 it was covered by newspapers
- Verified Riding the Claude name: the viral post title says Claude-distilled, but the model is a community distill of Qwen3.5-35B; a commenter asked "can you run Claude locally?" and the author replied "Claude by looks, Qwen is the content"; its likes were about 3 times those of the previous post, which didn't mention Claude
- Verified Auto-aggregator sites: speqai, everylocalai, upstar.dev and hdatf auto-index it; their impact is minor
- Inferred Reddit / HN / X: no hits on r/LocalLLaMA; HN library shows 0 items; x-radar shows 0 posts in 30 days
why ghost-infer (affifuddin-hazam/ghost-infer) got traction
Research date 2026-11-15 (task set 2026-11-07, data card refreshed to 11-15). Sources: our github library (repo_day / repo_profile), the hackernews library, the GitHub API, Trendshift, Threads web pages (read via Jina), Exa / Brave / Jina search, x-radar.
Conclusion first
ghost-infer is a local LLM inference engine for Windows, written in Rust + CUDA by a Malaysian solo dev (Apiz Pudin, Threads @apizpudin). It ships with a desktop agent app. The pitch: run a 35B MoE model on a 8GB GPU at 80 tok/s, which the README claims is 2x llama.cpp. Almost all the stars came from two video posts the author put up on Threads. 11-14 The one titled "Running Claude-distilled 35B on RTX 2060 8GB GPU, at 70 tok/s" got 128K views and 1.1K likes, and the repo went from 46 to 217 stars that day. The author already had a hit on Threads with PenipuMY, an anti-scam Telegram bot (one post with 11.8K likes, even covered by a Malaysian newspaper), which built him about 3,900 followers. This traffic came from that account.
The star growth was a one-off: 11-15 gained only 2 stars, never made the top 25 on Trendshift, and can't be found on Reddit, HN or X. Its differentiation is real (it targets just one NVIDIA card and a single user; MoE expert caching lives on the GPU; it's a one-click Windows app; it benchmarks against llama.cpp and Ollama with a reproducible script), but this category has been crowded for 2026 years, with projects in the thousands to tens of thousands of stars. It reached these two hundred-odd stars because of the hook "8GB older card + 35B + the Claude label + video demo" and because the author had an existing social account, not because the tech broke out in the open-source community.
1. What it is
- A 3.8MB Rust program (only two dependencies, no Python or Docker) that reads GGUF files and supports only NVIDIA GPUs, currently Windows only. It exposes an OpenAI-compatible
/v1/chat/completionsendpoint, plus a chat page with a little ghost mascot. - Core technique: the attention layers of an MoE model sit on the GPU, expert weights sit in system RAM, "hot experts" that a reply keeps reusing are cached on the GPU, and the remaining experts run on the CPU in parallel. Each layer runs as a CUDA graph, with the GPU choosing experts itself. There is also tensor-core attention (for long contexts), exact speculative decoding, a 4-bit KV cache, and LoRA.
- Desktop app GhostSetup: an agent that reads and writes files and runs commands (with permission modes); voice calls with the companion character Maya (a voice the author fine-tuned, maya1, hosted at affifuddin-hazam/ghost-voice on Hugging Face); "sleep training", which nightly trains facts from that day's chats into a LoRA on Qwen3-4B, currently experimental; and a Chinese/English bilingual UI.
- Speed table from the README (RTX 2060 SUPER 8GB + 24GB RAM, 2026-11-14, script
scripts/bench-all.ps1):
| Model | ghost-infer | llama.cpp | Ollama |
|---|---|---|---|
| Qwen3.6-35B-A3B | 80.5 | 39.5 | 34.7 |
| Qwen3-30B-A3B | 68.7 | 29.5 | 26.9 |
| Qwen3 8B | 75.3 | 64.7 | 65.8 |
These numbers have not been independently reproduced. When the model fits in VRAM (8B) it's only about 15% faster than the competition; the real gap shows up in MoE scenarios where the model is bigger than VRAM.
- Repo: created 2026-10-27, the author has made 138 commits, 1 other contributors 2424830124 (doing the Chinese translation, credited in v0.2.5). 0 issues, 0 merged PRs. From v0.1.0 (10-31) to v0.4.0 (11-13) there have been 10 releases. The author's account dates from 2023, has 2 followers, and its bio reads "Private projects, no code will be provided"; this is their first large public project.
2. Growth data
| Date | star | fork | Event |
|---|---|---|---|
| 10-27 | 0 | Repo created | |
| 10-31 | v0.1.0 installer | ||
| 11-03 | First Threads video post: "Running 35B model locally on 8GB VRAM & 16GB RAM with custom inference engine." 386 likes / 79 comments; the replies linked the repo with "Still new, please expect bugs" | ||
| 11-04 | 2 SEA-user forks | v0.2.0~0.2.2 | |
| 11-05 | 35 | 2 | (first observed in the github library) |
| 11-06~07 | 44→46 | 3 | |
| 11-08~13 | 46 unchanged | 3 | 11-10 v0.3.0 (only 9 downloads) |
| 11-13 22:08 UTC | v0.4.0 released (speed raised from about 45 to 80 tok/s) | ||
| 11-14 | 46→217(+171) | 13 | Second Threads video post: "Running Claude-distilled 35B on RTX 2060 8GB GPU, at 70 tok/s." 128k views, 1.1K likes, 145 comments, 61 reposts, 268 shares; the author linked the repo in a reply, which got 82 likes |
| 11-15 | +2 | Trendshift recorded 11-14 +179, 11-15 +2, not reaching the top 25 |
Sources: github library repo_day; GitHub API releases and forks; Trendshift (trendshift.io/repositories/351446); Threads post pages threads.com/@apizpudin/post/Dfsp6_pgKKD and /DfTRYBLgNTW.
A few supporting clues:
- The v0.4.0 installer was downloaded 159 times, while v0.3.0 has only 9 downloads, which matches the traffic from 11-14.
- 11-14 new forks by 10 accounts that look like real Southeast Asian users (StephenLovino, arvinmirafuentes21, wh-sugiharto, danielaniq28, vitou-vitou, labkita7, and others). The first fork appeared at 01:28 UTC, right after the post.
- The post's comments include Indonesian, Russian, Ukrainian and Italian, suggesting Threads' recommendations pushed it beyond Malaysia. The author replied to each in its own language.
- GitHub no longer exposes the star list (REST returns 404, GraphQL returns empty), so the starring accounts can't be checked one by one. However, the timing, download counts and fork accounts all line up with one real viral post, so this is judged to be organic traffic, not purchased stars.
3. Where users came from (growth tactics)
- The author's own Threads account (confirmed). Each of the two star surges corresponds to an original screen-recording video post by the author, with the repo link in the first reply. AI circles on Threads (the #AI and #Local LLM tags) pushed it further.
- An account the author built earlier (confirmed). @apizpudin's profile identifies them as an "Anti-Scam Activist"; the pinned PenipuMY post (a Telegram bot in Malaysia that checks for scams, with a same-named Google Play app listed in October) has 11.8k likes, and 11-02 posted that PenipuMY was in the newspapers. About 3,937 followers, with a Malay and Southeast Asian audience.
- Riding the "Claude" name (confirmed). The viral post's title says "Claude-distilled 35B", but the model is a community distill of Qwen3.5-35B-A3B made with Claude Opus 4.6 data, and the demo also used Huihui's abliterated (uncensored) version. Commenters asked "claude?? what do you mean" and "can you run Claude locally?"; the author replied "Claude by looks, Qwen is the content" and "at least it got Opus-style thinking and no restrictions". The previous post, without the Claude label, had only 386 likes.
- Hits the "old card + big model" pain point (confirmed). The title numbers are RTX 2060, 8GB, 35B, 70 tok/s. Most comments ask "can my 4050 6GB / RX 5700XT run it?"
- Auto-aggregator sites (confirmed, minor impact). speqai, everylocalai, upstar.dev, hdatf and thebestapplications index it; these are machine-generated pages.
- Channels where it wasn't found: no r/LocalLLaMA Reddit posts found; the HN library lists ghost-infer as 0 item(s); x-radar shows 0 post(s) in 30 days; no media coverage; not on GitHub Trending or the Trendshift top lists.
4. Competitors
"Running big MoE models on small hardware" is an extremely crowded niche in 2026, and our github library has plenty of similar projects:
| Name | Where | Launched | Scale | How ghost-infer differs |
|---|---|---|---|---|
llama.cpp(--n-cpu-moe) | Open source | 2023-03 | 135k stars | The de facto standard; experts are pinned to CPU or GPU per layer, with no dynamic hot-expert cache. ghost-infer uses it as its benchmark baseline and is about 2× faster on over-VRAM MoE models (self-reported) |
| Ollama | Open-source / desktop | 2023-06 | 187k stars | Easy one-click use, built on llama.cpp; self-reported about half the speed. ghost-infer also installs in one click, but is faster and ships with an agent |
| LM Studio | Desktop | 2023 | Large user base (closed source) | All platforms, multiple backends; ghost-infer is Windows + NVIDIA only |
| ktransformers | Open source | 2024-07 | 19.7k stars | Also puts experts on CPU and attention on GPU, but targets large-RAM workstations running DeepSeek, on a Python stack; ghost-infer targets 8GB gaming cards |
| PowerInfer | Open source | 2023-12 | 9,887 star | Similar hot/cold neuron split idea, research-oriented; ghost-infer is a finished application |
| ik_llama.cpp | Open source | 2024-06 | 3,393 star | A llama.cpp performance fork, command line; ghost-infer has an installer and a GUI |
| neurall/llama.cpp (hot-expert cache for MoE based on PR #27861) | Open source | 2026 | Small | The same hot-expert cache idea implemented inside llama.cpp, focused on multi-GPU; ghost-infer is a from-scratch rewrite, single card |
| Niko1221/Strata | Open source | 2026-09 | 44k stars, +2,709 over 7 days | Also pitches "one-click, run Qwen3.8-Flash-Next on any consumer hardware", Windows/Linux, two orders of magnitude larger in scope |
| JustVugg/colibri | Open source | 2026-07 | 51k stars | Pure C, experts streamed from disk, runs frontier MoE models; a more extreme take on "big models on small machines" |
| Edge0-AI/Edge0 | Open source | 2026-09 | 11.4k stars | A 35B MoE that uses only about 2.5GB of RAM, streamed from SSD with routing prediction, on all platforms including phones |
| magnitudedev/magnitude | Open source | 2026-06 | 7,802 star | "2× faster than Ollama", auto-tunes kernels to your hardware, pitch almost identical |
| wici-ai/inferssd, Helldez/BigMoeOnEdge | Open source | 2026-10 / 2026-07 | 1,098 / 850 star | Streams experts from SSD, built on llama.cpp |
| prabanta-dev/janas | Open source | 2026-09 | 712 star | "Faster than llama.cpp", pure C, runs models bigger than RAM |
| Headmaster218/MoE4All | Open source | 2026-08 | 361 star | Closest match: a Windows portable app that runs MoE models bigger than VRAM on gaming GPUs, but uses Vulkan so it supports A/N/I GPUs from all three vendors |
| sybil-solutions/glm-flash-lite | Open source | 2026-10 | 351 star | Elastic GPU expert cache + AVX2 CPU layer, same mechanism, aiming for 3090 |
| Indras-Mirror/Xolotl | Open source | 2026-10 | 519 star | Experts tiered across VRAM, RAM and NVMe, 4090 runs a 552B local coding agent |
Sources: GitHub repo_profile (created_at > 2026-03, sorted by stars), GitHub API, llama.cpp discussion #29429 / issue #29949, Trendshift's "similar repos".
Bottom line: "GPU caches hot experts, CPU computes the rest" isn't unique to this project. The llama.cpp community has PRs doing it, and so do glm-flash-lite and ktransformers. What's rare is the combination: an older 8GB NVIDIA card, one-click Windows install, an engine plus desktop agent and voice companion bundled together, and reproducible benchmarks against llama.cpp and Ollama. The leading projects in the same niche (Strata, colibri, Edge0) are two orders of magnitude bigger.
5. Why it took off (these ~200 stars)
- It had a ready-made social account. The author had a PenipuMY anti-scam bot that went viral on Threads in Malaysia, with 11.8k likes and about 3,900 followers. Both LLM posts went out on that account, and the second one got 128k views.
- The hook was well aimed. "Claude-distilled" + "RTX 2060 8GB" + "35B" + "70 tok/s", plus a screen recording in real time. The Claude name made non-technical users think they could run Claude locally, and the post got about 3× the likes of the previous one, which didn't mention Claude.
- The timing followed a major release. v0.4.0 shipped on the evening of 11-13, speed jumped from about 45 to 80 tok/s, and the next day he posted with the new numbers. v0.4.0 got 159 downloads and 11-14 traffic that matched.
- The product could absorb the traffic. The Windows installer doesn't need admin rights and downloads the model on first launch, so people in Southeast Asia with gaming laptops and older NVIDIA cards could install it straight away. The author replied in several languages to every "will my card run this?" comment.
It didn't sustain: it never reached Reddit, HN or X, the open-source home turf. By the second day it was back to +2 a day, and no other big accounts reposted it.
6. Differentiation, and whether that's why it took off
The differentiation has three layers: (1) optimization specific to a single user on one consumer NVIDIA card (a hot-expert cache plus one CUDA graph per layer), which the author's own tests put at 2× llama.cpp on MoE models that exceed VRAM; (2) a zero-dependency 3.8MB single-file build plus a Windows installer, with no Python, Docker or CUDA toolkit needed; (3) the engine, desktop agent, voice companion and sleep training packaged into one "ghost with a soul" app.
These differentiators make the demo video convincing (the numbers are hard), but they weren't the main reason it got stars. The star curve tracks the author's Threads posts, not spread through the tech community. Projects with similar tech get thousands of stars on Reddit and HN. This one never posted there, so it didn't get them.
7. Visibility and copycats
- Visibility: few people know about it yet. On X (x-radar), 0 posts; on HN, 0 posts; no Reddit posts found, no media coverage. It only caught fire once, in Threads circles in Malaysia and Southeast Asian AI.
- Copycats: none. 2424830124/ghost-infer is a Chinese translator's fork, not a copy. If anything, the project itself is following into an already crowded niche.
8. What others can learn
- Take older consumer hardware (2060 / 8GB), pair it with a large model (35B), and package the real speeds and a screen recording as a short-video post. That spreads far better than a benchmark table in the README.
- Put the hottest model brand (Claude) in the title, even when the model is a distillation, and accept that comments will ask "is this really Claude?"
- Build a social following in an adjacent niche (anti-scam tools) first, then launch the technical project. Cold start is then not zero.
- Post new numbers the day after a major release, so downloads concentrate on the new version.
- The limits of this path are also clear: one account and one post gives one spike. To keep growing, it has to get into r/LocalLLaMA and HN, the open-source home turf.
9. Still unclear
- Whether the benchmarks are inflated (were llama.cpp's parameters tuned to the best setting?). They were only run on the author's own 2060S, with no third-party reproduction.
- The star list isn't public, so the accounts that starred can't be checked one by one. Judging that the stars weren't bought relies on circumstantial evidence: the timeline, download counts and fork accounts.
- The post's publish time is inferred from Threads' "1d" label (about 11-14 UTC); no exact timestamp was available.
- Whether the author will later post on Reddit or HN, or ship a Linux build to widen the audience.