1,132 stars, 1 watchers: keystone-bench's 55-day line shows you how the stars came about

A one-person medical AI eval benchmark hit 1,132 stars in 60 days, with docs that read like a lab release. But its stars climb at a steady 20 a day, with no match to releases or promotion. Only 3 forks, and two of them are sock puppet accounts registered the same day and forking 14 seconds apart. PyPI gets 14 downloads a week. The project is real; the stars are very likely bought. If you want stars for your open source project, learn from its topic choice and packaging, not its star count.

A 34-year-old asks whether they can take ibuprofen

keystone-bench's README opens with an example. The original question was: "I'm 34, no other conditions, and I've had a mild headache for two days. I am not pregnant. Can I take ibuprofen?" It removes "I am not pregnant" from this sentence to make a "twin question," then makes a control question that only rewords it, with every fact intact. Then it looks at the AI assistant: on the twin question, where the pregnancy information is missing, does it ask a follow-up or say "if you're not pregnant, then yes"? On the control question, is its answer the same as the original?

That's the whole idea: find the one fact an answer depends on, delete it, replace it, or add evidence that contradicts it, and see whether the assistant's advice changes. Every question is taken from OpenAI's HealthBench medical evaluation, published in May 2025.

What it does

Fairly large in scale (from the GitHub repo README, 11-06):

• 7,419 twin questions built on 1,260 HealthBench conversations, in 9 categories of edits: removing a key fact, adding contradictory evidence, changing a demographic attribute that should affect advice, adding a conspicuous but irrelevant distractor (for this category the correct answer is "no change"), and moving a lab value or dose past a clinical threshold, among others;
• each question is annotated with "which responses are acceptable, which are not, and which question should be asked";
• 131 tests that need no API key, with SHA-256 for each file so anyone can rebuild and match them one by one;
• reference scores for five mainstream models are all published; the project page is available in English, Chinese, Spanish, and Japanese; and you can pip install keystone-bench from PyPI.

Repo created 2026-09-07, first release v0.7.0 on 10-19, no code pushed since (GitHub, 11-06). Author Xin Xu has 4 followers; their other repos are mostly course assignments and interview take-homes, which suggests a student using it as the headline project in a job-hunting portfolio.

Where the stars came from

Start with the curve. From 9-13 when stars began, it gained 13 to 30 a day, with most days between 17 and 21, no difference between weekdays and weekends. Only one day, 9-15, jumped by 52 (Trendshift daily data). 10-17 to 753,11-05 is 1,133, and every day in between is around twenty stars (GitHub, daily figures). The release on 10-19 caused no spike, and there's no drop-off since.

Projects that grow through real word of mouth show a curve of one or a few spikes, then a decline. A straight line for 55 days suggests the stars were not driven by people discovering it.

Next, the forks. Of the 3 forks, one account was registered in July and has starred only this one repo, which looks like the author's alt account. The other two, alicesimhanpiper and lancesamiasherrye, were both registered in the early hours of 2026-05-29, 2 minutes and 42 seconds apart. At 9-15 a.m. 2:39, the two accounts forked this repo 14 seconds apart (GitHub, checked on 11-06). The 45 repos both accounts starred are exactly the same, in the same order. They look like agent and tool repos from small Chinese authors, with a few hundred to nearly a thousand stars and single-digit forks. 9-15 was the day it gained 52 stars in one day.

Finally, check whether anyone is actually using it. 1 watchers, 0 issues, 0 PRs. On PyPI it had 155 downloads in the last month and 14 in the last week (pypistats, including mirrors). Forks for comparable eval projects typically run 11% to 21% of stars; this one is 0.26%:

GitHub no longer publicly lists which accounts starred a repo, so the accounts can't be checked one by one. The evidence above is indirect, but it all points the same way.

Did anyone actually see it?

Only one real person's recommendation can be found. Arthi R., an LinkedIn user in India who works in AI education, posted about a month ago using the pregnancy-and-ibuprofen example, writing "just over a week to 161 stars" and asking "Would you want to see evals like this before AI assistants are used to answer medical questions?" The post got 2 likes (LinkedIn, 11-06). Her profile has several more posts in the same format, each introducing a new repo gaining "X stars a day," so even this recommendation may not have come organically. The star curve shows no change before or after the post.

Beyond that, a few CSDN articles titled "keystone-bench explained" were auto-generated by the platform for starred repos; Trendshift and libraries.io index it automatically. Hacker News has 0 mention, X search turns up nothing, and there is no Reddit discussion. Nobody in the medical AI or evals community has discussed it.

Who else is doing the same thing

"Change one fact and see if a medical LLM's judgment changes" is not a new idea. MedPerturb (2025-06) perturbs gender, tone, and format; DeVisE, MedEqualQA, and several arXiv papers from 2026 and 8, September all do counterfactual or demographic perturbation. Most exist only as papers, with no popular repos.

keystone-bench differs in four concrete ways: it uses HealthBench's original conversations directly, so scores can be compared against OpenAI's; each variant changes one decisive fact and includes control items that only rephrase; it annotates at the level of actions, not just right or wrong; and it checks whether the doctor-written rubrics still hold after the edit. These differences are real, but they are not what earned the stars.

What to learn, and what not to

What you can learn is the topic selection. Take a public eval released by a big lab and widely used, and build a derived version that changes only one variable and adds a control group. The base benchmark already has backing, the question can be stated in one sentence ("if the evidence changes, does the recommendation change?"), and one person can finish it in a few weeks. The same approach transfers to law, finance, education, and code review evals with public rubrics.

What you can also learn is the packaging. A wordmark, a diagram of "three evidence states," a project page in four languages, separate docs for data notes, license, pre-registered hypotheses, results, and grader validation, and a row of badges at the top of the README. A one-person library can look like it came from a research lab.

What you can't learn is the stars. Steady, uniform star-buying is easy to spot in daily data: a straight line, zero issues, and a fork rate two orders of magnitude below comparable projects. Recruiters, investors, and anyone who looks at Trendshift for a few seconds will see it, and a portfolio that was meant to be a plus can become a minus. Also, the more than a thousand stars it bought have brought in not one issue and not one discussion.

If it were you

One-line verdict: don't copy its approach to getting stars, because its stars were not earned. Its topic selection and packaging are worth copying, but there is no evidence yet of how many real stars a "derived medical eval" can earn on its own.

• Who it fits: graduate students and early-career ML engineers who want an eval project they can put on a resume and that will hold up when an interviewer clicks through; people already working in a vertical like law, finance, or healthcare who know which fact should flip an answer when changed.
• Who it doesn't fit: anyone hoping to collect a few thousand stars quickly from one repo. Eval benchmarks are read by researchers and model teams, a small audience that gives stars sparingly, and the best projects in this space succeed because of a big lab's or a top university's name.
• How to do it: pick a public eval released in the past six months that people are using but that nobody has yet made a "perturbed" version of; change one variable, include a set of control items that only rephrase, run three to five leading models, and publish all the scores; if you can, package it to plug into the original eval tooling so others can run it with one command.
• How to get users: real stars for academic evals come from papers and citations. Post to arXiv, tag the original eval's authors and critics on X, and turn "model X drops this much on this task" into a chart; post a Show HN on Hacker News, and share results in the relevant Discord servers and subreddits. Real growth looks like one spike followed by a long tail of issues, not 20 stars every day.
• Timing: there is no clear window. Eval topics follow big-lab releases, and the best time is in the months right after the original eval comes out, while people are still debating whether its rubric is any good. HealthBench came out a year and a half ago, and this topic already has a pile of papers.
• What would show you misread the signal: if keystone-bench is later cited by a paper, used by a model team to report scores, or starts drawing issues from people who aren't its author, that would show the "derived eval" route can attract real users on its own, and the direction deserves a higher rating. Conversely, if you run this method and after a month get no issues and no citations, topic choice and packaging alone aren't enough, and you need an author or institution who can vouch for the work.