Debug Production From Public Code: DEV's Missing Badges

6 min read · Last updated: October 25, 2026

You can debug production from public code when the code is open source and you can form a hypothesis the code can prove or kill. Dhruv Jani did this to find why DEV's 4x, 8x, 16x and 32x contributor badges never arrived, then fixed it in forem/forem PR #24113.

Came here from Instagram, LinkedIn, X or YouTube? Here is the full source list you came for. Everything below is built from these, nothing else.

The write-up

• I'm Not on the DEV Team. I Still Debugged Their Missing Badges From Public Code, by Dhruv Jani on DEV Community, October 24, 2026

Key takeaways

• Four of the five badge slugs hardcoded in Forem's badge service had no matching row in production, so the 4x, 8x, 16x and 32x badges could not be earned.
• The service dropped missing slugs with .compact: no error and no log line.
• The fix shipped as PR #24113, merged as commit d1b962e by Ben Halpern, co-founder of DEV.
• The author's first regression test also passed on main, so it proved nothing.
• Log levels are a contract: missing optional data is info, not warn.

What happened to DEV's multi-commit contributor badges?

Forem's hourly badge worker never awarded the 4x, 8x, 16x or 32x contributor badges, because four of its five hardcoded slugs had no matching row in production. Dhruv Jani merged four PRs into forem/forem, got only the 1x badge, read the public code, and fixed the service in PR #24113.

Forem's automated review agent, OpenHands, ran a read-only lookup on the badges table and confirmed the diagnosis. Ben Halpern then merged the fix and commented that the bug "has been around so long it had completely faded from everyone's" mind. The 4x badge landed on Dhruv's profile after the merge.

Why does this matter to a developer?

The bug is a pattern, not a one-off. Code hardcodes a string, a database holds the real value, and nothing checks that the two agree. When they drift, a silent .compact turns a configuration error into a feature that quietly never works.

The second reason is the method. Dhruv had no admin panel, no production database access and no internal Slack. Everything came from the public forem/forem repository and public pages on dev.to. If you maintain or use open source software, you can do the same.

How do you debug production from public code?

Start from the symptom, read the schedule and the code path, then use public pages to test one specific guess. Here is the sequence Dhruv followed, step by step:

1. Find when the job runs. config/schedule.yml showed BadgeAchievements::BadgeAwardWorker runs at cron: "20 * * * *", which is :20 past every hour.
2. Confirm the input is right. GitHub's Contributors API (/repos/forem/forem/contributors) showed the account on page 2 with contributions: 4. Forem sets Octokit.auto_paginate = true, so the worker reached page 2.
3. Compare the working path with the broken one. award_single_commit_contributors awarded the 1x badge. award_multi_commit_contributors handles 4x, 8x, 16x and 32x. Reading them side by side exposed three bugs.
4. Look at public data. Forem only exposes resources :badges, only: [:index], so there is no public page per slug. The public dev.to/badges catalog still showed Badge #423 with the title "4x DEV Contributor Club" and the description "4x-commit-club".
5. Form a falsifiable guess. Badge#generate_slug parameterizes the title when no explicit slug is set, giving 4x-dev-contributor-club. That would not match the hardcoded "4x-commit-club".
6. Let someone with access confirm it. Dhruv filed Issue #24114 for badge IDs 423 to 426. The OpenHands review then confirmed: only one of the five slugs resolves.

No single step needed private access. Step 6 did, which is why the guess had to be testable.

What were the three bugs in the badge service?

The service had three separate bugs in Badges::AwardContributorFromGithub, according to Dhruv's write-up. One stopped the badges from being awarded at all. Two made the awards that did happen less correct and noisier. The table shows each one and its fix.

Bug	What the code did	Fix in PR #24113
Silent slug drops	Badge.id_for_slug(slug) returned nil for unmatched slugs and .compact removed them without an error or log line	Slug lookup deferred behind the GitHub auth provider check, with logging when configured slugs are missing
Missing context message	Single-commit badges passed rewarding_context_message_markdown: msg, multi-commit badges used a bare .create	Message attached to multi-commit awards
No idempotency guard	create_badges_for_user called BadgeAchievement.create(...) every hourly run	Replaced with .where(...).first_or_create(...)

The third bug is worse than it looks. If a badge ever had allow_multiple_awards: true, a bare .create would insert a duplicate achievement every hour. Each insert would fire after_create_commit :notify_recipient and send_email_notification, flooding inboxes from yo@dev.to.

Why did the first regression test pass on main?

The test passed on main because it was testing ActiveRecord's uniqueness validator, not the fix. Dhruv's spec ran described_class.call twice and asserted one badge achievement. The badge factory defaults to allow_multiple_awards: false, so the second bare .create failed validation silently and the count stayed at 1.

The review caught it. The corrected spec in commit 5e36999 creates the badge with allow_multiple_awards: true, where bare .create genuinely inserts duplicates on main. It then asserts expect { described_class.call }.not_to change(BadgeAchievement, :count) on the second run.

The review caught a second flaw too. Dhruv had logged a Rails.logger.warn for each missing slug. Until production slugs were aligned, DEV would log 4 warn lines every hour. Many self-hosted Forems enable GitHub auth without seeding DEV-specific badges, so that would be permanent noise. The fix, also in 5e36999, collects missing slugs and emits one Rails.logger.info line per run.

Check	First attempt	After review
Idempotency spec	Badge with allow_multiple_awards: false, passes on main and on the fix	Badge with allow_multiple_awards: true, fails on main
Missing-slug log	warn per slug, 4 lines an hour	One info line per run

What changes in practice?

If you run a Forem, awards for multi-commit contributors now carry the thank-you message and run idempotently, and missing optional badge rows show up as one info line instead of nothing. For everyone else, three habits change.

• A regression test is not real until it fails on main. Stash your change, run the spec against main, and check that it goes red.
• warn means someone should act. On a platform with optional data seeds, a missing optional row is info.
• Silent drops need a log line. .compact on lookups hides the exact error you will want later.

Dhruv's own summary: you don't need production access to debug production, only the public source, a falsifiable hypothesis and curiosity. Reviewing code you did not write is a skill of its own, and our guide to reviewing AI-generated code without missing critical bugs covers the same instinct of distrusting green results.

What should you do this week?

1. Pick one test you wrote recently and run it against main without your change. If it still passes, rewrite it.
2. Grep your code for .compact, || nil and empty rescue blocks around database lookups, and add a log line.
3. Compare hardcoded constants with database rows. Any slug, ID or enum in code that must match stored data needs a startup check.
4. Audit your warn logs. Anything expected in normal operation should drop to info.
5. Try an open source bug. Read the schedule, the code path and the public data, then write your guess in an issue before you write a fix.

FAQ

Can you really debug production without access to it?

Often you can get most of the way. Dhruv used a public repository, GitHub's Contributors API and the public dev.to/badges catalog to form a precise hypothesis. He still needed a maintainer-side check to confirm it, and the OpenHands review supplied that with a read-only lookup on the badges table.

Why didn't anyone on DEV get the 4x badge?

According to the OpenHands review quoted in the write-up, only one of the five BADGE_SLUGS keys resolves, and the four Nx-commit-club slugs have no matching row. So create_badges_for_user skips them and nobody can earn them. Dhruv's explanation is that the stored slug follows the title, not the hardcoded constant.

Why was the regression test misleading?

It asserted one badge achievement after two runs, but the factory default allow_multiple_awards: false meant Rails' uniqueness validation blocked the duplicate on its own. The test passed on main without the fix, so it measured the validator rather than first_or_create.

What is an idempotency guard in a background job?

It is a check that makes running the same job twice produce the same result as running it once. Here the guard is .where(...).first_or_create(...), which skips awards a user already has instead of retrying them every hour.

Pick one green test this week and run it against main. Browse the free tools on Art of Code, and if the lesson about shipping work that outlasts the code is your thing, read code was never the hard part.

Written by Adithya Guttha, Founder of Art of Code.