* fix(web): include base prefix in generated URLs
Forgot to add the base prefix to these URLs. Committed a fix for this
and added a test to ensure this does not repeat. Oops!
Closes: #1402
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(nginx): fix tests to work in GHA
Closes: #1371
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(test): does this work lol
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(test): does this other thing work lol
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(test): pki folder location
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Co-authored-by: Jason Cameron <git@jasoncameron.dev>
* docs: split nginx configuration files to their own directory
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add nginx config smoke test based on the config in the docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
TIL docker clients don't include the Accept header all the time. I would
have thought they did that. Oops.
Closes: #1346
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat: first implementation of honeypot logic
This is a bit of an experiment, stick with me.
The core idea here is that badly written crawlers are that: badly
written. They look for anything that contains `<a href="whatever" />`
tags and will blindly use those values to recurse. This takes advantage
of that by hiding a link in a `<script>` tag like this:
```html
<script type="ignore"><a href="/bots-only">Don't click</a></script>
```
Browsers will ignore it because they have no handler for the "ignore"
script type.
This current draft is very unoptimized (it takes like 7 seconds to
generate a page on my tower), however switching spintax libraries will
make this much faster.
The hope is to make this pluggable with WebAssembly such that we force
administrators to choose a storage method. First we crawl before we
walk.
The AI involvement in this commit is limited to the spintax in
affirmations.txt, spintext.txt, and titles.txt. This generates a bunch
of "pseudoprofound bullshit" like the following:
> This Restoration to Balance & Alignment
>
> There's a moment when creators are being called to realize that the work
> can't be reduced to results, but about energy. We don't innovate products
> by pushing harder, we do it by holding the vision. Because momentum can't
> be forced, it unfolds over time when culture are moving in the same
> direction. We're being invited into a paradigm shift in how we think
> about innovation. [...]
This is intended to "look" like normal article text. As this is a first
draft, this sucks and will be improved upon.
Assisted-by: GLM 4.6, ChatGPT, GPT-OSS 120b
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(honeypot/naive): optimize hilariously
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(honeypot/naive): attempt to automatically filter out based on crawling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): use mazeGen instead of bsGen
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: add honeypot docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(test): go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: fix spelling metadata
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(config): deprecate the report_as field for challenges
This was a bad idea when it was added and it is irresponsible to
continue to have it. It causes more UX problems than it fixes with
slight of hand.
Closes: #1310Closes: #1307
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(policy): use the new logger for config validation messages
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin/thresholds): remove this report_as setting
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: add dependabot cooldown
One of the things I need to worry about with Anubis is the idea that
could pwn a dependency and then get malicious code into prod without
realizing it, a-la Jia Tan. Given that Anubis relies on tools like
Dependabot to manage updating dependencies (good for other reasons),
it makes sense to have Dependabot have a 7 day cooldown for new
versions of dependencies.
This follows the advice from Yossarian on their blog at [1]. Thanks
for the post and easy to copy/paste snippets!
[1]: https://blog.yossarian.net/2025/11/21/We-should-all-be-using-dependency-cooldowns
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor: move lib/policy/config to lib/config
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor: don't set global loggers anymore
Ref #864
You were right @kotx, it is a bad idea to set the global logger
instance.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(config): add log sink support
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(test): go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin/policies): add logging block documentation
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(cmd/anubis): revert this change, it's meant to be its own PR
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add file logging smoke test
Assisted-by: GLM 4.6 via Claude Code
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix: don't expose the old log file time format string
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(internal): add ListOr[T any] type
This is a utility type that lets you decode a JSON T or list of T as a
single value. This will be used with Redis Sentinel config so that you
can specify multiple sentinel addresses.
Ref TecharoHQ/botstopper#24
Assisted-by: GLM 4.6 via Claude Code
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(store/valkey): add Redis(R) Sentinel support
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
check-spelling run (pull_request) for Xe/redis-sentinel
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* chore(store/valkey): remove pointless comments
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document the Redis™ Sentinel configuration options
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(store/valkey): Redis™ Sentinel doesn't require a password
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
* fix(data): add services folder to embedded filesystem
Also includes a regression test to ensure this does not happen again.
Assisted-By: GLM 4.6 via Claude Code
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: add asset build verification workflow
A CI pass that fails if generated files are out of date.
* chore: npm run assets
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(data): add ruleset to explicitly allow Docker / OCI clients
Fixes#1252
This is technically a regression as these clients used to work in Anubis
v1.22.0, however it is allowable to make this opt-in as most websites do not
expect to be serving Docker / OCI registry client traffic.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/gh-1252/docker-registry-client-fix
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* test(docker-registry): export the right envvars
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: add simdjson dependency for homebrew node
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: install go/node without homebrew
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: use right github commit variable
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: remove simdjson dependency
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: install ko with an action
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: add OCI registry caveat docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Tencent Cloud's abuse team reached out to me recently and asked for this
rule to be removed. Prior attempts to reach out to them to report
abusive traffic have failed, thus leading to this IP space block as a
last resort to try and maintain uptime for systems administrators.
Unfortunately, it's difficult for Tencent's abuse team to take action if
there is a blanket block like this. Let's see if this doesn't cause too
much grief.
* test(nginx-external-auth): bring up to code standards
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): close open redirect when in subrequest mode
Closes GHSA-cf57-c578-7jvv
Previously Anubis had an open redirect in subrequest auth mode due to an
insufficent fix in GHSA-jhjj-2g64-px7c. This patch adds additional
validation at several steps of the flow to prevent open redirects in
subrequest auth mode as well as implements automated testing to prevent
this from occuring in the future.
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Recent models like GPT-5 have broken these instructions. As such, I
don't think that it's worth having these around anymore. I think that
longer term it may be better to have a policy of having people disclaim
which models they use in commit footers rather than having a "don't use
this tool" policy, which people are just going to work around and
ignore.
* fix!(policy/checker): make List and-like
This has the potential to break user configs.
Anubis lets you stack multiple checks at once with blocks like this:
```yaml
name: allow-prometheus
action: ALLOW
user_agent_regex: ^prometheus-probe$
remote_addresses:
- 192.168.2.0/24
```
Previously, this only returned ALLOW if _any one_ of the conditions
matched. This behaviour has changed to only return ALLOW if _all_ of the
conditions match.
I have marked this as a potentially breaking change because I'm
absolutely certain that someone is relying on this behaviour due to
spacebar heating. If this bites you, please let me know ASAP.
Signed-off-by: Xe Iaso <me@xeiaso.net>
Assisted-by: GPT-OSS 120b on local hardware
* fix(policy/checker): more explicit short-circuit
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(default-config): block tencent cloud by default
This is what happens when you don't have an abuse contact.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Closes#1206
This can cause Anubis to have other issues, but at the very least these
issues are at the Anubis level, not the level of your target service so
it's less bad.
* fix(lib): show error message detail when hitting some common flows
Instead of giving the user nothing to go off of, this patch gives them
an opaque blob of ROT-13 encoded base64. The logic is that if you are
smart enough to figure out how to decode this, you're probably smart
enough to either fix your broken client or give it to the adminstrator.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/show-error-state
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
* feat(data): add default-config macro
Closes#1152
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add default-config-macro smoke test
This uses an AI generated python script to diff the contents of the bots
field of the default configuration file and the
data/meta/default-config.yaml file. It emits a patch showing what needs
to be changed.
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add httpdebug tool
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(data/clients/git): more strictly match the git client
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(default-config): make the default config far less paranoid
This uses a variety of heuristics to make sure that clients that claim
to be browsers are more likely to behave like browsers. Most of these
are based on the results of a lot of reverse engineering and data
collection from honeypot servers.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
* fix(lib): enable multiple consecutive slash support
Closes#754Closes#808Closes#815
Apparently more applications use multiple slashes in a row than I
thought. There is no easy way around this other than to do this hacky
fix to avoid net/http#ServeMux's URL cleaning.
* test(double_slash): add sourceware case
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix tests for double slash fix
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib/challenge): expose ResponseWriter to challenge issuers
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(metarefresh): randomly use the Refresh header
There are several ways to trigger an automatic refresh without
JavaScript. One of them is the "meta refresh" method[1], but the other
is with the Refresh header[2]. Both are semantically identical and
supported with browsers as old as Chrome version 1.
Given that they are basically the same thing, this patch makes Anubis
randomly select between them by using the challenge random data's first
character. This will fire about 50% of the time.
I expect this to have no impact. If this works out fine, then I will
implement some kind of fallback logic for the fast challenge such that
admins can opt into allowing clients with a no-js configuration to pass
the fast challenge. This needs to bake in the oven though.
[1]: https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/meta/http-equiv
[2]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Refresh
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(metarefresh): simplify random logic
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
* fix(decaymap): fix lock convoy
Ref #1103
This uses the actor pattern to delay deletion instead of making things
fight over a lock. It also properly fixes locking logic to prevent the
convoy problem.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* security: npm audit fix for GHSA-hfm8-9jrf-7g9w et. al
Closes#1097
I'm not sure that this is required, but I'd sleep better at night not
finding out that it is required the hard way.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: bump postcss version
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(challenge): demote temporal assurance to 80% instead of 95%
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(challenge/preact): wait a little longer to be extra safe
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(challenge/metarefresh): wait a little longer to be extra safe
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(CHANGELOG): add fix notes
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Some admins have noticed that clients are not waiting the right amount
of time in order to access a resource protected by the metarefresh
challenge. This patch adds a check to make sure that clients have waited
at least 95% (difficulty times 950 milliseconds instead of difficulity
times 1000 milliseconds) of the time they should.
If this scales, maybe time is the best way to go for Anubis in the near
future instead of anything else computational.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(worker): constrain nonce value to be a whole integer
Closes#1043
Sometimes the worker could get into a strange state where it has a
decimal nonce, but the server assumes that the nonce can only be a whole
number. This patch constrains the nonce to be a whole number on the
worker end by detecting if the nonce is a decimal number and then
truncating away the decimal portion.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(algorithms/fast): truncate decimal place on number of threads
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(default-config): block Huawei Cloud
Closes#978
Huawei Cloud has been egregious about its scraping. All attempts to
contact their abuse team have failed. If you work for Huawei Cloud,
please raise this issue internally and get the scraping to just stop.
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): ensure issued challenges don't get double-spent
Closes#1002
TL;DR: challenge IDs were not validated at time of token issuance. A
dedicated attacker could solve a challenge once and reuse it across
multiple sessons in order to mint additional tokens.
With the advent of store based challenge issuance in #749, this means
that these challenge IDs are only good for 30 minutes. Websites using
the most recent version of Anubis have limited exposure to this problem.
Websites using older versions of Anubis have a much more increased
exposure to this problem and are encouraged to keep this software
updated as often and as frequently as possible.
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
This was causing confusion and less technical users were thinking that
websites had been intruded upon, causing them to send me horrible things
over email.
All non-English strings were amended using Google Translate. Please fix
the localization as appropriate.
* refactor: make challenge pages return the challenge component
This means that challenge pages will return only the little bit that
actually matters, not the entire component.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): move Anubis version info to be implicitly in the footer
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): embed challenge ID into generated pages
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): make tests pass
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/policy/config): amend tests
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix tests again
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
* chore(web/js): delete proof-of-work-slow.mjs
This code has served its purpose and now needs to be retired to the
great beyond. There is no replacement for this, the fast implementation
will be used instead.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(web): handle building multiple JS entrypoints and web workers
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(web): rewrite frontend worker handling
This completely rewrites how the proof of work challenge works based on
feedback from browser engine developers and starts the process of making
the proof of work function easier to change out.
- Import @aws-crypto/sha256-js to use in Firefox as its implementation
of WebCrypto doesn't jump directly from highly optimized browser
internals to JIT-ed JavaScript like Chrome's seems to.
- Move the worker code to `web/js/worker/*` with each worker named after
the hashing method and hash method implementation it uses.
- Update bench.mjs to import algorithms the new way.
- Delete video.mjs, it was part of a legacy experiment that I never had
time to finish.
- Update LibreJS comment to add info about the use of
@aws-crypto/sha256-js.
- Also update my email to my @techaro.lol address.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): don't hard dep webcrypto anymore
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(lib/policy): start the deprecation process for slow
This mostly adds a warning, but the "slow" method is in the process of
being removed. Warn admins with slog.Warn.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(web/js): allow running Anubis in non-secure contexts
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/purge-slow
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Previously the X-Forwarded-For middleware could return two commas in a
row. This is a regression test to make sure that doesn't happen again.
Imports a patch previously exclusive to Botstopper.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): block XSS attacks via nonstandard URLs
This could allow an attacker to craft an Anubis pass-challenge URL that
forces a redirect to nonstandard URLs, such as the `javascript:` scheme
which executes arbitrary JavaScript code in a browser context when the
user clicks the "Try again" button.
Release-status: cut
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): fix challenge issuance logic
Fixes#869
v1.21.0 changed the core challenge flow to maintain information about
challenges on the server side instead of only doing them via stateless
idempotent generation functions and relying on details to not change.
There was a subtle bug introduced in this change: if a client has an
unknown challenge ID set in its test cookie, Anubis will clear that
cookie and then throw an HTTP 500 error.
This has been fixed by making Anubis throw a new challenge page instead.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): you win this time spell check
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Fixes#877
Continued from #879, event loop thrashing can cause stack space
exhaustion on ia32 systems. Previously this would thrash the event loop
in Firefox and Firefox derived browsers such as Pale Moon. I suspect
that this is the ultimate root cause of the bizarre irreproducible bugs
that Pale Moon (and maybe Cromite) users have been reporting since at
least #87 was merged.
The root cause is an invalid boolean statement:
```js
// send a progress update every 1024 iterations. since each thread checks
// separate values, one simple way to do this is by bit masking the
// nonce for multiples of 1024. unfortunately, if the number of threads
// is not prime, only some of the threads will be sending the status
// update and they will get behind the others. this is slightly more
// complicated but ensures an even distribution between threads.
if (
(nonce > oldNonce) | 1023 && // we've wrapped past 1024
(nonce >> 10) % threads === threadId // and it's our turn
) {
postMessage(nonce);
}
```
The logic here looks fine but is subtly wrong as was reported in #877
by a user in the Pale Moon community. Consider the following scenario:
`nonce` is a counter that increments by the worker count every loop.
This is intended to spread the load between CPU cores as such:
| Iteration | Worker ID | Nonce |
| :-------- | :-------- | :---- |
| 1 | 0 | 0 |
| 1 | 1 | 1 |
| 2 | 0 | 2 |
| 3 | 1 | 3 |
And so on.
The incorrect part of this is the boolean logic, specifically the part
with the bitwise or `|`. I think the intent was to use a logical or
(`||`), but this had the effect of making the `postMessage` handler fire
on every iteration. The intent of this snippet (as the comment clearly
indicates) is to make sure that the main event loop is only updated with
the worker status every 1024 iterations per worker. This had the
opposite effect, causing a lot of messages to be sent from workers to
the parent JavaScript context.
This is bad for the event loop.
Instead, I have ripped out that statement and replaced it with a much
simpler increment only counter that fires every 1024 iterations.
Additionally, only the first thread communicates back to the parent
process. This does mean that in theory the other workers could be ahead
of the first thread (posting a message out of a worker has a nonzero
cost), but in practice I don't think this will be as much of an issue as
the current behaviour is.
The root cause of the stack exhaustion is likely the pressure caused by
all of the postMessage futures piling up. Maybe the larger stack size in
64 bit environments is causing this to be fine there, maybe it's some
combination of newer hardware in 64 bit systems making this not be as
much of a problem due to it being able to handle events fast enough to
keep up with the pressure.
Either way, thanks much to @wolfbeast and the Pale Moon community for
finding this. This will make Anubis faster for everyone!
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Possible fix for #877
In some cases, the parallel solution finder in Anubis could cause
all of the worker promises to leak due to the fact the promises
were being improperly terminated. A recursion bomb happens in the
following scenario:
1. A worker sends a message indicating it found a solution to the proof
of work challenge.
2. The `onmessage` handler for that worker calls `terminate()`
3. Inside `terminate()`, the parent process loops through all other
workers and calls `w.terminate()` on them.
4. It's possible that terminating a worker could lead to the `onerror`
event handler.
5. This would create a recursive loop of `onmessage` -> `terminate` ->
`onerror` -> `terminate` -> `onerror` and so on.
This infinite recursion quickly consumes all available stack space, but
this has never been noticed in development because all of my computers
have at least 64Gi of ram provisioned to them under the axiom paying for
more ram is cheaper than paying in my time spent having to work around
not having enough ram. Additionally, ia32 has a smaller base stack size,
which means that they will run into this issue much sooner than users on
other CPU architectures will.
The fix adds a boolean `settled` flag to prevent termination from
running more than once.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add i18n smoke test
Makes sure that all of the languages that Anubis supports show up when
the challenge page is sent to a client.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(i18n): build anubis so that the smoke test doesn't backoff timeout
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(anubis): add /healthz route to metrics server
Also add health check test for Docker Compose and update documentation
for health checking Anubis with Docker Compose.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
This is not used yet, but it will be part of a larger strategy around
adding/removing weight based on JA4H (and other) fingerprint matches
with Thoth.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): fix race condition when rendering multiple challenge pages at once
Closes#832
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): make try again button work
Looks like the intent of this was "try the solution again". This fix
makes the client try the challenge again.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): don't block a user if they have an invalid challenge cookie
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
This was causing issues with git clone against highly loaded servers. I
thought that this would be pretty innocuous, but I guess I was wrong.
Oops!
Signed-off-by: Xe Iaso <me@xeiaso.net>
