Port hypothesis property check from 06-basic-text-test.py to PropCheck - #29
Open
janl wants to merge 163 commits into
Open
Port hypothesis property check from 06-basic-text-test.py to PropCheck#29janl wants to merge 163 commits into
janl wants to merge 163 commits into
Conversation
hulkoba
force-pushed
the
port-python-tests
branch
4 times, most recently
from
November 19, 2025 14:21
c413a13 to
25f1f55
Compare
refactor(basic-find-test): add sort to _find calls for limit and skip options
AlbaHerrerias
force-pushed
the
port-python-proptests
branch
2 times, most recently
from
November 26, 2025 14:25
d27e93d to
79942ff
Compare
…m Python to Elixir
AlbaHerrerias
force-pushed
the
port-python-proptests
branch
from
December 2, 2025 13:50
79942ff to
b4392b1
Compare
Running the search tests with Clouseau 3.0 amplified the flakiness for containers which is making hard to use the CI -- disable those for now until they root cause is investigated and a fix is provided. Mind that native builds, e.g. FreeBSD and Windows are still there to catch Search-related regressions.
`doc_lookup/3` could also return an `{ok, nil}` tuple if this is not the owner
node. We already expect it in the only place we call `doc_lookup/3` from but
the spec didn't have that case so we add it in here.
In the Explain output, bookmarks for queries on `text` (Search) indexes are added in their unpacked format to the options. This leads to at least two problems: - it is inconsistent with how empty bookmarks are rendered for `json` indexes, where the string `"nil"` is used, - for non-trivial bookmarks, contents cannot be translated by `jiffy:encode/1` to a JSON representation so that it will crash. Mitigate this issue by deferring the unpacking of bookmarks to the execution phase so that explain will get affected by that. Thanks @mojito317 for reporting this issue, and thanks @rnewson for revisiting my original fix and guiding me about it.
…rk-formatting fix(`mango`): formatting of `text` bookmarks in `_explain` output
Since we inspect the #full_doc_info{} record to skip deleted docs anyway, move
all the doc_id clauses to the `doc_fdi/3` callback to have one less callback to
worry about.
There is not performance penalty as by the time we call `doc_id/3` we already
have the full `#full_doc_info{}` record opened anyway.
In apache#5625 we introduced a BTree cache for dbs, and since view files also use BTrees, let's use a cache for them as well. This should help with concurrent view queries. I test the improvments I used k6 for benchmarking with a 1M doc, q=16 db, and a view with 10M rows. With a 400 rps request arrival rate saw more than a 50% improvement in average, p90, and p95 latencies: * Average: 14 -> 7 msec * Median: 12 -> 7 msec * P90 21 -> 10 msec * P95 27 -> 12 msec * Max 120 -> 77 msec Since view results could have large kp node values due to custom reduce functions, add max term limit for cache entries. To avoid adding yet another config value use a few multiple of BTree chunk size. This should help avoid config values clashing if users raise the chunk size but forget to raise their cache term limit (it would render the cache unusuable suddenly). More detailed test results along with a test at a higher depth of 4 to show further improvements. ``` BENCH_RATE=400 BENCH_DURATION=5m BENCH_VIEW_LIMIT=10 k6 run ./k6_couchdb_constant_arrival_view_query.js HTTP http_req_duration..............: avg=14.65ms min=2.08ms med=12.7ms max=120.08ms p(90)=20.53ms p(95)=26.78ms { expected_response:true }...: avg=14.65ms min=2.08ms med=12.7ms max=120.08ms p(90)=20.53ms p(95)=26.78ms { name:view_get }............: avg=14.65ms min=2.93ms med=12.7ms max=120.08ms p(90)=20.53ms p(95)=26.78ms http_req_failed................: 0.00% 0 out of 120002 http_reqs......................: 120002 399.891866/s ``` ``` $ BENCH_RATE=400 BENCH_DURATION=5m BENCH_VIEW_LIMIT=10 k6 run ./k6_couchdb_constant_arrival_view_query.js HTTP http_req_duration..............: avg=7.45ms min=2.38ms med=6.9ms max=77.78ms p(90)=10.21ms p(95)=11.5ms { expected_response:true }...: avg=7.45ms min=2.38ms med=6.9ms max=77.78ms p(90)=10.21ms p(95)=11.5ms { name:view_get }............: avg=7.45ms min=2.38ms med=6.9ms max=77.78ms p(90)=10.21ms p(95)=11.49ms http_req_failed................: 0.00% 0 out of 120004 http_reqs......................: 120004 399.799018/s ``` ``` $ BENCH_RATE=400 BENCH_DURATION=5m BENCH_VIEW_LIMIT=10 k6 run ./k6_couchdb_constant_arrival_view_query.js HTTP http_req_duration..............: avg=6.21ms min=2.22ms med=5.78ms max=77ms p(90)=8.4ms p(95)=9.5ms { expected_response:true }...: avg=6.21ms min=2.22ms med=5.78ms max=77ms p(90)=8.4ms p(95)=9.5ms { name:view_get }............: avg=6.21ms min=2.22ms med=5.78ms max=77ms p(90)=8.4ms p(95)=9.5ms http_req_failed................: 0.00% 0 out of 120003 http_reqs......................: 120003 399.931891/s ```
Build dev improvements
- If there is no checkpoint and no `since_seq`, then replicate from scratch. - If there is no checkpoint but `since_seq` is defined, then replicate with the `since_seq` field. - If both checkpoint and `since_seq` exist, use the checkpoint to replicate. - If the request includes `since_seq` field, the replication ID will be changed.
Previously we didn't handle the case when the "infinity" value was explicitly set in the config file. We only handled the case when it was the default value.
When debugging processes getting stuck in hibernation bug [1] a few benchmarks
showed that hibernation can be pretty expensive. I saw 20% or so reduction in
latency in couch_work_queue if we hibernate after every single item insertion.
Erlang documents warn about this [2]:
> Use this feature with care, as hibernation implies at least two garbage
collections (when hibernating and shortly after waking up) and is not something
you want to do between each call to a busy server.
In a few places like the `couch_work_queue` and `couch_db_updater` we did
exactly that. However, since we added that more Erlang/OTP implemented a new
`gen_server` option - `{hibernate_after, Timeout}`. It will trigger hibernation
after an idle time. That seems ideal for us - it keeps expensive hibernation
out of the main data path, as docs warn us about, but once the server goes idle
we still get to run it to dereference any ref binaries.
Since we encountered the recent hibernation bug [1] also add an option to
disable it altogether, just to have a way to mitigate the issue when running on
OTP 27 and 28 before the fix is out.
[1] erlang/otp#10651
[2] https://www.erlang.org/doc/apps/stdlib/gen_server.html
[3] apache@d9eb87f
Send 404 for /_all_dbs and /_dbs_info with extra path parts
There are two main improvements: 1) Remove dynamic runtime reloading. That feature is not useful any longer after we stopped doing online relups, and instead became a potential landmine that can lead to metrics unexpectedly faling if stats description files, or disk access to them changes at runtime. The only minor benefit previous behavior had was was for local dev/run development, coupled with online module reloading; but that hardly justifies the complexity and the confusion resulting from stats all of the sudden breaking in production. 2) Strengthen a few asserts during stats loading. Only accept missing stats description from applications, and stop on any other error. We don't want to run with stats missing or unloadable or constantly spewing "unknown metric" error.
Mango unit tests
…-arm ci: temporarily disable freebsd-arm worker because it is too slow
…ct-index-test Port Python test to Elixir: `12_use_correct_index_test`
…ult-field-test Port Python test to Elixir: `06-text-default-field-test`
Port Python test to Elixir: `07_text_custom_field_list_test`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The current property is hardcoded at running 1_000 tests, which
takes about 10s on my machine. I have run it up to 20_000 tests
and it never failed so far.
Run this with new command
make elixir-test-search-propsas Icould not make it work with the test suite.
Removes the existing test from the Python test file.
NOTE: this currently runs against nouveau as I did not have a
Clouseau setup handy, could someone wiggle things so they run
with Clousau?