indra-belief-model

LLM-based evidence quality scoring for INDRA biomedical text-mining extractions.

What this does

INDRA's NLP readers extract structured biological relationships from scientific papers. For example, from the sentence:

"The kinase-dead RSK1 mutant, however, was unable to phosphorylate YB-1 at S102."

a reader might extract: RPS6KA1 [Phosphorylation] YBX1 @S102

This scorer judges whether such extractions are correct. Here, the extraction is incorrect — the sentence describes a negative result (the mutant was unable to phosphorylate).

Input

Native INDRA Statement + Evidence objects, resolved through ScoringRecord:

Field	Example	Source
Claim	`RPS6KA1 [Phosphorylation] YBX1 @S102`	Statement type + agents + modification site
Evidence	"The kinase-dead RSK1 mutant..."	Source sentence from paper
Entity aliases	RSK1, YB-1, p90Rsk...	Gilda grounding via `GroundedEntity.resolve()`

Output

{"verdict": "correct", "confidence": "high"}

Mapped to a continuous score: {correct+high: 0.95, correct+medium: 0.80, ..., incorrect+high: 0.05}.

How it works

Model: gemma-4-26b (Ollama remote or local MLX 8-bit).

Production scoring architecture

The CLI default is the monolithic scorer: a deterministic LLM call per (Statement, Evidence) pair with type-adaptive contrastive examples (a second call fires only for [Complex] claims — see Tier 2). The decomposed four-probe scorer remains available for ablations with --arch decomposed.

Two-tier monolithic path

Tier 1: Deterministic grounding (no LLM call)

Status	Action	Example
MISMATCH	Auto-reject	"RhoA" → RHOA != ARHGEF25
PSEUDOGENE + AMBIGUOUS	Auto-reject	"DVL" → DVL1P1 (pseudogene)
AMBIGUOUS	Pass to Tier 2	"9G8" → SRSF7/SLU7 (tied scores)
MATCH	Pass to Tier 2	"FAK" → PTK2 (confirmed alias)

Tier 2: LLM text comprehension

Commit-first "disconfirm" system prompt: the model commits a defeating objection to a structured field before it rationalizes a verdict (negation, hedging, family/member equivalence, relation-direction reversal).
Seven adaptive contrastive pairs (14 examples) selected by statement type.
One deterministic LLM call per pair at low temperature — plus, for [Complex] claims, a focused relation-nature step (Gilda-grounded entity aliases) that rejects a non-binding relationship mistaken for a Complex.

The default variant is disconfirm_relnature_rf (set MONO_VARIANT="" for the plain six-rule baseline; MONO_VARIANT=disconfirm for disconfirm without the relation-nature step; MONO_VARIANT=disconfirm_relnature for the reasoning-SECOND variant, which is no longer the default). That reasoning-first prompt is what the shipped gemma_bedrock_rf calibration profile is keyed on, so changing the variant changes the (model, prompt_sha) calibration key and the fitted profile stops applying — see _FITTED_CONFIGS in src/indra_belief/calibration_constants.py.

Adaptive few-shot selection

The example bank has type-specific contrastive pairs. For each record, 7 pairs are selected by priority:

Own type from bank (e.g., Activation pairs for an Activation claim)
Adjacent types from TYPE_ADJACENCY map (e.g., IncreaseAmount for Activation)
Universal patterns (logical inversion, hedging scope)
Fill from the base contrastive pair set

Types with bank examples: Activation (2 pairs), Inhibition (2), Phosphorylation, Complex, IncreaseAmount, DecreaseAmount, Dephosphorylation, Autophosphorylation, Translocation, Ubiquitination.

Run the scorer

PYTHONPATH=src python -m indra_belief.scorers.scorer \
    --model gemma-remote \
    --arch monolithic

Design decisions we already paid for

Earlier iterations measured the following approaches and rejected them. If you're considering a change that resembles one of these, check the data before re-proposing:

Approach	Outcome	Why it fails
Decomposed multi-call scorer	Strictly dominated on holdout_cc (F1 0.657 vs 0.751 monolithic)	Natural-language extraction can't bridge INDRA's soft ontology boundaries — requires multiple LLM probes to agree on a fuzzy contract
Native tool-calling (agentic lookup)	84.9%, below baseline	Model ignores tool results after committing to a verdict in its first pass
Structured provenance, full population	-6.7pp accuracy	Attention dilution on 26B model outweighs disambiguation benefit — selectively enabling provenance only for flagged-grounding records preserves the signal without the cost
Graduated warnings for every grounding quirk	3 regressions per 1 fix	Redirects attention from sentence comprehension; now limited to PSEUDOGENE and LOW_CONFIDENCE
Indirect-evidence marker in the prompt	+5pp false negatives	Prejudices model toward rejection; removed
LOW_CONFIDENCE auto-reject (blanket)	53.6% precision at scale (32 false rejections on 3,754 records)	The gilda score threshold is too noisy to gate on deterministically; the signal is still available to the LLM as context

Headline baselines measured during iteration: gemma-4-26b + adaptive bank + voting reaches ~84% accuracy on the 501-record stratified sample. Small-holdout numbers (200 records) overstate by ~4-5pp relative to large-scale evaluation (3,000+ records) — check the larger set before celebrating.

Setup

Dependencies

pip install gilda indra openai anthropic   # anthropic only for the claude-* path

# Download the benchmark corpus (460MB, not included in repo)
# Place at data/benchmark/indra_benchmark_corpus.json.gz
# Source: https://doi.org/10.5281/zenodo.7559353

Model configuration

The scorer calls an LLM via ModelClient(model_name). Model names map to entries in model_client.py's LOCAL_MODELS dict, or to Anthropic model IDs (any string starting with claude-).

Local Ollama (recommended for getting started):

# Install Ollama: https://ollama.com
ollama pull gemma3:27b          # or any model you prefer
ollama serve                    # starts on localhost:11434

Then add an entry to LOCAL_MODELS in src/indra_belief/model_client.py:

"ollama-local": {
    "base_url": "http://localhost:11434/v1",
    "model_id": "gemma3:27b",
    "reasoning_in_content": False,
    "max_tokens": 1000,
    "timeout": 120,
},

Use it: ModelClient("ollama-local") or --model ollama-local from the CLI.

Remote Ollama (e.g., a beefy server on your network):

Same as above but point base_url at the remote host. The gemma-remote entry in the registry shows this pattern — it targets an Ollama instance over Tailscale.

Local MLX (Apple Silicon):

scripts/serve_mlx.sh serves a reader through mlx_lm.server, which speaks the same OpenAI-compatible protocol as Ollama. Setup, once:

uv venv ~/.venvs/mlx-serve --python 3.12
VIRTUAL_ENV=~/.venvs/mlx-serve uv pip install mlx-lm

scripts/serve_mlx.sh                    # gemma-4-26b-a4b 8-bit on :8085
MODEL=mlx-community/gemma-4-31b-it-8bit PORT=8084 scripts/serve_mlx.sh

The MLX stack lives in its own virtualenv at ~/.venvs/mlx-serve by choice, not by necessity — there is no dependency conflict, and an earlier revision of this paragraph claiming one was wrong. Re-runnable evidence: VIRTUAL_ENV=$PWD/.venv uv pip install --dry-run mlx-lm resolves 34 packages and would install exactly three — mlx==0.32.0, mlx-lm==0.31.3, mlx-metal==0.32.0 — upgrading nothing and removing nothing. mlx-lm 0.31.3 declares no torch and no sympy at all; its core requirements are mlx>=0.31.2 (marker platform_system == "Darwin"), numpy, transformers>=5.0.0, sentencepiece, protobuf, pyyaml and jinja2 — print them with ~/.venvs/mlx-serve/bin/python -c "from importlib.metadata import distribution; print(distribution('mlx-lm').requires)", whose remaining entries (datasets, lm-eval, tqdm, the CUDA/CPU mlx variants) all sit behind extra == markers and so are never pulled by a bare install. Torch, sympy, pysb and transformers already coexist in the project interpreter: .venv/bin/python -c "import torch,sympy,pysb,transformers as tf; print(torch.__version__, sympy.__version__, pysb.__version__, tf.__version__)" exits 0 and prints 2.11.0 1.11.1 1.17.0 5.9.0. What is true is only a violated declaration — .venv/bin/python -m pip check reports, among other metadata complaints, torch 2.11.0 has requirement sympy>=1.13.3, but you have sympy 1.11.1 — a complaint about declared bounds, not a failed import, and not what forces the split.

What the split buys is a judgement, not a necessity. The cost it avoids is measurable and modest: installing mlx-lm into .venv would add three distributions totalling ~192 MB, ~188 MB of it the Apple-Silicon-only mlx-metal binary.

~/.venvs/mlx-serve/bin/python -c 'from importlib.metadata import distribution as D; print([(n, D(n).version, round(sum(D(n).locate_file(f).stat().st_size for f in D(n).files)/1e6, 1)) for n in ("mlx", "mlx-lm", "mlx-metal")])'
# [('mlx', '0.32.0', 1.8), ('mlx-lm', '0.31.3', 1.6), ('mlx-metal', '0.32.0', 188.4)]

Do not quote du -sm ~/.venvs/mlx-serve (316 MiB) as the avoided cost: that is the whole serving venv, and most of its non-MLX bulk — numpy, transformers — .venv already carries. mlx-metal's files also unpack into the mlx/ import directory (there is no mlx_metal/ beside it), which is why du -sm on ~/.venvs/mlx-serve/lib/python3.12/site-packages/mlx reports 189 MiB of allocated blocks rather than the ~2 MB mlx's own files come to.

That weight stays out of the uv.lock-resolved environment, which has no mlx entry at all (grep -c '^name = "mlx' uv.lock → 0) and is exercised on ubuntu-latest CI (.github/workflows/ci.yml:9). The marker doing the work there is mlx-lm's own: it requires mlx>=0.31.2; platform_system == "Darwin", so off Darwin a resolver never requests mlx — it is not that mlx would refuse to install. (mlx 0.32.0 carries no Darwin marker itself; its Darwin-only piece is mlx-metal==0.32.0; platform_system == "Darwin".)

The scorer never imports mlx: it reaches the server over HTTP as a plain openai_compat backend. The one in-process MLX path, scripts/run_probe_battery.py, imports mlx_lm lazily inside its read functions — grep -cE '^(import|from) mlx' scripts/run_probe_battery.py → 0 at module scope, while grep -nE '^[[:space:]]+(import|from) mlx' on the same file returns three indented hits inside function bodies. Gold loading, prompt rendering, record construction and artifact verification therefore stay importable in .venv, and that script is run under ~/.venvs/mlx-serve/bin/python when it actually needs the model.

The script's MODEL and PORT defaults must stay equal to the local-gemma-4-26b entry in src/indra_belief/model_client.py — mlx-community/gemma-4-26b-a4b-it-8bit on port 8085. Change one without the other and every call 404s. tests/test_readme_code_claims.py compares the script's defaults against the registry entry and fails if they diverge — and fails too if this section stops naming them.

Why serve locally at all: this is currently the only reader we can read token logprobs from. Bedrock's gemma-4 routes accept top_logprobs and return an empty array, so p_raw cannot be measured there at all. Two consequences for callers:

top_logprobs is capped at 11, so a caller that assumes a larger ceiling gets a 400 rather than a truncated list: grep -n 'top_logprobs' ~/.venvs/mlx-serve/lib/python3.12/site-packages/mlx_lm/server.py → self._validate("top_logprobs", int, min_val=0, max_val=11, whitelist=[-1]). The registry mirrors the cap as max_top_logprobs so callers clamp before the request (grep -n max_top_logprobs src/indra_belief/model_client.py).
Serve at temperature 0.0 — the script's default, and a correctness precondition for scoring, because the sampled verdict must be reproducible. The logprobs themselves are indifferent to it: mlx_lm computes logprobs = logits - logsumexp(logits) before the sampler runs (generate.py:420-421 in mlx_lm 0.31.3), so temperature, top-p and top-k change which token is sampled but never the distribution we read.

Smoke-check a running server before trusting a scoring run:

.venv/bin/python scripts/probe_logprobs.py --model local-gemma-4-26b

It exits 0 only when logprobs came back and were non-degenerate. A route that accepts top_logprobs and then returns nothing exits non-zero with status empty rather than looking like a clean pass.

Anthropic API:

export ANTHROPIC_API_KEY=sk-...

client = ModelClient("claude-sonnet-4-20250514")

Any claude-* model name routes to the Anthropic backend automatically.

Key LOCAL_MODELS fields:

Field	Purpose
`base_url`	OpenAI-compatible endpoint (Ollama serves this at `/v1`)
`model_id`	Model name as known to the server (`ollama list` to check)
`reasoning_in_content`	`True` if CoT appears in `content` (Qwen CRACK); `False` for models with a separate `reasoning_content` field (Gemma 4) or no reasoning
`max_tokens`	Completion token budget — reasoning models need more (8000+)
`num_ctx`	Ollama-specific: context window size (passed via `extra_body`)
`timeout`	Seconds before retry — increase for large models or slow hardware
`supports_logprobs`	`True` if the route actually returns `choices[].logprobs.content[]`. Absent/`False` means the field is accepted and ignored — the Bedrock gemma-4 failure mode
`max_top_logprobs`	Server-enforced ceiling on `top_logprobs`. `mlx_lm.server` validates with `max_val=11` and 400s above it — NOT the usual OpenAI/vLLM 20

Usage

Score a Statement's evidence

An INDRA Statement bundles a list of Evidence objects. score_statement mirrors that abstraction: one per-sentence verdict per evidence, returned in order.

from indra.statements import Phosphorylation, Agent, Evidence
from indra_belief import ModelClient, score_statement

stmt = Phosphorylation(
    Agent("RPS6KA1"), Agent("YBX1"),
    residue="S", position="102",
)
stmt.evidence = [
    Evidence(source_api="reach",
             text="RSK1 phosphorylates YB-1 at S102 in response to stress."),
    Evidence(source_api="sparser",
             text="The kinase-dead RSK1 mutant was unable to phosphorylate YB-1 at S102."),
]

client = ModelClient("gemma-remote")
verdicts = score_statement(stmt, client)
# verdicts is list[dict], one per evidence:
#   verdicts[i]["verdict"]    → "correct" | "incorrect" | None
#   verdicts[i]["score"]      → 0.95 (correct+high) … 0.05 (incorrect+high)
#   verdicts[i]["confidence"] → "high" | "medium" | "low"
#   verdicts[i]["tier"]       → which scoring path produced the verdict

The importable score_statement / score_evidence run the monolithic scorer — the default arch (empirically dominant on holdout_cc, F1 0.751 vs the decomposed 0.657). For the decomposed four-probe path, import the same names from indra_belief.scorers.decomposed.

To score just one evidence of a Statement (skipping the rest of stmt.evidence), use score_evidence(stmt, ev, client).

Composition with INDRA belief

score_statement is the per-sentence comprehension layer. The edge-level question — given all evidence for a statement, what is the belief? — is answered by a calibrated score that operationally treats those verdicts as noisy measurements of one latent fact: whether the statement is correct. The mixed- evidence limitation of that assumption is stated below. The two layers chain directly:

from indra_belief import score_statement
from indra_belief.statement_belief import statement_belief
from indra_belief.noise_model import RECALIBRATED_PRIORS
from indra_belief.calibration_constants import calibration_for_run

verdicts = score_statement(stmt, client)  # list[dict], one per stmt.evidence
rows = [
    {"source_api": ev.source_api, "verdict": v["verdict"],
     "confidence": v.get("confidence"), "tier": v.get("tier")}
    for ev, v in zip(stmt.evidence, verdicts)
]
# Canonical edge belief for verdicts loaded from a persisted scoring run.
# Resolve model + scorer-prompt identity from that same run's call logs; a
# model name alone is deliberately insufficient.
profile = calibration_for_run("data/results/my_run.jsonl")
sb = statement_belief(rows, RECALIBRATED_PRIORS, soft=profile)
# sb.belief             → hybrid log-odds score (fitted) / hard gate (unfitted)
# sb.parametric_only    → belief before any LLM gating (ablation)
# sb.verdict_statement  → tiered decision: correct | review | incorrect

For a ship-approved reader configuration, calibration_for_run returns the reader's measured verdict-by-gold confusion matrix and quantities derived from it—never hand-set weights. Its log_lr_confirm field is log(P(confirm|correct) / P(confirm|incorrect)); log_lr_reject is the analogous rejection log-likelihood ratio. A confirmed read contributes the stronger of the reader's confirm log-LR and its INDRA source-reliability log-odds, so the confirmation contribution cannot undercut an already stronger curated-source contribution; a rejected read contributes log_lr_reject, and an unscored direct input uses source reliability alone. Correlated reads from the same source are averaged, independent sources are summed with the explicit fit-set prior, and a sigmoid converts the resulting log-odds to belief.

Production currently enables two exact configurations: remote Gemma with prompt fingerprint b44638216740… (4/4 on the independent holdout) and reasoning-first Bedrock Gemma with 07377e338ff2… (4/4 on external curator gold). Remote MedPsy's b44638216740… profile remains a measured diagnostic candidate but is disabled: its matched holdout failed the ECE leg (3/4), while its external run used the different 07377e338ff2… prompt and cannot validate that fit. Missing, mixed, or mismatched prompt provenance therefore returns None and retains the hard gate.

That source term is a posterior reliability estimate from a separate 9,342-row source-prior fit, not another likelihood ratio. The fitted-reader scalar is therefore an explicit hybrid calibration score, not a pure Bayesian posterior; changing the prior anchor is a global score shift, not a clean deployment- prevalence correction. The evaluation target is also conservative: evidence labels roll up to statement gold with any-incorrect-wins. It is a useful review proxy, not a literal observation of one latent statement truth when evidence is mixed.

The soft= argument name is retained for API compatibility; it now accepts this measurement profile, not survival weights.

Representative INDRA curations

The viewer's representative lane starts from a 5,000-pair uniform evidence-row reservoir drawn without replacement with Algorithm R from exactly 44,944,056 grounded/assembled evidence rows in the CoGEx 2025-09-16 dump. The 5,000 rows are the sampling frame, not the size of CoGEx, and the sampling unit is an evidence row rather than a statement or every raw INDRA extraction. The source dump and reproduced reservoir are pinned by SHA-256.

At serving time, the tracked manifest retains all 5,000 reservoir keys for provenance but blocks two exact pairs that occur in older benchmarks, leaving 4,998 eligible keys. Every card shown to a curator is atomically reserved in a persistent draw ledger; completed INDRA history and all prior reservations, including skips, are removed before the next random draw. Production must set CURATION_DRAW_LEDGER_DIR to storage shared by every viewer instance and acknowledge it with CURATION_DRAW_LEDGER_SHARED=1; sampling fails closed when that guarantee is not configured. Rows that no longer materialize through INDRA or lack usable text are retried, so the served population is conditional on materializability and text availability.

The June 29 snapshot was not drawn from this frame. It predates the July 3 reservoir lane and came from the older hand-selected, high-coverage agent-panel sampler. The first qualifying reservoir curation is ID 19920 on July 6.

The tracked mock7ee@gmail.com artifact data/benchmark/representative_indra_curations_400.jsonl is a first-write-wins unique-pair progress snapshot; _400 names the benchmark target, not the current row count. It contains all 403 unique exact pairs from 415 qualifying submissions available through curation ID 20334 at the recorded export cutoff. For each pair, the first qualifying submission supplies the tag and derived binary label; the 12 later repeat submissions are excluded from canonical rows and labels rather than aggregated as votes, while their provenance remains in audit metadata. The resulting snapshot is 199 correct / 204 incorrect. Each row contains one curation event and embeds the judged INDRA statement structure for clean-checkout inspection; no duplicate-event arrays or any-incorrect-wins conflict rollups are present. The 400-pair target is complete and exceeded by 3 pairs. Benchmark status remains pending because the historical completed-sequence randomness is unproven.

All 403 snapshot pairs are reservoir members and have zero prior-benchmark or pre-reservoir-curation overlap. The latter is pinned against mock7ee_pre_reservoir_pair_manifest.jsonl (124 genuine old-viewer submissions, 123 unique pairs; the unrelated API auth probe is excluded). That proves frame membership, not that the historical completion sequence was a simple random sample: the legacy UI retained no draw/skip log, retried unusable rows, and allowed pairs to be drawn again. First-write deduplication removes those repeat events from the artifact but cannot reconstruct a no-replacement draw history. The artifact is therefore described as reservoir-sourced, not as a provable simple-random sample of the reservoir.

For an unfitted reader, the hard-gate fallback retains confirmed/unscored evidence and removes rejected evidence before applying the parametric noisy-OR. The tiered verdict_statement is the production decision (deterministic hard-flag → incorrect; else any LLM incorrect → review; else correct) and is independent of the belief scalar. Source priors live in noise_model.py (INDRA_PRIORS, RECALIBRATED_PRIORS).

Score a corpus + browse the results

For corpora larger than a single Statement (e.g. an INDRA-native JSON dump from rasmachine), the monolithic pipeline is the production path. It scores each evidence and writes append-only per-evidence JSONL alongside a run .meta.json and .progress.ndjson:

set -a; . ./.env; set +a   # GEMINI_API_KEY / AWS_BEARER_TOKEN_BEDROCK / HF_TOKEN
PYTHONPATH=src python scripts/run_rasmachine_monolithic.py \
    --model gemma-remote \
    --input data/corpora/latest_statements_rasmachine.json \
    --output data/results/rasmachine_run.jsonl

Estimate cost first: from indra_belief.corpus import estimate_cost returns projected LLM-call counts and USD per model before you spend.

The viewer/ SvelteKit app browses finished runs. It is a read-only, in-memory projection over the per-run exports under data/exports/<run>/ (per_statement.json + per_evidence.jsonl + export_meta.json + metrics.json), loaded by SvelteKit server load functions (+page.server.ts) — no database. Current calibration comparisons fail closed unless both products carry matching byte-level corpus and gold digests, the same exact evaluated evidence- and statement-key sets, and a compatible metrics contract. Temporal deltas are stricter still: they require the same exact reader configuration. Fit-set results are labeled in-sample and are never presented as external validation. The publication-grade statement comparison has its own frozen artifact and status contract; see research/indra_belief_comparison.md and /frontier?view=belief.

cd viewer && npm install && npm run dev  # http://127.0.0.1:5174

Observed LLM cost (per run)

Each ordinary run export carries the real USD it cost to score, computed from the token usage actually observed during the run — not an estimate. Pricing for these exports lives in src/indra_belief/corpus/cost.py; the viewer only reads baked numbers.

At export time, every evidence row's call_log (one entry per LLM call, each carrying prompt_tokens, out_tokens, and the real model_id) is priced via token_cost_usd and summed. Per-row cost_usd is baked into per_evidence.jsonl; a run total + input/output token totals + usd_per_1k_evidence go into export_meta.json. The run feed (/runs) shows a compact per-run cost; the run detail (/runs/<id>) shows total, cost per 1k LLM-scored evidence, tokens, and the model(s) billed.

Three honest states — the viewer never invents a price:

known — every scored row used a model with a verified price (local / self-hosted models are genuinely free → $0.00).
partial — some rows used a priced model and some an unverified one; the total covers only the priced rows, with the unavailable-row count shown.
unavailable — no row had a verified per-token price, or the export predates cost capture. Shows "cost unavailable" with token counts, never a fabricated $0.

AWS Bedrock Claude (sonnet-4-6, haiku-4-5) and Gemma 4 (gemma-4-26b-a4b, gemma-4-31b, gemma-4-e2b) are priced at published AWS/Anthropic on-demand list rates; local models are zero marginal cost. A model in neither table reads "unavailable" rather than a fabricated $0. To price a model, add its per-1M-token input/output rate to MODEL_PRICES_PER_M_TOKENS (or its id to ZERO_COST_MODEL_IDS if free) in cost.py, then re-export the run.

The statement-level INDRA comparison does not silently inherit that mutable run-export table. Its LLM bundles bind data/comparison/pricing.json: structured AWS Bedrock us-east-1 on-demand pricing, requested tier default, resolved tier Standard, exact provider model and token rates, retrieval date, and one cost-comparability identity. All-source and five-reader costs are observed projections of the same run and are explicitly non-additive.

Benchmark evaluation against a holdout file

PYTHONPATH=src python -m indra_belief.scorers.scorer \
    --model gemma-remote \
    --arch monolithic \
    --holdout data/benchmark/holdout_large.jsonl \
    --output data/results/run.jsonl \
    --resume data/results/run.jsonl  # resume interrupted runs

How we iterate

Contributor-facing rules to keep the repository legible:

main is the canonical state. Every "ship" decision ends with git push. Local ship decisions don't count.
Immutable identities are explicit. Dataset, schema, model, prompt, and decision-artifact identities retain their real names and hashes; prose describes the current contract rather than narrating refactor chronology.
Public API is score_statement(statement, client) + score_evidence(statement, evidence, client). score_statement mirrors INDRA's abstraction (a Statement owns a list of Evidence) and returns one dict per evidence. score_evidence is the atomic per-sentence call. score(client, record, …) is the benchmark-harness path used by indra_belief.scorers.scorer.main; treat it as internal.
Comments explain current constraints. Historical implementation rationale belongs in git log; source comments state only the causal constraint that governs current behavior.

Project structure

src/indra_belief/
  model_client.py          # Model transport (OpenAI-compat + Anthropic)
  noise_model.py           # INDRA SimpleScorer (parametric belief from source priors)
  statement_belief.py      # verdicts → hybrid log-odds score (hard-gate fallback)
  curation.py              # INDRA-curation gold rule + hash bridge + index
  metrics.py               # Binary confusion P/R/F1 + ECE calibration
  sampling.py              # Two-stage / priority sampling + Wilson half-width
  results.py               # Run-result loading + row shaping
  scorers/
    scorer.py              # Public score_statement / score_evidence + benchmark main
    _shared.py             # Verdict→score mapping shared across scorers
    context.py             # Per-record scoring context
    context_builder.py     # Grounding + alias context assembly
    commitments.py         # Claim-commitment extraction
    grounding.py           # Gilda-backed entity grounding
    kg_signal.py           # Knowledge-graph corroboration signal
    parse_claim.py         # Statement → typed claim parse
    relation_patterns.py   # Regex relation cues
    monolithic/            # Default scorer
      scorer.py            # MONO_VARIANT dispatch (default disconfirm_relnature_rf)
      _prompts.py          # Baseline six-rule system prompt
      _prompts_disconfirm.py  # Commit-first disconfirm prompt + backstop
      _prompts_relation.py    # [Complex] relation-nature step (Gilda aliases)
    probes/                # Decomposed four-probe scorer (--arch decomposed)
      orchestrator.py      # Probe pipeline + router
      router.py            # Statement → probe set
      subject_role.py object_role.py relation_axis.py scope.py bind_check.py
      adjudicator.py       # Probe verdicts → final
      _llm.py types.py
    panel/                 # Objection-panel ablation
      orchestrator.py detectors.py adjudicator.py types.py
  corpus/
    cost.py                # estimate_cost + MODEL_PRICES_PER_M_TOKENS (only surviving surface)
  tools/
    gilda_tools.py         # Gilda lookup helpers
  data/
    entity.py              # GroundedEntity: single gilda resolution per entity
    scoring_record.py      # ScoringRecord: wraps INDRA Statement + Evidence
    corpus.py              # CorpusIndex: source_hash → Statement lookup
    example_bank.json      # Type-specific contrastive pairs

viewer/                    # SvelteKit dashboard — read-only projection over data/exports/<run>/
  src/lib/
    format.ts              # Cue extraction, verdict rendering, sentence formatting
    residuals.ts           # Residual histogram bucket logic
    index.ts               # Re-exports
    components/            # BeliefPrimitive, BeliefRuler, SiteNav, Validity
    data/                  # In-memory data layer over the JSONL exports
      runs.ts              # Run discovery (dirs with export_meta.json)
      queries.ts           # Per-run / per-statement / per-evidence selectors
      curation.ts          # INDRA-curation gold lane (twin of curation.py)
      adjudicate.ts review.ts store.ts types.ts
  src/routes/                # each route pairs a +page.svelte with a sibling
                             # +page.server.ts load (runs/[run_id]/ adds
                             # +layout.server.ts); the server loads run the
                             # $lib/data selectors over the per-run JSONL exports
    +page.svelte           # Dashboard: focus + findings + validity + runs feed
    +layout.svelte         # Shared nav shell
    +error.svelte          # Generic 4xx/5xx error page
    runs/+page.svelte                      # Runs index
    runs/[run_id]/+page.svelte             # Per-run detail (+layout.server.ts loads the run)
    statements/+page.svelte                # Matrix (paginated, URL-stated)
    statements/[stmt_hash]/+page.svelte    # Per-stmt deep-dive (evidence cards + rollup)
    compare/+page.svelte                   # Model-vs-model dig (L0–L3, optional gold mode)
    adjudicate/+page.svelte                # Blind human verdict (curation revealed as 3rd judge)
    review/+page.svelte                    # Faithfulness / correctness review queue

data/
  benchmark/
    holdout.jsonl          # 200-record balanced evaluation set
    holdout_large.jsonl    # 4,625-record half-corpus evaluation
    example_pairs.json     # Entity pairs excluded from holdouts
  exports/<run>/           # Per-run viewer exports (per_statement.json + per_evidence.jsonl + export_meta.json)
  corpora/                 # Sampled INDRA Statement dumps to score
  results/                 # Evaluation results

scripts/
  run_rasmachine_monolithic.py  # Production scoring runner
  check_contamination.py        # Pre-eval gate: examples must not overlap holdout
  check_doc_anchors.py          # Live-doc guard: referenced implementation files exist
  serve_mlx.sh                  # Local MLX reader on Apple Silicon (the one logprob-capable route)
  export_representative_curations.py  # Export first-write unique-pair representative snapshot

.github/workflows/
  ci.yml                        # pytest + guards + viewer/deck checks on every push and PR

References

Gyori et al. (2023). "Automated assembly of molecular mechanisms at scale from text mining and curated databases." Molecular Systems Biology, e11325. Benchmark corpus: Zenodo 7559353
Gilda — Biomedical entity grounding
INDRA — Integrated Network and Dynamical Reasoning Assembler

Name		Name	Last commit message	Last commit date
.githooks		.githooks
.github/workflows		.github/workflows
data		data
reports		reports
research		research
scripts		scripts
src		src
tests		tests
viewer		viewer
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
Dockerfile.live		Dockerfile.live
README.md		README.md
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
run_test.py		run_test.py
uv.lock		uv.lock

Navigation Menu

Uh oh!

Repository files navigation

indra-belief-model

What this does

Input

Output

How it works

Production scoring architecture

Two-tier monolithic path

Adaptive few-shot selection

Run the scorer

Design decisions we already paid for

Setup

Dependencies

Model configuration

Usage

Score a Statement's evidence

Composition with INDRA belief

Representative INDRA curations

Score a corpus + browse the results

Observed LLM cost (per run)

Benchmark evaluation against a holdout file

How we iterate

Project structure

References

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

indra-belief-model

What this does

Input

Output

How it works

Production scoring architecture

Two-tier monolithic path

Adaptive few-shot selection

Run the scorer

Design decisions we already paid for

Setup

Dependencies

Model configuration

Usage

Score a Statement's evidence

Composition with INDRA belief

Representative INDRA curations

Score a corpus + browse the results

Observed LLM cost (per run)

Benchmark evaluation against a holdout file

How we iterate

Project structure

References

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages