Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
316ab65
feat: add Qwen3.5/3.6 model support (dense + MoE)
johndpope May 15, 2026
e07c4cb
feat: add LDLM latent diffusion autoencoder module
johndpope May 15, 2026
c8de26f
feat: LDLM training pipeline + transformers 5.8.0 compat fixes
johndpope May 15, 2026
5246eb7
feat: config-driven wandb logging for LDLM
johndpope May 15, 2026
dd93e20
feat: wandb dashboard template for LDLM training
johndpope May 15, 2026
b363683
feat: auto-prune old checkpoints (keep latest 2)
johndpope May 15, 2026
311073f
fix: handle nested Qwen3_5MoeConfig (text_config.hidden_size)
johndpope May 15, 2026
ac56db4
fix: auto-compute decoder nhead from dim (2048/16=128)
johndpope May 15, 2026
9b1b6c0
fix: skip FSDP, use device_map=auto for frozen encoder
johndpope May 15, 2026
7227474
fix: load frozen encoder on CPU (device_map=cpu) to avoid OOM
johndpope May 15, 2026
2821420
feat: LDLM benchmark + bugfixes for Qwen3.6-27B training
johndpope May 15, 2026
b8dc423
docs: add LDLM section with throughput benchmark and training guide
johndpope May 15, 2026
bdcfd58
feat: add Qwen3.6-35B-A3B LDLM benchmark (3,238 tok/s) and update docs
johndpope May 15, 2026
fdeeeea
docs: add reproducible benchmark commands and honest assumptions/caveats
johndpope May 15, 2026
c522280
feat: multi-GPU training support (encoder on GPU0, trainable on GPU1)
johndpope May 15, 2026
4c49b69
feat: log reconstruction text samples to wandb (target vs predicted)
johndpope May 15, 2026
cdf3db0
feat: implement LDLM paper training recipe (arXiv:2605.07933)
johndpope May 16, 2026
886cdb0
fix: device mismatch in hidden state normalization buffers
johndpope May 16, 2026
b6c901e
fix: use native checkpoint manager, increase save_steps to 500
johndpope May 16, 2026
aa9c709
fix: reduce diffusion head depth to 3 to avoid OOM with self-conditio…
johndpope May 16, 2026
332e7fa
docs: add LDLM architecture comparison table (paper vs our Qwen3.6 im…
johndpope May 16, 2026
400bd93
ok
johndpope May 16, 2026
886b9a2
feat: add Cola DLM (Text VAE + block-causal DiT) as Repr-Align aux head
johndpope May 16, 2026
fdedf3a
feat: precomputed anchor cache for Repr-Align + local hardware docs
johndpope May 17, 2026
bba845e
feat: standalone-run fallbacks for single-GPU smoke tests
johndpope May 18, 2026
54efadc
fix: Repr-Align anchor cache mismatch + Qwen3 FA2 attr for transforme…
johndpope May 18, 2026
91536e2
feat: torchao weight-only quantization of frozen Linears for one-laye…
johndpope May 18, 2026
b131562
[distributed] feat: DeepSpeed ZeRO integration + CPU/NVMe offload
johndpope May 18, 2026
f876b59
fix: deepspeed weight loading via distributed parallel_load_safetensors
johndpope May 18, 2026
339cf57
wip
johndpope May 18, 2026
6220676
fix: DeepSpeed checkpointer classmethod + NVMe offload hardening + sy…
johndpope May 18, 2026
5b04d3d
feat: Vast.ai cloud training scripts for 27B Repr-Align
johndpope May 19, 2026
fd28b1d
docs: cloud training guide (Vast.ai + S3 setup)
johndpope May 19, 2026
5fe0894
fix: code review issues — remote_device, repo URL, CUDA_VISIBLE_DEVIC…
johndpope May 19, 2026
a18114f
fix: inject HF_TOKEN + AWS creds into cloud onstart scripts
johndpope May 19, 2026
d1dba3c
fix: lazy-load seed_omni to avoid torchvision import in text-only tra…
johndpope May 19, 2026
7e46e98
chore: add torchvision to requirements.txt and regenerate uv.lock
johndpope May 19, 2026
5bb04f2
fix: enable CPU optimizer offload in cloud_27b.yaml to avoid OOM
johndpope May 19, 2026
f885cf1
feat: save_total_limit + WandB + 200-step smoke config
johndpope May 19, 2026
eb9939b
feat: sanity_infer.py + push_ckpt_s3.sh for checkpoint validation
johndpope May 19, 2026
83c3d0b
fix: disable grad checkpointing, fix train_size tokens, add max_steps…
johndpope May 19, 2026
81b2ed0
fix: disable epoch-end saves to prevent 211GB ZeRO checkpoint filling…
johndpope May 19, 2026
b25993c
feat: save_optimizer_state flag to skip 211GB ZeRO checkpoint
johndpope May 19, 2026
20f1931
fix: fresh WandB run name to avoid step counter conflict from restarts
johndpope May 19, 2026
0f42298
docs: random sub-sample trick for Repr-Align memory reduction (FSRT-i…
johndpope May 19, 2026
0fca6fd
feat: random token sub-sampling for repr_align loss to cut gradient m…
johndpope May 19, 2026
be2d9ea
feat: enable repr_align_sub_sample_ratio=0.25 for 27B cloud training
johndpope May 19, 2026
3d4d8c7
fix: switch to adamw optimizer + disable time-based ZeRO checkpoint
johndpope May 19, 2026
ad7b3e1
config: local 27B Repr-Align config for 5090 + RTX PRO 4000
johndpope May 19, 2026
4590238
config: point local 27B config to full 100k dataset
johndpope May 19, 2026
6688c03
config: finalize local 27B config with anchor cache at 1k examples
johndpope May 19, 2026
57746c0
feat: debug logging + CLAUDE.md Vast.ai section
johndpope May 19, 2026
eb79705
fix: disable wandb in cloud config (API key not configured on instance)
johndpope May 19, 2026
92e1ca4
fix: re-enable wandb (API key now configured)
johndpope May 19, 2026
154461b
feat: NaN abort guard in training loop
johndpope May 19, 2026
98f4a23
fix: freeze visual encoder for plaintext Repr-Align training
johndpope May 19, 2026
f13a956
fix: error_rank0 → warning_rank0 in NaN abort; add fwd debug logging
johndpope May 19, 2026
b34787a
debug: print-based fwd diagnostics for NaN loss_components
johndpope May 19, 2026
e26afe2
debug: sys.stderr fwd diagnostics for NaN loss_components
johndpope May 19, 2026
ca54d70
fix: load Qwen3.6-27B VL weights into veomni text model
johndpope May 19, 2026
18966f6
fix: enable SDPA support for Qwen3_5PreTrainedModel
johndpope May 19, 2026
c2b0d89
fix: convert Qwen3_5TextConfig → Qwen3_5Config in VL alias
johndpope May 19, 2026
14dde6d
fix: handle nested rope_theta in Qwen3_5RotaryEmbedding
johndpope May 19, 2026
5716c8b
debug: log shape mismatch instead of crashing in load_hf_weights_zero3
johndpope May 19, 2026
46b0cb8
fix: restore head_dim after Qwen3Config super().__init__() override
johndpope May 19, 2026
878a77c
fix: apply_rotary_pos_emb_partial use unsqueeze not transpose
johndpope May 19, 2026
b60e823
docs: record Vast.ai rate + provisioning checklist in CLAUDE.md
johndpope May 19, 2026
f3251ec
docs: record instance 37044404 billing breakdown + cost lessons
johndpope May 19, 2026
3709152
fix: add --price (bid cap) to Vast.ai provisioning scripts
johndpope May 19, 2026
b12855a
feat: 3 causal diffusion variants + MTP head for Qwen3.5/3.6 MoE
johndpope May 19, 2026
5334692
updates
johndpope May 19, 2026
94af175
Create qwen3_1_7b_cola_card_smoke.yaml
johndpope May 19, 2026
67f784e
fix: Cola DLM smoke test - device, ZeRO-2, grad_norm issues
johndpope May 19, 2026
9a56982
fix: smoke config - rmpad_with_pos_ids false, ZeRO-2 mode
johndpope May 19, 2026
d239d6a
feat: per-layer subsampling for repr_align loss (all Qwen3 variants)
johndpope May 19, 2026
a2a265c
perf: hook-based selective hidden state capture for repr_align (all Q…
johndpope May 19, 2026
7330ea2
docs: document repr_align subsampling knobs in CLAUDE.md
johndpope May 20, 2026
8e412a1
fix: correct 27B align_layers index to 64 (last layer), not 63
johndpope May 20, 2026
706f674
feat: smoke config — all 28 layers in pool, 4 sampled per step
johndpope May 20, 2026
81352b7
fix: guard all_reduce result as tuple before zip in loss component re…
johndpope May 20, 2026
b9f0483
chore: add 4-layer layer-sample smoke config and 27B GGUF forward test
johndpope May 20, 2026
7edaf07
docs: add repr_align layer+token subsampling section with measured stats
johndpope May 20, 2026
8420cdd
fix: update 27B GGUF path to non-MTP variant
johndpope May 20, 2026
e30f86c
PersistentSparseAdam: rotating optimizer-state subsampling (-4x VRAM)
johndpope May 20, 2026
a4ceb25
fix: remove ds_offload_optimizer: null from smoke config (argparse re…
johndpope May 20, 2026
76c9a7f
ok
johndpope May 20, 2026
7b2c918
QLoRA + Repr-Align: train 1.7B in 3.5 GiB (was 31 GiB)
johndpope May 20, 2026
27e5932
updates
johndpope May 20, 2026
456edca
fix QLoRA 27B smoke test: skip vision encoder, force GPU placement, d…
johndpope May 20, 2026
0376443
add QLoRA 27B Repr-Align reproduction guide to README
johndpope May 20, 2026
ca4be88
Create qlorafy_27b_train.yaml
johndpope May 20, 2026
5cd231c
add QLoRA wandb metrics, 27B training config with wandb enabled
johndpope May 20, 2026
ff4aa3b
add generation probe: log diffusion samples to wandb every 100 steps
johndpope May 20, 2026
992bada
updates
johndpope May 20, 2026
c25910d
ok
johndpope May 20, 2026
48fae6b
ok
johndpope May 20, 2026
1303f77
VFM: noise adapter + flow map wrapper + joint training objective
johndpope May 20, 2026
52aa0a3
perplexity
johndpope May 20, 2026
f3ae467
VFM training script, configs, NaN skip fix, eval pipeline
johndpope May 20, 2026
ce42e01
fixes for recreation
johndpope May 20, 2026
6100431
qlora - replay loss
johndpope May 21, 2026
8837a31
dump anchros
johndpope May 21, 2026
e922818
wip
johndpope May 21, 2026
ecf910b
fix anchor precompute: hooks instead of output_hidden_states, 4-bit q…
johndpope May 21, 2026
11d9fa0
updates
johndpope May 21, 2026
a056edf
feat: multi-block inference decoder (d3LLM pipelined parallel decoding)
johndpope May 21, 2026
f10cdd4
ok
johndpope May 21, 2026
a485e7f
ok
johndpope May 21, 2026
4bd6518
fix: align QLoRA configs with anchor cache (max_seq_len=2048, rmpad) …
johndpope May 21, 2026
e033c19
fix: make d3LLM trajectory route + Gated DeltaNet runnable
johndpope May 21, 2026
986d24c
ok
johndpope May 21, 2026
6549945
Merge pull request #4 from scrya-com/develop
johndpope May 21, 2026
4bc352b
Merge pull request #5 from scrya-com/feature/d3llm-trajectory-distill…
johndpope May 22, 2026
af78458
Merge pull request #6 from scrya-com/develop
johndpope May 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -181,4 +181,4 @@ eval/eval_infill/infill_results
eval/eval_infill/infill_results/*

data
logs/
logs/CLAUDE.md
81 changes: 81 additions & 0 deletions .opencode/skills/create-issue/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
---
name: create-issue
description: Use when the user asks to create a GitHub issue. Gathers context from the current conversation (findings, commands, file changes, wandb links) and drafts a well-structured issue with reproduce steps, results tables, and links. Use ONLY when the user explicitly asks to create or draft a GitHub issue.
---

# Create GitHub Issue

When the user asks to create a GitHub issue, follow this workflow:

## 1. Determine the target repo

- Run `git remote -v` to list remotes.
- Default to `origin`. If the user specifies a different remote (e.g. "send to upstream"), use that.
- If the user says "not upstream" or similar, confirm which remote to target.

## 2. Gather context from the conversation

Pull from the current session everything relevant to the issue:

- **What was done** — features added, bugs fixed, experiments run
- **Commands to reproduce** — exact shell commands the user ran
- **Results** — loss curves, benchmark numbers, comparison tables
- **Wandb links** — grep logs for `wandb` URLs or ask the user
- **Bugs found and fixed** — with file paths and line numbers
- **Key files changed** — with one-line descriptions
- **Next steps / open questions**

If any of these are missing, ask the user before proceeding.

## 3. Draft the issue body

Structure:

```
## What is [topic]?

One-paragraph summary of the feature/experiment/finding.

## [Key finding or result]

Tables, numbers, comparisons. Use GitHub-flavored markdown tables.

## Reproduce

Step-by-step commands the reader can copy-paste. Include:
- Prerequisites (hardware, data, model weights)
- Exact commands with all flags
- Any environment variables needed

## Wandb / Logs

Links to wandb runs, log file paths, or other artifacts.

## Bugs found and fixed

Numbered list with file:line references.

## Key files

Table of file → purpose.

## Next steps

- [ ] Checkbox items

## Open questions

Numbered list of unresolved items.
```

## 4. Create the issue

```bash
gh issue create --repo <owner>/<repo> --title "<title>" --body "<body>"
```

Use a HEREDOC for the body to preserve formatting. Target the repo determined in step 1.

## 5. Confirm

Return the issue URL to the user.
Loading