WaterSheep ships an Apache-2.0 open-source alternative to Jev

What shipped on 10 October?

The dated public announcement is Samrat Dutta’s DEV Community post “WaterSheep: an open-source alternative to Jev,” published 10 October 2026 at 10:02 UTC. The same day the GitHub repository SamratDuttaOfficial/WaterSheep was pushed (last_push 15:23 UTC) and the Hugging Face model card samratduttaofficial/WaterSheep was lastModified at 15:23 UTC. The Hub card labels the release WaterSheep version 0.1.0 with checkpoint id watersheep-20260928-125452.

That puts WaterSheep in the same narrow product class as TypeSafe’s Jev funding story, Microsoft-Decision-1 in Foundry, and HAL-X’s free THX-01 API: models that return typed decisions with probabilities instead of free-form prose. WaterSheep’s differentiator, as stated by the author, is that code, weights, training scripts, and a Jev-compatible local server are Apache-2.0.

Ai Lookout opened the DEV HTML, the GitHub README, LICENSE, NOTICE, and pyproject.toml, plus the Hub README and config.json, on 11 October 2026. We did not download model.safetensors, run the ONNX file, or call the live Hugging Face Space with production traffic.

What kind of model is WaterSheep?

WaterSheep is a fine-tune of answerdotai/ModernBERT-base with a decision head. The Hub tags list transformers, onnx, safetensors, zero-shot-classification, decision-model, calibration, multi-label, and jev-alternative. The pipeline tag is zero-shot-classification. English is the only language listed.

Question types on the README are noul (yes/no probability), choice (best option among labels), score (expected level on a digit scale), and multi (every option above a threshold). Every answer includes a probability for each option. The author highlights multi-label as a capability Jev’s public surface does not expose the same way.

Calibration is described as a temperature per question type fitted on a validation split. Training data mixes openly licensed public datasets listed in NOTICE with synthetic decisions from Qwen3.5-4B. Those dataset names appear on the Hub card; we did not re-download or re-license them.

How do you run it, and how Jev-compatible is the server?

The lightest path is transformers: pip install transformers torch, then pipeline(model="samratduttaofficial/WaterSheep", trust_remote_code=True). trust_remote_code is required because modeling_watersheep.py and pipeline_watersheep.py ship in the Hub repo. Offline loading after hf download is documented.

For a Jev-shaped HTTP surface, the Python package installs from GitHub and exposes watersheep --serve on port 8766. The README states it answers POST /v1/systemone with the same request and response shape as Jev, so TypeSafe’s Python SDK can point at TYPESAFE_BASE_URL=http://127.0.0.1:8766. The author says any API key value works locally and that he tested typesafe-sdk 0.7.2; the JavaScript SDK is marked untested.

That local-server story is useful next to hosted decision APIs such as CARVE’s open Jev-like stack and Vega’s physics-oriented decision model. Compatibility claims are about request shape, not about matching Jev’s accuracy, latency, or pricing. Ai Lookout did not start the server or replay the author’s triage.py snippet.

Additional surfaces documented on the card: a browser JS helper at samratduttaofficial.github.io/WaterSheep/watersheep.js, Hugging Face Inference Endpoints JSON, onnx/model_quantized.onnx for ONNX Runtime, and a public Space demo. We confirmed the Space API object resolves; we did not exercise the interactive demo.

Which scores are the author’s?

The Hub README table reports 77.8% accuracy and 0.026 expected calibration error on an in-distribution test split, and 61.2% accuracy with 0.043 ECE on held-out datasets never seen in training. Those are author measurements. Per-benchmark rows include strong safety-style sets (prompt_injection 91.4%, hatecheck 75.1%) and weak legal rows (legal_abercrombie 21.1%, legal_function_of_decision_section 24.3%). Several legalbench tasks show ECE above 0.25.

The DEV post and README both urge confidence thresholds: act automatically when confident, escalate when not. The coverage figure marks example thresholds at 0.70, 0.90, and 0.97. Rating-scale questions are called the weakest type. None of these figures were reproduced here.

Compared with Microsoft’s vendor-run 36-benchmark card for Microsoft-Decision-1 or HAL-X’s THX-01 ticket tables, WaterSheep’s evidence is a single author harness with transparent weak rows. That honesty helps readers; it is not an independent bake-off.

What are the license and product limits?

LICENSE is Apache License 2.0. NOTICE lists dataset attributions. The Hub card and README state English only; long inputs are truncated and the package marks truncated answers with truncated: true; probabilities are calibrated on data like the training mix; and the model is “Not for high-stakes decisions (medical, legal, financial, hiring) on its own.”

The author explicitly disclaims affiliation with TypeSafe. Retraining is documented via ./scripts/run.sh. That is a genuine open stack for the decision-model niche, with the usual fine-print that Apache weights plus third-party datasets still require reading NOTICE before commercial shipping.

What should builders do next?

If you already route tickets with typesafe-sdk, point a staging client at a local WaterSheep server and compare agreement, calibration, and latency against your current Jev or Decision-1 path on a held-out ticket set you own.

If you need multi-label issue tagging, exercise type=multi first; that is the clearest API difference called out in the announcement. Keep a human fallback below your chosen confidence cutoff, especially on legal and rating-scale questions where the author’s own tables are weakest.

Do not treat the 77.8% / 61.2% rows as portable to your domain. Log disagreement cases and, if you retrain, keep the evaluation script and splits as first-class artifacts the way the repository’s results/ folder does.

• Pin watersheep-20260928-125452 or a git SHA before any production experiment.
• Measure noul/choice/score/multi separately; do not average them into one vanity score.
• Keep high-stakes queues on a human or a separately evaluated hosted scorer.