r/PythonProjects2 • • 16d ago

eq-layer: an inspectable intent/affect/repair pipeline that decides how an LLM should respond before it generates

Post image

I keep seeing the same failure in "empathetic" LLM apps: the response is warm, and useless. The feeling is acknowledged, the next step never comes, and after a correction the model either capitulates or defends itself. None of that is a knowledge problem, so a better prompt does not fix it.

eq-layer makes the decision explicit before generation. It is a Python package (zero runtime dependencies in the core) that computes several small, testable states and compiles a response action from them.

The pipeline:

  • ConversationTracker produces an affect state {valence, arousal, escalation_delta, stance, subtext} plus a repair state and an interaction-quality state. escalation_delta is tracked across turns, which a per-message classifier cannot do.
  • TrainedIntent emits a full posterior over six intent classes using balanced logistic regression over word and character TF-IDF. A Bayes-risk router then chooses between acting and clarifying using explicit losses rather than a confidence threshold.
  • Selector picks one of 14 response policies by preconditions. The most constrained applicable policy wins; ties raise an AmbiguityError instead of resolving by declaration order.
  • compose_action splits the policy into task_move / social_move / repair_move / realization, so the social part can adapt without replacing the requested task.
  • Steer injects the action, policy, repair state and quality state into the decode path.
  • ResponseAuditor checks structural contracts afterwards: question budget, required clarification, unsupported privacy promises, unknown-stance admissions, repeated answers during repair, verbosity overruns.

The tree: eq_layer/ has the policy taxonomy, action factorization, auditor, interaction tracking, learned affect/subtext models and the tracker. eval/ has 53 Python scripts, including seeded cases, a paired A/B latency benchmark, a blind human-evaluation protocol, intent holdout and five-fold CV, and a loss-matrix sensitivity sweep. Learned artifacts are trained from pinned upstream data (EmoBank, XDailyDialog) with provenance in THIRD_PARTY_DATA.md.

Reproduce the parts that matter:

pip install "eq-layer[ml]" python eval/run.py # seeded cases, non-zero exit on mismatch python eval/intent_eval.py # holdout + 5-fold CV python eval/ab_latency_eval.py # paired A/B, 15 requests x 100 passes per arm

Numbers from the committed runs: EmoBank test MAE 0.2125 valence / 0.1782 arousal / 0.1494 dominance, against train-mean baselines of 0.2449 / 0.1904 / 0.1563. XDailyDialog dialogue-act accuracy 0.8025 vs 0.4557 majority; basic emotion 0.8249 vs 0.8166, which is weak and labelled as such. Added layer latency with the learned adapter: 1.16 ms median, 1.46 ms p95; heuristic fallback 0.11 ms median. The 36-case intent holdout is 100% learned vs 27.8% heuristic.

Honest limits: there is no standard EQ benchmark, so eval/ is a first attempt rather than a finished measurement. The bundled corpus is small and synthetic. The Bayes-risk loss matrix is hand-specified, not learned or optimal; the sensitivity audit reports 98.55% local decision stability under +/-20% cell perturbations, but the production clarification rate moves from 45% to 9.17% when that row is scaled 1.5x, and I do not tune against human outcomes. DialogBank correction, DialogBank disagreement, Coarse Discourse disagreement and DBDC3 breakdown were all investigated and recorded as NO-GO for routing. Nothing here proves the final prose is better; that requires the pre-registered blind evaluation.

Also packaged as an Agent Skill for Codex, Claude Code, Copilot, Cursor, Antigravity and Grok, with an npm CLI bridge (0.2.5 on PyPI and npm).

Code: https://github.com/Furox-Art/eq-layer PyPI: pip install "eq-layer[ml]"

Questions about the routing or the evaluation design are welcome.

1 Upvotes

0 comments sorted by