Text features (experimental)

Use this skill when you have many free-text rows and a supervised target, and you want calibrated numeric features instead of bag-of-words. The main model proposes operationalizable questions; Jev answers each row; the sidecar fits a model on a development split and scores the frozen test split once.

This is experimental. Turn on text_features under Customize → Experimental (the master experimental-judgment switch must also be on). Headless: set OPENAI4S_EXPERIMENTAL_JUDGMENT=1 and OPENAI4S_JUDGMENT_TEXT_FEATURES=1. When the capability is off, the helpers return status: "disabled" and do not raise.

When to use it

• Lots of free text (notes, abstracts, reports) plus a label or score.
• You need features a colleague can read: each column is a question, a probability or graded expectation, and a spread.
• You can hold out a test set and leave it untouched while questions change.

When not to use it

• Small samples, or no supervised target — there is nothing to freeze against.
• Sensitive data. Selected row text is sent to api.typesafe.ai. The service is hosted in the United States. Do not enable this on clinical notes, secrets, or anything that must not leave the machine.
• You need a human gold standard. Jev features are not a human gold standard. They are calibrated model judgments of the text you sent, with measurement error. Do not treat a Noul probability as a verified fact.

Data that leaves the machine

Every featurize call sends the user-selected row text in state.text to TypeSafe Jev, together with the question instructions. Identifiers go in state.id so you can audit which rows were judged. Quote location and numeric modeling stay in this sidecar.

Import and run

The directory contains a hyphen, so import it with importlib:

from importlib import import_module

tf = import_module("text-features.kernel")

questions = tf.propose_questions(
    "Predict whether an abstract reports a significant clinical result.",
    examples,
    n=12,
)
table = tf.featurize(
    rows,
    questions,
    text_field="text",
    id_field="id",
)
study = tf.run_feature_study(
    rows,
    target="label",
    split_by="patient_id",  # or time_col="date"
    rounds=3,
    text_field="text",
    id_field="id",
)

propose_questions asks host.llm for Noul (yes/no facts) and Score (written-level grades) questions, then validates and deduplicates them.

featurize calls host.judge("features.custom", ...) once per row. Each Noul becomes one column, P(yes). Each Score becomes two columns: the expected level normalized to [0, 1], and the standard deviation of that distribution on the same scale. Unavailable rows are filled with NaN and counted; they are not replaced with a default. Every column keeps the question text and template version.

run_feature_study reuses audit-dataset, plan-ml-experiment (grouped or chronological split), and evaluate-model (metrics and bootstrap 95% CI). Question edits, feature screening, and thresholds use the development rows only. The test split is judged once, after the question set is frozen. The report includes lift versus a constant baseline and a bootstrap interval, plus the question-set version and per-feature provenance. Binary targets are encoded in the reported target_classes order; predictions are probabilities of the second class. Rows with missing targets are excluded from fitting and metrics. Baseline and feature-model lift use the same rows with available predictions.

Modeling uses numpy / pandas / scikit-learn when the science extra is installed. They are imported lazily. They are not core dependencies. Without them the sidecar still featurizes and falls back to a linear least-squares fit.

Required output

Name the split, the frozen question-set version, each feature's source question and template version, unavailable-row counts, cost (requests and tokens), the baseline, the lift, and the bootstrap interval. Never describe the features as labels, facts, or a human gold standard.