spell no. 001
basically it reads student writing and flags stuff that might be a warning sign, so a real person can go take a closer look
askcan a model catch warning signs in writing well enough that it's actually worth a counselor's time, and can it explain why it flagged something
missing something matters way more than a false alarm here, so i track the miss rate on every run
i co-founded this and led the build. the first version lived in its own repo, built on Flask, Celery, Redis, and Docker running on Railway. i later moved everything to FastAPI and put the research pipeline and the live app in one repo together
the expensive mistake here is missing something, not a false alarm. so besides accuracy and F1 i track the false-negative rate on every single run, and i keep a dumb TF-IDF plus logistic-regression baseline around just to compare against. a number doesn't mean much until you check it against the simplest thing that could've worked.
the other big piece was trust. the app redacts email addresses and phone numbers before the model sees anything, and saves reviewer feedback to PostgreSQL so predictions can get checked later and reused for retraining. the code also has Integrated Gradients to show which words actually pushed a prediction one way or the other, but it's slow on a CPU, so it's switched off in the live demo
i benchmarked DistilBERT, BERT, ELECTRA, and RoBERTa, three seeds each, all on the same 232,074-sample balanced dataset and the same fixed 80/10/10 split. RoBERTa won by a lot: 99.45% F1 and 99.45% accuracy on the held-out test set with a 0.75% false-negative rate, vs 97.95% F1 and a 2.34% miss rate for DistilBERT. that's the model running live. but i'd be lying if i let that number stand alone. the labels come from which subreddit a post was in (r/SuicideWatch and r/depression vs r/teenagers), so a lot of the job is just spotting teen-forum chatter, and a plain TF-IDF baseline already gets 93.5%. on a handful of school-style test sentences, it missed even a bare "I want to kill myself." and every indirect warning sign i tried, like "everyone would be better off without me." the README has the whole audit, dataset, split, duplicates, baseline, and failure cases. the live demo sends your text to a synchronous FastAPI endpoint and gets the prediction straight back, running in Docker on Railway with PostgreSQL for feedback. a Celery and Redis queue exists in the code but the demo doesn't use it. it's decision support, not a diagnosis, every flag is meant to be reviewed by an actual person
since september 2025 it's been running at James Logan High School. so far it has surfaced nearly 100 potentially concerning submissions for educators to review, and it averaged 3.7/5 across a multi-question usefulness survey of 5 teachers
the model learned from Reddit posts, and student essays read pretty differently. how well does it actually hold up on real student writing, including the essays it never flagged, measured against what teachers decide
every run pulls from a shared base config plus a model-specific one, with a fixed seed and saved metadata, so i can rerun a result instead of just trying to remember it. the RoBERTa run that shipped (seed 42) used a learning rate of 2e-5, batch size 16, 3 epochs, and a max sequence length of 256, same as every other model in the benchmark
python src/train.py \
--base_config config/base_config.yaml \
--model_config config/model_configs/roberta.yaml \
--seed 42