Articles · Equity · AI in education · EdTech

Designing school AI for the student the system forgot

AI is most often wrong about students at the edges. How the default user gets built into school technology, and the design rules that change it.

By Dr. William Hickman · 5 min read

Every technology imagines someone. A door imagines a person tall enough to reach the handle. A form imagines a single last name and a fixed address. An AI tutor imagines a student too, and if you ask the people who build them to describe that student honestly, you usually get the same sketch: a motivated kid with a quiet room, a working laptop, strong English reading skills, a fast connection, and no disability.

I call that sketch the default user. Defaults are not evil. You cannot design for every human at once. The trouble starts when two things happen together: the system begins making decisions that matter, and the students outside the default turn out to be the same students who have been outside it before.

I have spent my career with those students, first as a special education teacher and school psychologist, later as a principal and superintendent, and now as someone who builds AI tools for schools. Here is what I have learned about designing for them.

Who falls outside the picture

Name them, because a student without a name does not get designed for.

  • A student with dyslexia who understands the math but stalls on word problems.
  • A student who uses a screen reader.
  • A student learning English who knows the concept but not yet the vocabulary.
  • A student with ADHD who needs shorter steps and moves while she thinks.
  • A student who reads questions aloud to herself because that is how she processes them.
  • A student whose only device is a phone.

Every one of them is in your district. Most AI products meet them as exceptions to handle later.

Why AI is most often wrong at the edges

A machine learning model learns by finding the patterns that show up most often. Common patterns are learned well. Rare ones are learned poorly or not at all. In data work, points far from the usual pattern are called outliers, and they are routinely smoothed over or deleted so the main pattern comes through.

That habit makes sense when the outlier is a typo. It is a disaster when the outlier is a child.

Jutta Treviranus, director of the Inclusive Design Research Centre at OCAD University in Toronto, described a striking example in 2018. She showed machine learning models built to guide self-driving vehicles a recording of a friend who propels her wheelchair backward. Every model chose to run her over. After the models were trained on more data about wheelchair users, they ran her over with even more confidence. The extra data taught them the typical case, and her friend was not typical.

More data about the middle does not help the students at the edges. It can make the middle louder.

Where it shows up in schools

Writing detectors. In 2023, a Stanford team led by Weixin Liang and James Zou ran essays by non-native English speakers through seven popular AI detectors. On average, the detectors labeled those essays AI-generated more than 61 percent of the time. Clear, simple writing, the kind we teach English learners to produce, looked like a machine.

Proctoring. Software built to flag movement and noise will, by design, flag students with ADHD, students with tics, and students who read aloud.

Timed screeners. Picture an eighth grader flagged as several grades behind. He answers slowly but correctly, and he is still building academic English. Read without a clock and with a few words explained, he may land at grade level. The number was measuring the wrong thing about him.

Early warning labels. A risk flag is only as good as what happens next. If it leads to a tutor and a call home, a false alarm costs a little extra kindness. If it leads to lowered expectations, a false alarm can cost a child years.

The paradox worth holding

Here is the other half, and it is just as true. AI may be the most powerful accessibility tool ever built: text read aloud in a natural voice, live captions, dictation, an instant explanation of a confusing word in a student’s own language, a patient tutor that breaks a giant assignment into small steps. For many students with disabilities, that is the best thing to happen to school in a generation.

The same technology closes doors when it is bolted on in front of students instead of built with them: a chatbot a screen reader cannot escape, a puzzle that asks a blind student to click on traffic lights, a voice system that cannot understand a student’s speech.

One distinction helps. Some tools are a prosthesis, such as an AAC device, captions, or text-to-speech. They stay as long as the need does, and taking them away removes access. Others are a scaffold, built to come down as a skill arrives. Plans fail when they confuse the two, either by taking away a student’s voice or by never planning how a scaffold hands the student back to people.

The design rules I follow

When I build tools that make decisions about students, I hold myself to a short list.

  1. Recruit testers from the edges, early, and pay them. For a reading tool, that means students with dyslexia and students learning English. For anything with voice, it means a range of accents and speech patterns. It goes in the budget on day one.
  2. Look at outliers before removing them. A reading speed far slower than the rest might be a glitch. It might also be a student using a screen reader.
  3. Build accommodation into the product. Extended time is a setting from the start, not a favor someone has to request by email. Tools should honor a student’s IEP or 504 accommodations by default.
  4. No false precision. If a tool cannot tell the difference, it should say so, and every flag should come with a next step that is a conversation, not a consequence.
  5. No permanent labels. Children change fast. A label that cannot be removed is not predicting a future. It is helping write one.
  6. A family view. Anything a tool says about a child should be readable by a parent, in plain language and in the home language, with a clear way to say “this is wrong.”

And one boundary for every school: decisions about eligibility, services, and accommodations belong to the IEP or 504 team and the family, never to a dashboard.

The next play

  1. Name your edges. For your most consequential AI tool, write down the three student groups most likely to be failed by it.
  2. Ask for results by group: English learners, students with disabilities, and students of different races. If the vendor has none, require a local pilot before renewal.
  3. Turn off behavior flags for students with documented accommodations, by default.
  4. Check where assistive devices sleep. Any device that stays at school on nights and weekends should go home with a charger and a one-page card for families.
  5. Invite your special education and English learner leads to your next technology decision, before the contract is signed, not after.

The full investigation, including the five BUILT Questions for any system you buy or build, is in Built for Someone Else.

Dr. William Hickman (Dr. Philip William Van Hickman) is an inventor, former superintendent, and the founder of MindAble Solutions. He is the author of eleven books, including The Intelligent Classroom. About him · Official site