Blocking Deepfake and AI Companion Tools in Schools

Unprotected vs. Protected

How one fabricated image becomes a district crisis

Administrators who have handled a deepfake incident describe the same pattern: the harm was done quickly, quietly, and largely on school-adjacent infrastructure — a school Chromebook, the campus Wi-Fi, a photo taken at a school event. By the time the target or a parent reports it, the image has been re-shared beyond anyone's ability to recall it.

Unprotected Network

Without Category Filtering

A public photo of a classmate is saved from social media — no barrier at the source
The student opens a face-swap site during class — the request goes through unblocked
A fabricated image is generated in minutes and spread through group chats
The image has been re-shared beyond recall before an adult hears about it
Crisis response begins after irreversible harm — discipline theater, not prevention
District cannot show it took reasonable technical precautions

Protected Network

With Category Filtering Active

Generation sites blocked at the category level, not one URL at a time
The student attempts a face-swap site — the request is blocked and logged instantly
Without a generated image, there is nothing to share or redistribute
Enforcement follows take-home Chromebooks off campus via device-level policy
Attempted visits appear in reports, giving counselors an early signal
New and renamed tools inherit the block automatically with daily updates

Know the Threat

Three families of AI tools that put students at risk

These are not fringe corners of the web. They are polished consumer apps, heavily marketed on the same social platforms students use every day, and most of them work in a browser with no installation and no age check that matters.

Threat Category	Severity	Scale	How It Harms Students	Specific Risks
Deepfake & Face-Swap	Critical	200+ domains	Let a user place a real person's face into a photo or video they never appeared in, including sexualized "undress" services built for exactly one purpose. All the raw material a student needs — a yearbook photo, a team picture, a screenshot from social media — is already public.	Non-consensual intimate imagery of real minors Harassment with fabricated "evidence" Criminal exposure for students and liability for district
Voice Cloning	High	250+ services	Reproduce a recognizable copy of someone's voice from a short clip — a classmate's story post, a teacher speaking at an assembly, a coach's announcement. Because audio feels harder to fake than images, listeners tend to believe it.	Fake confessions and fabricated arguments Impersonation of staff or family members in scams Social engineering of front-office staff
AI Companion & Character Chat	Critical	470+ services	Simulate a friend, confidant or romantic partner. Engineered to maximize attachment and time-on-app, many drift into adult conversation with no meaningful age gate. Everything a lonely teenager tells them — fears, family details, location, photos — lands with an unvetted company.	Emotional dependency by design Grooming-adjacent exposure patterns Student data exfiltration without FERPA-aware contracts

What You Are Protecting Against

The specific harms these categories carry

Each of these has already produced real incidents in K-12 settings. None of them requires a sophisticated attacker — only an ordinary student, an ordinary phone photo, and an unblocked website.

Harassment with Fabricated "Evidence"

Bullying escalates sharply when the bully can manufacture proof: a fake image of a classmate in a humiliating situation, fake audio of them insulting a friend.

Targets struggle to defend against authentic-looking content
Material persists in chats and reposts long after punishment
Category-level blocking prevents generation at the source

Non-Consensual Intimate Imagery

"Undress" and nudify services turn any ordinary photo into sexualized imagery of a real minor. Beyond the devastating impact on the victim, possession and distribution can carry criminal exposure.

Serious liability questions for districts whose equipment enabled it
Output produced in minutes from any public photo
Blocking the generation site eliminates the entire chain

Impersonation and Scams

Cloned voices are convincing enough to fake a principal's announcement, a parent's phone call, or a "friend" asking for money or credentials.

Staff are prime targets for voice-clone scams
Audio impersonation used to inflame communities
Social engineering of front-office staff into disclosing information

Emotional Dependency by Design

Companion apps monetize attention, so they are optimized to be endlessly agreeable, always available and hard to leave. For an isolated adolescent this can crowd out real friendships and counseling.

Several services steer toward romantic or explicit territory
Unthinkable interaction from any human adult in a school
Repeated access attempts surface as wellbeing signals

Grooming-Adjacent Exposure

Ungoverned one-on-one chat that normalizes secrecy, flattery and sexualized talk teaches exactly the interaction patterns predators exploit.

Private channel with no oversight or mandated reporter
Not an environment a school should route minors into
Whether bot or human hiding behind one, the risk is identical

Student Data Walking Out the Door

Students confide in these tools: names, addresses, schedules, family conflicts, mental-health struggles, photos. That information flows to companies with no vetted privacy agreement.

No FERPA-aware contract and sometimes no identifiable operator
No district can negotiate terms with 470 chat services
Blocking the category is the only practical control

Speed of Harm

A safety problem that moved faster than any policy cycle

School leaders have spent years building thoughtful responses to social media, cyberbullying and explicit content. Generative AI compressed a decade of that curve into a couple of school years.
A student who once needed technical skill to fabricate an image now needs a free website and thirty seconds, and the harm lands on a real, identifiable child in your building.
Essay writers and answer engines raise academic-integrity questions, and we cover those separately in blocking AI cheating tools in schools. Deepfake, voice-clone and companion services are a different problem: nobody is weighing their instructional upside.
These services rename, clone themselves and register fresh domains constantly. A manually maintained deny list falls behind within weeks. Our blocklist is screened daily against roughly 300,000 newly registered domains.

Anatomy of an Incident — The Intervention Point

1. Source: a public photo of a classmate is saved from social media.
2. Generation: the student opens a face-swap site during class. This is the step filtering removes — the request is blocked and logged.
3. Distribution: without a generated image, there is nothing to share.
4. Aftermath: instead of a crisis response, the school sees a blocked-attempt report and can follow up quietly with education, not discipline theater.

The window between "idea" and "irreversible harm" can be under five minutes. Prevention has to sit inside that window — after the fact is too late.

Category Isolation in Action

Deepfake / face-swap: blocked — all grades, all devices, all networks
Voice cloning: blocked — all grades, all devices, all networks
Companion / character chat: blocked — all grades, all devices, all networks
Instructional AI: decided separately, by grade band — never affected by safety blocks

A crude "block anything AI" rule catches translation aids, accessibility tools and the writing assistants a curriculum may deliberately adopt. Category isolation prevents that collision.

Precision Blocking

Safety blocks that don't hijack the AI conversation

The most common reason districts delay acting on deepfake and companion tools is that "AI" gets debated as one giant question. Category isolation dissolves that deadlock: the three safety families can be blocked today, unanimously.
Every domain in the blocklist carries a specific category and sits alongside the 120M+ domains in our broader school web filtering database — the deepfake block never collides with the chemistry class using an approved tutor bot.
Enforcement has to travel. Most misuse happens in bedrooms, not classrooms, on the district Chromebook that went home in a backpack. Policy that follows managed devices off campus is covered in depth in our Chromebook and 1:1 device filtering solution.

Coverage at Scale

Why these tools belong in your CIPA posture

The Children's Internet Protection Act was written before generative AI existed, but its logic maps onto these services cleanly. Schools and libraries taking E-Rate discounts certify that a technology protection measure blocks material that is obscene or harmful to minors — and a site whose function is generating sexualized imagery of real people, or holding adult-themed conversations with children, sits squarely inside that description. For the full legal picture, see our plain-English guide to what CIPA requires.

200+ Deepfake & face-swap tools isolated and blocked

250+ Voice cloning services identified and categorized

470+ AI companion chat services in the blocklist

300K+ Newly registered domains screened every day

18 AI categories with 165+ subcategories in the AI blocklist

16,328+ AI domains indexed across all categories

Daily Updates flowing to your filter automatically

CIPA CIPA also obligates schools to monitor minors' online activity and educate students on appropriate behavior, including interacting safely with others online

Putting It into Practice

Five steps to a defensible safety posture

Most districts complete this inside a single maintenance window. Nothing here requires new hardware or a policy rewrite.

01

Adopt Safety-First Defaults Configure

Confirm the deepfake/face-swap, voice-cloning and companion-chat categories are set to block in your base policy, before any grade-band tuning. These are the categories where "temporarily open" is never an acceptable state.

02

Apply the Block to Every Grade Band Scope

Unlike video or social categories, there is no age at which these services become appropriate on a school device. Apply the block uniformly from elementary through high school, staff networks included — staff devices are where cloned-voice scams tend to land.

03

Extend Enforcement to Take-Home Devices Coverage

Push the same policy to 1:1 Chromebooks and loaner laptops so the block holds on home Wi-Fi and hotspots. Off-campus hours are when companion-app use peaks and when fabricated-image incidents are typically set in motion.

04

Brief Counselors, Deans and SROs Personnel

Make sure the people who handle student welfare know these categories are blocked and what a blocked-attempt report looks like. Repeated attempts to reach companion chat can be a wellbeing signal worth a gentle conversation, not a disciplinary one.

05

Review Reports Each Term Audit

Category-level reporting shows attempted access trends by building and grade, feeds your internet-safety education planning, and gives you the audit trail that supports the E-Rate certification. Ten minutes a term keeps the posture current.

Across the District

What blocking these categories changes for each role

A safety posture only sticks when the people responsible for student welfare, discipline and infrastructure all see their part of the problem addressed.

Superintendents & Boards

A concrete, documented answer to the question parents and reporters now ask after every headline incident: what has the district done about deepfakes?

Three category blocks, enforced everywhere, with logs
Adopted without waiting on the broader AI strategy
Demonstrable technical precautions for liability defense

Principals & Counselors

Fewer fabricated-image crises to manage after the fact, and an early-warning signal before harm occurs.

Blocked-attempt reports flag students circling harmful tools
Time for a conversation instead of an investigation
Companion-chat access patterns as wellbeing signals

Technology Directors

One decision instead of hundreds: no chasing individual app names, no maintaining a homegrown deny list that ages out weekly.

Categories update themselves daily
Plug into the filtering you already run
Produce the reporting your E-Rate paperwork leans on

Beyond the Block Page

Filtering is the floor, not the whole answer

A block page stops the easy path; it does not teach a student why fabricating an image of a classmate is devastating, or give a lonely teenager something better than a chatbot.
The districts that handle this well pair the technical control with clear reporting channels for students who are targeted.
Counseling capacity for students showing dependency patterns, and age-appropriate lessons on synthetic media as part of the digital-citizenship curriculum CIPA already expects.
An internet safety program that says nothing about the AI tools students actually misuse is a program that has quietly gone out of date. Treating deepfake, voice-clone and companion categories as part of the harmful-to-minors baseline keeps the certification honest.

A practical pairing: when the filter logs repeated attempts to reach companion-chat services from one student's device, route that signal to a counselor rather than a dean. The same data that proves compliance can quietly surface a young person who needs a human to talk to — which is, in the end, the point of all of this.

Questions

What school leaders ask about deepfake and companion blocking

No. Companion and character chat is its own category within the blocklist's 18 categories and 165+ subcategories, entirely separate from writing assistants, tutoring bots or research tools. You can block every companion service today and still pilot an approved instructional chatbot tomorrow — the two decisions never touch the same switch.

CIPA requires blocking material that is obscene or harmful to minors, and it requires monitoring and educating students about safe online interaction. Services built to sexualize images of real people, or to hold adult-themed private conversations with children, fit that harmful-to-minors framework naturally. CIPA does not name these tools — it predates them — but an auditor asking how you protect minors will expect an answer that includes them.

A school filter governs school-managed devices and networks; it cannot reach a personal phone on home Wi-Fi, and no vendor should claim otherwise. What you can control is meaningful: district Chromebooks stay filtered wherever they go, campus networks are covered for every device on them, and your education program reaches the students themselves — which is the only control that follows them everywhere.

That churn is exactly what the blocklist is built for. We screen roughly 300,000 newly registered domains every day and add new AI services to their categories as they appear, with daily updates flowing to your filter. A face-swap tool that relaunches under a fresh domain inherits the same block, without your team ever hearing its new name.

We recommend it. Staff rarely have a work purpose for face-swap or companion services, and staff accounts are the prime target for voice-clone and impersonation scams. Keeping the block uniform also simplifies policy and removes the awkward scenario of harmful tools being reachable from a teacher workstation a student can see or borrow.

Some will try, which is why anonymizer and proxy categories are blocked alongside the AI categories, and why HTTPS traffic is still categorized rather than waved through. No filter is unbeatable, but raising the effort from "type a URL" to "defeat several layers on a monitored device" deters the impulsive misuse that causes most real incidents — and attempts themselves show up in reports.

Same underlying blocklist, different decision. Blocking AI cheating tools is an academic-integrity call with legitimate room for debate about what to allow for learning. This page covers the safety categories — deepfake, voice cloning, companion chat — where the recommendation is simply to block, for every grade, everywhere your policy reaches.

Close these doors before an incident opens them

See exactly which deepfake, voice-cloning and companion-chat services are reachable from your network today, and how three category blocks change that picture district-wide.

Request a Demo View Pricing