A slide deck cannot tell you whether a filtering database will misclassify your biology curriculum, wave through a brand-new proxy site, or bury your help desk in unblock tickets during the first week of term. Only your own domains, tested against the real dataset, can do that. This page sets out precisely what an evaluation includes, how to design a pilot that produces defensible numbers, and what those numbers should look like before you commit budget.
Most vendor evaluations begin with a form and end three weeks later with a scheduled call. Ours does not have to. The domain checker at /check-domain.php queries the production classification service directly and returns the category assignment, the confidence signal and the supporting context for whatever host you type in. It is the same answer a filtering appliance would receive from the API at query time, which means you can start stress-testing the data during the ten minutes you set aside to read this page.
Use it adversarially. Try the sites your students actually argue about. Try the regional newspaper that another vendor classified as adult content because of a single crime report. Try the university department page that mentions reproductive biology. Try the game-mirror domain that appeared on your network last Tuesday. Twenty minutes of that will tell you more about classification quality than any recorded product tour, and it costs you nothing and commits you to nothing.
Vague trials waste everybody's time. Here is the honest boundary line, so you can plan your pilot around what you will actually have in hand.
Keep the compliance line clear throughout. CIPA imposes a filtering obligation over exactly three things: obscene material, child sexual abuse material, and material harmful to minors. Every other category in the taxonomy — gambling, social media, streaming video, games, weapons, AI chatbots — is local policy. That distinction matters during evaluation because it tells you which classification errors are a compliance problem and which are merely an operational annoyance. Our page on the CIPA-compliant web filter works through the statutory requirement in detail.
Do not take any of these on trust. Each one is falsifiable with a few hours of work, and a vendor worth buying from will tell you exactly how to try.
This assumes one technical lead giving the trial roughly half a day per working day, with short input from a curriculum colleague and a librarian. It is deliberately achievable during term time.
Before touching an API key, agree the failure conditions in writing. A pilot without pre-agreed thresholds always ends in a debate about vibes. Decide now: what accuracy rate on your own domain sample is acceptable? How many wrongly blocked educational sites in a hundred would be too many? How long may a newly registered proxy remain unclassified before that is a problem? How fast must a re-classification request be resolved? Circulate that single page to everybody who will have an opinion later, including the person who signs the purchase order. If the vendor cannot meet a threshold you wrote before you saw any results, that is a clean, defensible no.
Also record your current baseline. If you already run a filter, gather last term's unblock ticket count, the categories those tickets clustered in, and any incidents where something got through. Without a baseline you cannot demonstrate improvement, and a purchase justified by improvement you never measured is hard to defend at renewal.
This is the single highest-value hour of the whole pilot and the step most teams skip. A list built from your actual traffic is the only sample that reflects your students, your curriculum, your region and your language mix. The method is described in full in the next section. Aim for a deduplicated list of somewhere between twenty and fifty thousand distinct hostnames covering at least two full weeks of term-time traffic, including at least one week that is not an exam period.
Write the results to a local table with the hostname, the returned category, the timestamp and the raw response. Keep the raw response: halfway through the trial somebody will ask a question that only the untouched payload can answer. Note your coverage figure immediately — the proportion of your submitted hosts that came back with a classification at all. Coverage and accuracy are different measurements and conflating them is the most common analytical error in filtering pilots. A database can be extremely accurate on the sixty per cent of your traffic it recognises and still be unusable because of the forty per cent it does not.
Draw a stratified random sample from the classified results and have humans judge each one. This is unavoidable manual work and it is where the real answer lives. The sampling design, sample sizes and scoring rules are set out below. Split the reviewing between at least two people so you can measure how often your own reviewers disagree — if two of your staff cannot agree on whether a site is educational, you have learned something important about your policy before you have learned anything about the vendor.
In parallel with general review, run a dedicated pass over legitimate educational material. Ask heads of department for the domains they rely on, and specifically for the ones that have been blocked before. This test has a different acceptance threshold from the general accuracy test, because a false block on a curriculum resource costs a lesson while a false block on an obscure forum costs nobody anything.
Freshness testing has to run in the background for several days to mean anything, so start it early in the second week. At the same time, put each delivery method — API, CSV download, DNS blocklist and the full database download — against your actual enforcement architecture, and time how long each takes to integrate. The comparison table further down maps each option to the situation it suits.
Produce a two-page summary: coverage, sampled accuracy with a stated margin of error, false positive rate on educational domains, AI tool detection rate, update latency, integration effort per delivery method, and the responsiveness of the re-classification path. Compare each figure to the number you wrote down before you started. Resist the urge to renegotiate your own thresholds after seeing the data — that is how districts end up with a filter everybody complains about for four years. Then look at pricing with real numbers in hand rather than an impression.
Curated demo lists flatter every vendor. Your logs do not.
The reason vendor-supplied test lists always look impressive is that they are drawn from the head of the distribution, where every provider on the market performs well. Nobody misclassifies Wikipedia. The differences between filtering databases live in the long tail: regional news sites, hobby forums, small-language content, freshly registered domains, content delivery hosts, single-purpose game mirrors and the sprawling ecosystem of AI wrappers. Your traffic logs contain that tail in exactly the proportions your district experiences it, which is why they are the only honest source for a test list.
Any of these will do, in rough order of preference. Your DNS resolver query log is usually the richest and least intrusive source, since it captures every lookup regardless of protocol and contains hostnames rather than full URLs. Firewall or proxy connection logs are the next best option and have the advantage of recording which requests were actually completed. If you run a cloud filtering product already, its own reporting export will do, though be aware it is biased: it can only show you domains its own classifier already recognised, so you will under-sample precisely the unknown tail you most want to test. Chromebook fleets often expose a per-device browsing history export through the management console, which is useful for off-campus behaviour but needs careful handling under your privacy policy.
Start by stripping everything below the registrable domain unless subdomain-level classification matters to your policy, then deduplicate. A fortnight of traffic from a mid-sized district typically collapses from tens of millions of queries to somewhere between thirty and eighty thousand distinct registrable domains, which is a very workable test corpus. Remove your own internal namespace, your management infrastructure, telemetry endpoints and the update servers of every product you run — these are noise that no filtering database is asked to classify meaningfully and they will distort your coverage figure upward if you leave them in.
Now attach a weight to each remaining domain: the number of distinct devices that touched it, not the number of requests. Request counts are dominated by background polling and a single video stream can generate more hits than an entire year group's research session. Device counts approximate how many students would be affected if the classification were wrong, which is exactly the quantity you care about. Keep this weight column; it drives the stratified sampling in the accuracy step.
Finally, add material your logs will not contain. Include a set of domains you know should be blocked under the three CIPA content classes, obtained from a threat feed or a previous filter's block log rather than by browsing for them. Include fifteen or twenty freshly registered domains from a zone file feed. Include the AI tools your staff have mentioned in the last month. Include at least a dozen genuinely awkward edge cases: sexual health charities, harm reduction services, LGBTQ+ support organisations, historical archives containing violent imagery, literature sites hosting texts with explicit passages, and news outlets covering conflict. Those edge cases will not change your headline accuracy figure, but they will tell you more about a vendor's editorial judgement than the other forty thousand domains combined.
Handle the extract properly. A domain list derived from student traffic is personal data in most jurisdictions until it is properly aggregated. Strip user and device identifiers before the list leaves your environment, keep only the deduplicated hostname and the device count, document the lawful basis for the processing, and delete the working files when the pilot ends. Any vendor that encourages you to be casual about this is telling you something about how they will handle your data later.
Accuracy claims are meaningless without a stated sample, a stated method and a stated margin of error. Here is how to produce all three in a couple of afternoons.
Nobody is going to hand-check forty thousand domains, and you do not need to. A well-drawn random sample of a few hundred gives you a headline accuracy figure with a margin of error small enough to make a purchasing decision. The arithmetic is standard: a simple random sample of 384 items yields roughly a five percentage point margin of error at ninety-five per cent confidence, and 600 items tightens that to about four points. Reviewing three thousand domains would take weeks and only narrow the interval to under two points, which will not change your decision. Spend the saved effort on the targeted tests instead, where the findings are qualitative and far more revealing.
A pure random sample from a long-tailed corpus is dominated by obscure domains that one device touched once. That is not what your users experience. Split the corpus into three strata by device count — a high band covering domains touched by many devices, a middle band, and the long tail — and sample independently within each. Roughly one hundred and fifty from each band works well. Score each band separately, then combine them into an overall figure weighted by how much real traffic each band represents. The result is a single number that genuinely reflects what a student would encounter, and it also exposes the common pattern where a database is excellent on popular sites and mediocre in the tail.
Have the reviewer open each sampled domain and decide, using your own written policy, which category it should belong to and whether it should be blocked for the age group in question. Crucially, the reviewer should record their judgement before revealing the vendor's classification. If the vendor's answer is visible first, agreement rates rise by a wide margin through simple anchoring, and your measurement is worthless. A spreadsheet with a hidden column, revealed only after the whole batch is scored, is entirely sufficient.
Reduce every scored domain to one of four cells. They are not equally important, and collapsing them into a single accuracy percentage hides the distinction that actually matters to a school.
Should be blocked, and the database says block. This is the case that satisfies the technology protection measure. Track it separately for the three CIPA content classes, because a miss here is a compliance failure rather than an inconvenience.
Should be allowed, but the database says block. Every one of these is a lost lesson, a help desk ticket and, at scale, an E-Rate audit finding. Weight these heavily and record which category caused the misfire.
Should be blocked, but the database says allow. Split this figure into two: misses inside the three CIPA classes, which are serious, and misses in discretionary categories such as gaming, which are policy noise.
Should be allowed, and the database allows it. The largest cell by far, which is exactly why a raw accuracy percentage looks flattering for every vendor and tells you almost nothing on its own.
From those four cells, produce four figures and keep them separate in your report. Coverage is the share of submitted domains that received any classification. Precision on blocks is true positives divided by all blocks, which answers the question "when it blocks something, how often is it right?" Recall on the CIPA classes is true positives divided by everything that genuinely belonged in those three classes, which answers "how much of what must be blocked did it catch?" And the educational false positive rate is over-blocks measured only across the curriculum domain set. A vendor that quotes a single number without telling you which of these four it is has not measured anything you can use.
Have both reviewers independently score the same fifty domains and compare. In practice, human reviewers in the same district disagree on somewhere between five and fifteen per cent of borderline cases, and that figure sets a hard ceiling on the accuracy you can meaningfully measure. If your reviewers disagree eight per cent of the time, the difference between a vendor scoring ninety-four and one scoring ninety-six is inside your own noise floor and should not decide a procurement. This is also the moment districts usually discover their acceptable use policy is ambiguous, which is a genuinely useful thing to learn from a filtering trial.
Under-blocking gets a district a phone call. Over-blocking gets it an audit finding, a curriculum complaint and, in the worst case, a First Amendment argument. Test it deliberately.
CIPA requires a technology protection measure against three specific content classes and it also anticipates that filtering will be disabled for adults conducting bona fide research. A district that blocks far beyond the statutory scope, with no reliable way to unblock and no record of how decisions are made, is not demonstrating better compliance — it is demonstrating an unmanaged filter. Auditors and complainants both notice.
Every over-block becomes a teacher improvising mid-lesson, a ticket in your queue, and a small withdrawal from the credibility account your team draws on when you need staff to accept a security control. Districts that measure it usually find the same handful of categories generate most of the pain: health, art history, literature, current affairs and anything using the word "gaming" in an academic sense.
Assemble one hundred to two hundred domains that must never be blocked, drawn from department reading lists, library databases, exam board sites, university outreach pages and the local public library catalogue. Run them through the API and inspect every single result — no sampling here, the list is small enough to review completely. Anything blocked is a finding worth a conversation.
Certain content sits genuinely close to a boundary, and the way a database handles those cases reveals its editorial care more clearly than any aggregate statistic. Sexual health and puberty education resources are routinely swept into adult categories by classifiers relying on keyword density. Harm reduction and addiction support services get caught by drug filters despite being exactly the material a struggling student needs. Holocaust archives, war photography collections and human rights documentation contain violent imagery in an unambiguously educational frame. Classic literature hosted on free-text archives includes explicit passages that are on the syllabus. LGBTQ+ support organisations have historically been mis-categorised in ways that produced real legal difficulty for districts. Build every one of these into your test list and look at the answers individually.
Do not simply count them. For each over-block, ask three follow-up questions. Was the domain in a category you would expect, meaning the classification was defensible but your policy is too broad? Or was it in a plainly wrong category, meaning the data is at fault? Second, does the vendor offer a category granular enough to let you allow the educational subset without opening the whole parent category? Third, how long does a correction take once you report it? A database that occasionally misfires but corrects within a day is far more workable than one that is marginally more accurate and treats every dispute as a support ticket to be closed.
A useful acceptance threshold. Many districts settle on this: zero tolerance for over-blocks on the curated curriculum list, and no more than one to two per cent over-blocking in the general sample. If a vendor cannot clear the curriculum list cleanly after one round of corrections, the operational cost across a school year will outweigh whatever the aggregate accuracy figure claims.
16,328+ tracked AI tools sounds impressive. The number that matters is how many of the ones your students are using this month are in there.
AI services are the hardest category in school filtering right now, for a structural reason. A new wrapper around an existing model can be registered, marketed on a social platform and adopted across a year group inside a week. There is no established host reputation, no inbound link history and often no meaningful text on the landing page for a classifier to work with. Traditional category databases update on a rhythm designed for a web that moved much more slowly. This is precisely why the AI tools list is maintained as its own tracked set rather than left to emerge from general crawling.
Recency test. Ask three teachers and, more usefully, a handful of sixth formers to name every AI tool they have seen used in the last month. You will get a list of perhaps twenty-five names, several of which no adult in the building has heard of. Check each one. The headline count is irrelevant if the tools actually circulating in your corridors are missing.
Granularity test. Blocking all AI is rarely the policy anyone actually wants. Districts typically want the approved classroom assistant permitted, essay-generation services blocked, image generators restricted by year group, and companion or roleplay chatbots blocked outright on safeguarding grounds. Ask whether the data supports those distinctions or whether it offers one undifferentiated bucket. A single flag forces you into a policy your staff will spend the year fighting.
Evasion test. Take five well-known AI services and look for their alternative access paths — regional mirror domains, API subdomains, embedded widgets on third-party sites, and the mobile app endpoints that bypass the browser entirely. This is where most AI blocking quietly fails in practice, and it is worth knowing before deployment rather than after a safeguarding incident.
Stability test. Re-run the same set of AI domains at the start and end of your two-week trial. Classifications should be stable for established services and should have grown for the newer ones. Wholesale changes to established entries between runs suggest an unstable pipeline; no change at all across two weeks in the fastest-moving category on the internet suggests the list is not really being maintained.
Daily updates is a claim about process. Here is how to convert it into a measured number in the time your pilot runs.
The domains that cause a district the most trouble are almost always the youngest. A proxy service that appeared on Monday and spread through a year group by Wednesday is a genuinely hard problem, and it is the problem where the difference between a database refreshed daily and one refreshed weekly becomes visible to students long before it becomes visible to IT. Fortunately, freshness is one of the few filtering claims you can measure precisely rather than argue about.
Obtain a feed of newly registered domains — several registry and research sources publish daily lists, and many are freely available to educational institutions. Take a sample of two hundred domains registered within the previous forty-eight hours. Query each one through the API on day one of your test and record the response: classified, unclassified, or classified into a generic placeholder. Then re-query the identical list every twenty-four hours for the remainder of the pilot, keeping every daily snapshot.
What you are producing is a curve: the proportion of a cohort of brand-new domains that has received a classification, plotted against days since registration. That curve is the single most informative freshness measurement available to you, and you can generate it without any cooperation from the vendor. Expect a rapid rise over the first several days as sites become reachable and acquire enough content to classify, then a slow tail of domains that were parked, never developed or deliberately hidden. A meaningful share of registered domains never serve real content at all, so do not treat anything short of full classification as a failure.
Register two or three domains yourself for a few pounds each. Put ordinary, unambiguous content on one — a small blog about a hobby. Put something that clearly falls into a discretionary blocked category on another. Leave the third parked and empty. Then watch how each is handled and how long it takes. This is a small experiment that tells you a great deal: whether new sites default to blocked, allowed or unknown, how quickly real content is picked up, and whether an empty parked domain is silently swept into a suspicious bucket. Each of those behaviours has direct consequences on your network, and none of them appears in a product datasheet.
Freshness is not only about new domains; it is also about how fast the data responds to you. Submit three genuine re-classification requests during the pilot — ideally real disagreements you found during human review rather than manufactured ones. Record when you submitted, when you received an acknowledgement, when the change appeared in the API, and whether anyone explained the reasoning. Over a four-year contract, that loop will matter more to your daily workload than a percentage point of aggregate accuracy.
All four delivery methods carry the same underlying classifications. They differ entirely in operational shape, and a pilot is the right time to find out which one fits your stack.
| Delivery method | Best suited to | Freshness | Works offline | Integration effort | Main limitation |
|---|---|---|---|---|---|
| Real-time API | Proxies, firewalls and custom policy engines that can call out at request time; also the natural choice for building your own dashboards | Always current | No | Low — a single HTTP call and a cache | Adds a network dependency on the request path; needs sensible caching and a defined fail-open or fail-closed behaviour |
| CSV download | Teams importing into an existing filtering appliance, an SIEM, or a data warehouse for reporting and analysis | Daily refresh | Yes | Low — scheduled fetch and import | Point-in-time snapshot; anything classified between imports is invisible until the next run |
| DNS blocklist | Districts enforcing at the resolver, including Pi-hole style deployments, RPZ zones on BIND, and managed DNS filtering services | Daily refresh | Yes | Lowest — often a single zone file load | Domain-level only, so no path granularity; encrypted DNS on unmanaged devices can bypass it entirely |
| Full database download | Large districts, consortia and vendors embedding classification in their own product, or anywhere data residency rules prevent outbound queries | Daily refresh | Yes | Highest — storage, indexing and update pipeline | Substantial infrastructure to host 120M+ records and apply daily deltas; provisioned at contract stage rather than trial |
Take one representative enforcement point in your environment — not a lab VM, something that resembles what you will run — and integrate at least two methods against it. Time the work honestly, including the parts nobody counts: reading the documentation, handling authentication, deciding what happens when a lookup fails, and explaining the design to the colleague who will maintain it after you. If the API integration takes an afternoon and the full download takes three weeks of engineering, that difference belongs in your evaluation report alongside the accuracy figures, because it is a real cost.
Test the failure modes explicitly. Block outbound access to the API from your test host and observe what your enforcement point does. Does it fail open, letting everything through, or fail closed, breaking the internet for a school? Neither answer is universally right, but you must know which one you have chosen and be able to justify it. Similarly, deliberately import a truncated CSV and confirm your pipeline rejects it rather than silently deploying a half-empty block list. These are the incidents that actually happen in production, and a pilot is a cheap place to discover them. More detail on the underlying dataset and its structure is on the database page.
Including us. The quality of the answers, and the willingness to give specific ones, is itself a data point.
Pay attention to how questions are answered, not just what is said. A vendor who tells you plainly which categories are weaker, which parts of the world their coverage is thinner in, and which use cases they are a poor fit for is giving you information you can plan around. A vendor for whom everything is a strength has simply not told you where the problems will be, and you will find them yourself in October.
A filtering decision made entirely inside the IT office is a filtering decision that gets relitigated all year.
Technical teams are well placed to measure coverage, latency and integration effort. They are poorly placed to decide whether a particular history resource is appropriate for a Year 9 class, and they should not be asked to. Every district that has run a filtering pilot well has involved the people who teach and the people who run the library, and has done it early enough that their input shaped the decision rather than arriving as a complaint afterwards.
Nobody will complete an open-ended survey about internet filtering. What works is a short, specific request. Ask each head of department for the twenty domains their subject cannot teach without, plus any that have caused trouble in the past. That is a fifteen-minute task with an obvious payoff, and the aggregated result is the curriculum test list described earlier. Ask the librarian separately: library staff usually have the sharpest instincts about over-blocking, deal with the widest range of student research requests, and often maintain informal records of every resource a filter has ever obstructed.
When teaching staff review sampled domains, hand them a one-page rubric rather than asking whether a site "seems appropriate". A workable rubric asks three questions in order. First: does this fall into one of the three content classes CIPA requires us to block? If yes, block, no further discussion. Second: does it breach a specific line in our acceptable use policy? If yes, block and cite the clause. Third: is it merely a distraction or off-task? If so, note it — but recognise that this is a classroom management question, and filtering it network-wide is a policy choice with real costs. This structure keeps the trial focused and stops it drifting into a general debate about screen time.
When a teacher's flagged domain gets corrected during the pilot, tell them, and tell them how long it took. Two or three visible corrections do more for staff confidence in a new filter than any amount of launch communication, and they establish from the outset that reporting a problem is worth the effort. If your future unblock process is going to depend on staff bothering to report, that habit is best formed during the trial.
Read these against the thresholds you wrote down on day one, not against a vendor's summary slide.
The failure mode here is not technical. It is deploying a correct block list to eight thousand devices during first period.
Deploy the classification data across the whole district with enforcement disabled, logging what would have been blocked without actually blocking it. Two weeks of this produces a precise, district-wide prediction of your block volume, the categories driving it and the schools most affected — all without a single interrupted lesson. It also surfaces the site that your entire maths department depends on but that never appeared in any test list, which is a discovery you would much rather make in a log file than in a phone call.
Turn on blocking for the three statutory content classes first and let that settle for a week. This is the part with a legal obligation attached, it is the least controversial with staff, and it generates the fewest tickets. Layer discretionary policy — social media, games, streaming, AI tools — on afterwards, one category group at a time, so that when complaints arrive you know precisely which change caused them.
Choose a school with a supportive leadership team and a technically confident site contact, and run full enforcement there for a fortnight before extending. A single school is small enough that every problem gets attention and large enough to be realistic. Your ticket rate per hundred students from that school is the number that lets you staff the wider rollout properly instead of guessing.
Staff must know, on day one, how to request that a site be unblocked and roughly how long it will take. Put it on a single page, link it from the block page itself, and commit to a turnaround you can genuinely meet. An unblock process that is fast and visible converts filtering from something done to teachers into something they participate in, and it dramatically reduces the pressure to weaken policy wholesale.
Repeat a small version of your accuracy sample once a term — a hundred domains takes an afternoon. Watch the categories your unblock tickets cluster in. Re-run the curriculum list at the start of each academic year, since resources change. This ongoing record is also the most useful thing you can hand an auditor: not a vendor's marketing claim, but your own evidence that the technology protection measure is in place, is working, and is actively managed.
Tell us roughly how many students you support, what you enforce with today, and how large a domain sample you want to test. We will size an evaluation key to your list and answer classification disputes directly during the pilot — and if the fit is poor, we will tell you that too.