Much of the web's structured data sits in databases that can only be reached by filling in a search form. Crawlers that want to surface this "deep web" first have to decide which of the millions of forms they meet are real database gateways, as opposed to login boxes, newsletter sign-ups, or a site's generic search bar. Existing detectors handle this as a one-shot binary classification over word tokens pulled from HTML attributes. That worked on the hand-written HTML of the mid-2000s. Modern pages often carry minified identifiers, generated class names, and labels in many languages. When a form's attributes read like f_x3k or inp-7a2, a classifier still produces a confident score. It cannot report that it had nothing to go on.
This note proposes a detector that reads each part of a form separately, using character n-grams (short overlapping letter sequences, which survive obfuscation and multilinguality far better than words). Each part's verdict is expressed as a belief function, a probability-like assignment that can put weight on "one of these two classes" or on "I don't know" instead of forcing a split across single classes. The parts are combined with conflict-aware fusion. The final accept threshold is set relative to the other forms on the same site, in the way a document binarizer sets a pixel's threshold relative to its neighborhood. The practical payoff is a three-way decision. The crawler accepts, rejects, or spends one probe query to check the form, and it can tell apart forms that are uninformative (worth probing) from forms that are truly ambiguous between site search and database search (where probing is less informative than reading more page context).
Core Claim
Separating ignorance from confusion at the level of individual HTML evidence sources, and letting that ignorance control how much a site's local score distribution overrides a global threshold, should give higher search-interface precision at a fixed probe budget than calibrated single-score classifiers. The largest gains should appear on obfuscated or non-English forms and in sparse domains where crawlers struggle to bootstrap. The claim is falsifiable in a specific way. If a calibrated character n-gram classifier combined with per-site score normalization matches the full system on the probe-budget curve, the evidential component adds nothing, and we will report that.
Research Program
The first task is to build the evidence sources. Each candidate form is split into separate strings: the action URL, the method, input names and ids, placeholder and label text, submit-button text, and a window of visible text around the form. Following the evidence that character 4- and 5-grams beat word features for noisy, closely related strings (Tübingen-Oslo at VarDial 2018 shows this for language varieties; MOROCO shows string kernels beating character CNNs even after named entities are stripped), each source gets a hashed character 3–5-gram representation.
The frame of hypotheses is kept small on purpose: database search, site-wide search, login/account, and subscribe/contact. This gives only 16 subsets, which avoids the power-set blow-up that confines most belief-function work to toy problems. The confusion subset that matters most is {database search, site search}, because that is the known hard case in form filtering.
The second task is to turn each source's output into a belief function. Each source gets a small evidential head trained with the Dirichlet-evidence approach of Flexible Visual Recognition by Evidential Modeling of Confusion and Ignorance. In that method, low total evidence becomes weight on the whole frame (ignorance), and evidence split across classes becomes weight on class subsets (confusion). That paper reports misclassification-detection AUROC of 90.0 against 57.3 for softmax on CIFAR-100. Here that result only motivates the design; it is not a web-form result. A minified id string should then yield mostly ignorance, not a spurious vote.
The sources are combined with Dempster's rule after the parameter-free conflict discounting of ECoLaF. In ECoLaF, a source whose belief sits far from the others is down-weighted. This is the case where an action URL says "search" but the labels say "email." We reuse ECoLaF's mechanism unchanged and apply it to HTML fields instead of image sensors.
The third task is the local threshold. PatchRefineNet shows that one global cutoff on a score map is systematically too optimistic in some regions and too pessimistic in others. Feng's two-window binarization handles uneven illumination by comparing each pixel against local statistics. Our analogue of a region is a site: its templating conventions shift the scores of all its forms together. Let $b_f$ be form $f$'s combined belief-derived score for database search, computed with the standard pignistic transform. Let $\mu_s,\sigma_s$ be the mean and standard deviation of that score over the other forms crawled on site $s$. Let $\bar{u}_s$ be the average ignorance mass on that site. The accept threshold is
$$\tau_f = \bar{u}_s\,\tau_{g} + (1-\bar{u}_s)\,(\mu_s + k\,\sigma_s),$$
where $\tau_g$ is a global threshold tuned on validation data and $k$ is a scale parameter, also tuned on validation. The weighting means that when a site's forms are mostly unreadable, its local statistics are not trusted and the detector falls back to the global rule. The known weakness of local thresholding is sensitivity to window size. Here the "window" is set by site structure rather than by a free parameter, which is supported by the MetaQuerier finding that 94% of web databases are within three hops of the root.
The fourth task is the decision policy. A form whose ignorance exceeds a threshold is sent to a probe: one benign query is submitted, and the crawler checks whether a result listing appears. A form with high confusion but low ignorance is instead re-scored with wider page context. Everything else is accepted or rejected directly. The probe threshold is swept to trace precision against probe budget. That curve is the main evaluation object, because it measures what a crawler operator actually pays.
Baselines
The naive reference is a word-token bag-of-attributes logistic classifier over the concatenated form HTML with a fixed 0.5 cutoff, with probes spent on the forms closest to the cutoff. This shows how much of any gain comes simply from spending probes.
The strongest published detector is the Improved Random Forest of Feature Weighting Random Forest for Detection of Hidden Web Search Interfaces. It uses Chi-square-weighted feature subsampling over form attributes. On the UIUC Web Integration Repository domains (Books, Movies, Airfares, Jobs) combined with a Search Engine Guide crawl, it reports 93.88% precision, against 91.84% for a plain random forest and 85.71% for an SVM. We will reimplement it with its reported optimal feature-selection rate of 0.5 and train it on exactly our splits. Beating it means exceeding its precision at matched recall on the original-style data, and exceeding it by a clear margin on the obfuscated and multilingual splits. We also add the searchable-form classifier stage of ACHE as a second published competitor.
The strongest simple competitor is not from the literature, and we treat it as the real bar. It is a single calibrated character 3–5-gram classifier over all form strings together, temperature-scaled, with per-site z-score normalization of its scores and probes spent on the highest-entropy forms. It has the same features and the same site-local idea, but no belief functions.
Three ablations isolate the proposal's ingredients. The first removes local thresholding and uses $\tau_g$ only. The second replaces evidential fusion with averaged per-source softmax. The third keeps the fusion but routes probes on total uncertainty instead of on ignorance alone. All systems get the same training data, the same hashing dimension, the same hyperparameter search budget of 50 trials on a shared validation split, and the same probe budgets.
Why not just…?
Why not just use one calibrated character n-gram classifier and probe the uncertain forms? This is the most serious threat. Softmax entropy peaks both when a form is unreadable and when it truly sits between site search and database search. The evidential work cited above suggests that softmax separates these two poorly. If that holds here, entropy-routed probes will be partly wasted on confusion cases, where a single probe often cannot tell a site search from a database search, because both return listings. The comparison against the strongest simple competitor, run on the probe-budget curve, settles this directly. If the two curves overlap, we drop the evidential claim.
Why not just normalize scores per site and skip belief functions? Per-site z-scoring is the local-thresholding idea without the ignorance weighting. It should fail on sites with only one or two forms, and on sites where every form is obfuscated. In both cases the local mean and spread are noise, so the threshold becomes arbitrary. Ablation (a) against the z-score baseline, stratified by forms per site and by obfuscation rate, shows whether the ignorance weighting matters.
Why not just ask a large pretrained language model to read the HTML? This is a credible competitor on English, readable forms. The file contains no web-form result for such models. The character-level literature shows transformers discarding surface cues that separate closely related strings (Greek Proverb Atlas). The per-form cost is also orders of magnitude higher at crawl scale. We will include a zero-shot pretrained-model classifier on a 2,000-form subsample as a reference point. If it dominates on the obfuscated split, the right conclusion is to feed its output in as one more evidence source, not to claim our system is superior.
Why not just probe every form? Probing is the ground truth, but it costs requests. Submitting to login, contact, and subscription forms is also impolite and sometimes harmful. The probe-budget curve makes this explicit. The "probe everything" endpoint appears on it, and the question is how close each method gets to that endpoint's precision while probing a small fraction of forms.
Proposed Experiments
The first experiment runs on the UIUC Web Integration Repository domains so we can compare against the published IRFA figure. On these relatively clean forms we expect parity, not a large win, and parity is an acceptable result.
The second experiment is a controlled obfuscation stress test. We apply realistic minifier-style renaming to ids, names, and classes, and translate labels into several languages, at increasing rates. We measure precision at fixed recall and the AUROC with which ignorance mass flags obfuscation-induced errors.
The third experiment uses a freshly crawled, manually labeled sample of contemporary forms from the eight ACHE domains. We verify labels by probing, and report the precision-versus-probe-budget curves for all baselines and ablations.
The decisive checks are three. The full system must beat the calibrated character baseline in probe-curve area on the obfuscated and fresh-crawl splits. Ignorance-routed probes must resolve a larger share of errors than entropy-routed probes at equal budget. The ignorance-weighted local threshold must beat per-site z-scoring on sites with few forms.
As a downstream test, we plug each detector into an ACHE-style crawler on the sparse Movie domain, where ACHE reports that bootstrapping stalls. We measure forms harvested per crawled page. The file reports ACHE's gain only as "up to three times" more forms than fixed-focus crawlers, with no per-domain numbers, so our comparison uses our own reimplementation rather than quoted figures.
References
- Tübingen-Oslo Team at the VarDial 2018 Evaluation Campaign: An Analysis of N-gram Features in Language Variety Identification
- MOROCO: The Moldavian and Romanian Dialectal Corpus
- Flexible Visual Recognition by Evidential Modeling of Confusion and Ignorance
- A Conflict-Guided Evidential Multimodal Fusion for Semantic Segmentation
- PatchRefineNet: Improving Binary Segmentation by Incorporating Signals from Optimal Patch-wise Binarization
- Contrast adaptive binarization of low quality document images
- Toward Large Scale Integration: Building a MetaQuerier over Databases on the Web
- Feature Weighting Random Forest for Detection of Hidden Web Search Interfaces
- An adaptive crawler for locating hiddenwebentry points
- Towards a Greek Proverb Atlas: Computational Spatial Exploration and Attribution of Greek Proverbs