Data Modeling

Mathematical and Computational Data Modeling | Online ISSN 3143-9217
2
Citations
10.2k
Views
45
Articles
Your new experience awaits. Try the new design now and help us make it even better
Switch to the new experience
Figures and Tables
RESEARCH ARTICLE   (Open Access)

BeamNet: A DOM-Based Approach to Personalized Web Content Curation and Real-Time Notification Delivery

Md. Mubdiur Rahman 1*

 

+ Author Affiliations

Data Modeling 2 (1) 1-8 https://doi.org/10.25163/data.2110896

Submitted: 23 May 2021 Revised: 09 July 2021  Published: 20 July 2021 


Abstract

The volume of information generated across the web has grown to a point were locating what actually matters to an individual reader has become, somewhat paradoxically, harder rather than easier. This study presents BeamNet, a mobile-first system designed to let users assemble their own curated web feed and receive push notifications when the content they care about changes. prior approaches to web content extraction have relied variously on Document Object Model (DOM) tree traversal, text-pattern matching, and headless-browser rendering, each carrying its own trade-offs between accuracy and computational cost. BeamNet allows a user to select, from within a mobile application, multiple visible text elements on a target webpage; the Lowest Common Ancestor (LCA) of the selected elements is then computed and stored, together with the source URL, as a reusable extraction pattern. A server-side headless browser subsequently revisits each stored source at fixed intervals, re-applies the LCA pattern to retrieve updated content, and compares the result against previously stored data to detect change, triggering a Firebase Cloud Messaging notification when new content is found. Informal testing across a heterogeneous sample of websites indicated a content-extraction error rate of approximately 2%–5%, arising primarily from cases in which unrelated text elements shared the same LCA as the intended target. the findings suggest that a lightweight, LCA-based extraction strategy can support personalized, user-controlled content curation without requiring site-specific scraping rules, though scalability and extraction precision remain open engineering challenges that warrant further, more rigorously controlled evaluation.

Keywords: personalized web feed; web scraping; DOM-based content extraction; push notification; content curation

1. Introduction

The contemporary web environment presents a distinct cognitive and procedural challenge characterized not by a scarcity of information, but by its overwhelming proliferation. Individuals attempting to monitor heterogeneous online sources—such as news outlets, community discussion forums, and academic bulletin boards—frequently encounter a systemically inefficient workflow. This manual process requires users to repeatedly navigate static web interfaces to identify incremental updates, resulting in cumulative time loss and user fatigue. The study presented herein addresses this persistent operational inefficiency by proposing a streamlined mechanism for content monitoring and aggregation.

Filtering unwanted content from wanted content is, of course, not a new problem, and a reasonably mature body of work already exists around it. Extraction techniques capable of separating meaningful page content from surrounding boilerplate have been proposed and refined for some time now (Uzun, 2020), and these methods form part of the foundation this project builds on. What has changed more recently, arguably, is less the extraction problem itself than the scale at which it must be solved — the web is now unambiguously a big-data environment, and a great deal of contemporary research energy has gone into making machine learning pipelines fast enough to train on that scale (Najafabadi et al., 2015). That emphasis on throughput, however, is not obviously the right one for the problem tackled here. A single user assembling a personal feed from a handful of trusted sources is not training a model on millions of documents; the relevant bottleneck is not raw processing speed but structural generality — the ability to handle many different, unpredictable page layouts without hand-written rules for each one.

That distinction matters because it shapes which family of extraction techniques is actually appropriate. Several exist. Some rely on full DOM-tree parsing; others attempt to avoid that overhead altogether — Wu (2016), for instance, describes a system that curates news content using text-detection heuristics rather than DOM analysis, sidestepping the computational cost of tree traversal but, arguably, trading away some of the structural precision that a DOM-aware approach can offer. Still other work treats the problem as closer to string matching than to tree parsing (Nusret et al., 2017), and separately, headless-browser-based scraping has been explored as a way of handling pages that render content dynamically via JavaScript, which static HTML parsers tend to miss entirely (Gheorghe et al., 2018). Each of these lines of work solves part of the puzzle; none, on its own, was designed with an ordinary user's day-to-day feed-building in mind.

This paper describes BeamNet, an attempt — modest in scope, deliberately so — to put that missing piece in the user's hands. Rather than presupposing a fixed set of source websites or a pre-trained extraction model, BeamNet lets the user themselves indicate, by simply tapping on the elements that matter to them within a mobile interface, what should be tracked and from where. The system then infers a reusable structural pattern from that selection and uses it, going forward, to monitor the source for change. The remainder of this paper sets out that design in more detail: Section 2 describes the methodology and system architecture; Section 3 reports what was observed when the system was exercised against a sample of real websites and discusses what those observations do, and do not, establish; Section 4 concludes and outlines directions the work still needs to go.

2. Methodology

The aim of this section is to describe the BeamNet pipeline in enough procedural detail that another developer or research group could, in principle, reconstruct the system independently — following, as far as a software system reasonably can, the same logic of reproducibility expected of an empirical methods section.

2.1 System overview and design rationale

BeamNet is composed of three cooperating parts: a mobile front-end application through which the user selects content, a server-side scraping and comparison engine, and a notification-delivery layer. The decision to separate these concerns was not incidental — keeping the extraction logic on the server, rather than the device, was chosen so that scraping could continue on a schedule even while the user's app is closed, at the cost of shifting computational load (and, candidly, some financial cost) onto server infrastructure.

2.2 Content selection procedure

The workflow begins, for the user, with an embedded web view that opens to a search engine, allowing them to navigate to whichever website they wish to track — no pre-registration of source sites is required, which was a deliberate departure from the fixed-source-list model common to conventional feed readers. Once on the target page, the user is shown a selectable overlay of visible text elements and is required to choose at least two elements that they consider representative of the content type they want tracked (e.g., headline text from several news items on the same page). This two-element minimum exists, procedurally, to give the subsequent structural-inference step enough signal to generalize from.

2.3 Structural pattern inference (Lowest Common Ancestor extraction)

Once elements are selected, their positions within the page's Document Object Model — the tree representation of the HTML document, in which every element is a node (World Wide Web Consortium DOM specification; see also Uzun, 2020, for a related discussion of DOM-derived extraction features) — are used to compute their Lowest Common Ancestor, that is, the deepest node in the tree from which all selected elements can be reached. This LCA node, together with the relative path from it to each selected element type, is stored as the extraction pattern for that source, and the pattern-URL pair is saved as a single “source” that can then be attached to one or more user-defined “feeds,” each feed grouping together sources the user considers topically related.

2.4 Server-side scraping and change detection

For each active source, a server-side headless browser instance — chosen over a static HTML parser specifically because a meaningful proportion of modern pages render their content client-side via JavaScript, which a non-executing parser would simply miss (Gheorghe et al., 2018) — periodically re-visits the stored URL, reconstructs the DOM, and re-applies the stored LCA pattern to retrieve the current instance of the tracked content. The retrieved content is timestamped and compared against the most recent previously stored version for that source; where a difference is detected, the new content is written to the database and a change event is queued. Full technical parameters governing this step (polling interval, timeout handling, and retry logic) were configured empirically during development and are acknowledged here as an area needing more systematic, published specification in future iterations of this work.

2.5 Notification delivery

When a change event is queued, the system issues a push notification to the relevant user via Firebase Cloud Messaging, alerting them that new content is available in the corresponding feed. Users retain the ability to toggle notifications on or off at the level of an individual feed, or globally through the application settings, which was intended to give them a degree of control that fixed-alert systems typically do not offer.

2.6 Extraction accuracy assessment

To obtain a preliminary indication of extraction reliability, the system was exercised, informally, against a heterogeneous sample of publicly accessible websites (news outlets and notice-board-style pages were disproportionately represented, though the exact sampling frame was not formally pre-registered — a limitation discussed further in Section 3). Extraction outcomes were manually inspected for cases in which text unrelated to the user's original selection was nonetheless returned, on the grounds that it shared the same LCA node as the intended content. It should be stated plainly, in the interest of transparency, that this assessment was exploratory rather than a controlled experiment: no statistical test was applied, no confidence interval was computed, and the sample was not drawn according to a documented, reproducible sampling protocol. 

3. Results and Discussion

3.1 System functionality

In practical use, the BeamNet workflow behaved largely as intended. A user could move from an unfamiliar website to a functioning, notification-enabled feed entry in a handful of steps: navigating to the site, selecting representative elements, and confirming the source (Figure 1). Multiple such sources, once created, could be grouped into a single feed spanning different websites but a shared topic of interest (Figure 2), and the resulting aggregated view presented all tracked content from those sources side by side (Figure 3). This end-to-end flow — unremarkable as it may sound described plainly — is, we would argue, the core contribution of the system: it removes the need for a user, or a developer acting on their behalf, to write source-specific extraction rules for each website they wish to follow.

3.2 Extraction accuracy

The exploratory accuracy assessment described in Section 2.6 produced an error rate of roughly 2% to 5%, varying by website, with most misclassifications traceable to a single recurring cause: unrelated text elements sharing the LCA node identified as the extraction anchor (Table 1). This is, we think, an instructive failure mode rather than merely an unfortunate one — it points fairly directly at the structural assumption underlying the whole approach, namely that content the user cares about and content the user does not care about will, on the pages that matter, tend to sit in structurally distinct parts of the tree. That assumption holds reasonably well on the more rigidly templated sites (news listings, in particular) and rather less well on pages with denser, more irregular markup. We would resist over-claiming here: a 2–5% range across an informally sampled set of sites is a rough signal, not a validated benchmark, and a more rigorous

Table 1: Summary of the exploratory extraction-accuracy assessment, reporting the observed error-rate range and its primary source across a heterogeneous, non-randomized sample of websites. The table reports a single metric, content extraction error rate, observed to fall between 2% and 5% depending on the site tested, alongside the primary cause of misclassification identified during review. No confidence interval or formal statistical test accompanies these figures, consistent with the exploratory, non-randomized nature of the assessment described in Section 2.6. The note beneath the table flags this limitation explicitly so the values are not mistaken for a validated benchmark. This table underlies the accuracy discussion in Section 3.2.

Metric

Observed Range

Primary Error Source

Content extraction error rate

2%–5%

Shared LCA node with unrelated text elements

 

Figure 1. Basic tree structure of the Document Object Model (DOM), illustrating how HTML elements nest into parent-child relationships that BeamNet's Lowest Common Ancestor (LCA) inference relies on. The diagram shows a simplified DOM tree with a root node branching into nested child elements, mirroring how a real webpage's markup is structured beneath the hood. It is used to explain, at a conceptual level, why two selected content elements sharing a common ancestor node can be generalized into a reusable extraction pattern. This figure is purely illustrative of the underlying web-standard concept and does not depict BeamNet's own interface or output. It grounds the technical explanation of the LCA-based extraction method introduced in Section 2.

 

 

 

Figure 2. Flowchart of the BeamNet content-curation algorithm, tracing the sequence of steps from a user's initial element selection to the generation of a reusable extraction rule. The flowchart lays out the algorithm's decision points in order, showing how raw user input (clicking representative elements on a page) is converted into a generalized LCA-based pattern that can later be reapplied to detect new, similar content on the same site. Branches in the flowchart indicate where the algorithm checks for structural consistency before finalizing a pattern. This figure is referenced in the system-design discussion and complements the narrative description of the extraction process given in the text.

Figure 3. Main components of the BeamNet system and the communication pathways connecting them, showing how the client interface, extraction engine, and content-aggregation backend interact. The diagram depicts the system at an architectural level, showing discrete components (such as the source-selection client, the LCA-based extraction module, and the feed-aggregation service) connected by directional arrows indicating data flow between them. It is intended to give readers a top-down view of how user actions on the client eventually produce the aggregated content shown to the user. This figure supports the system-overview discussion and precedes the more granular data-flow diagram in Figure 4.

follow-up study — with a documented sampling frame, inter-rater reliability checks on what counts as an extraction error, and comparison against an established baseline such as the pattern-matching approach benchmarked by Nusret et al. (2017) — would be needed before the figure could be treated as a reliable estimate of real-world performance.

3.3 Comparison with existing approaches

Set against prior extraction strategies, BeamNet's LCA-based method occupies something of a middle ground. It is more general than fully hand-authored, site-specific scraping rules, yet it does not attempt the linguistic sophistication of text-detection-framework approaches such as Wu's (2016), which forgo DOM analysis altogether in favor of language-level heuristics — an approach that may generalize better across languages but, arguably, less well across arbitrary page structures. Nor does BeamNet attempt the speed optimization that has driven much of the pattern-matching literature (Nusret et al., 2017); given that a personal feed involves comparatively few sources refreshed at modest intervals, raw throughput was, frankly, judged less important during design than structural robustness and ease of use for a non-technical end user.

3.4 User-facing workflow

The content-selection interface (Figure 5a–b) and multi-source feed assembly (Figure 5c) were designed around the premise that curation should not require any technical background from the user — no CSS selectors, no XPath, nothing of that sort has to be written or understood. Whether this design goal was actually achieved for a representative range of users is, admittedly, a question this paper cannot answer with confidence, since no formal usability study or user survey was conducted alongside system development; this remains an important gap for subsequent work. Once a source has been selected and confirmed, the user is not limited to tracking it in isolation. The "Add Feed" screen (Figure 5c) lets a user name a topic-based feed and attach multiple sources to it in sequence, so that, for example, several international-news sites can be folded into a single "World News" feed rather than monitored as separate, unrelated entries. This grouping step is what turns BeamNet from a single-source scraper into something closer to a personal aggregator: once two or more sources are attached to a feed, their extracted content is merged and re-sorted into one continuous stream, shown to the user in the aggregated feed view (Figure 5d). That view interleaves headline, source attribution, and a short excerpt for each item, ordered chronologically across sources rather than grouped by site, which was a deliberate choice intended to spare the user the extra step of checking each source individually. Whether that ordering choice holds up well as the number of attached sources grows substantially beyond the two or three shown in Figure 5c is, admittedly, untested; our informal use during development involved feeds of similar size, and larger, more heterogeneous feeds may surface ordering or de-duplication issues that this preliminary design has not yet had to confront.

3.5 Limitations and directions for future work

Several limitations deserve to be stated openly rather than glossed over. First, scalability: extending the current headless-browser-per-source architecture to a large user base would place substantial, and possibly prohibitive, demand on server resources, and lighter-weight headless browsing approaches — stripped of the graphical rendering overhead that conventional browsers carry — would likely need to be explored. Second, the image-curation logic that pairs images with their associated text in card-style views remains imprecise and was not a focus of the present evaluation. Third, and perhaps most consequential for the system's long-term usefulness, is the absence of any content-quality or trustworthiness layer: BeamNet currently curates whatever the user selects, without regard to the reliability of that content. Given the growing literature on automated misinformation detection (Ebadi et al., 2022) and on personalized news recommendation more broadly (Ito et al., 2015), a natural next step would be to layer a fact-checking or stance-detection component — of the kind demonstrated by Bahuleyan and Vechtomova (2017) for topic-independent rumor-stance classification — on top of the existing curation pipeline, so that personalization is not achieved at the expense of accuracy.

 

 

Figure 4. Flow of data through the BeamNet system, tracing a single piece of content from initial source scraping through to notification delivery on the user's device. The diagram follows one item of tracked content step by step: extraction from the source website, pattern matching against the stored LCA rule, storage in the aggregation backend, and final delivery as a notification or feed entry. Arrows indicate the direction of data movement between each processing stage. This figure is more granular than the component-level view in Figure 3 and is intended to clarify the system's runtime behavior rather than its static architecture.

Figure 5. BeamNet source-creation and feed-assembly workflow, shown across four sequential app screens. (a) Content-selection interface (view 1): the embedded in-app web view a user navigates to locate and load a target website before selecting content to track. (b) Content-selection interface (view 2): once the page has loaded, candidate content elements are visually highlighted so the user can tap to select at least two items to track, without needing to understand the underlying HTML structure. (c) “Add Feed” interface: the user names a topic-based feed (here, “World News”) and attaches multiple previously confirmed sources to it, with the option to add further sites or remove existing ones before finalizing. (d) Aggregated feed view: the resulting feed interleaves headlines, source attribution, and short excerpts from all attached sources in a single chronologically ordered stream, rather than grouping content by site. Panels (a)-(b) support the per-source content-selection procedure described in Section 2.2, while panels (c)-(d) illustrate the multi-source feed-assembly and aggregation workflow discussed in Section 3.4.

 

 

4. Conclusion

This paper set out to address a fairly ordinary frustration — the effort of manually re-checking familiar websites for new content — through BeamNet, a system that lets users define, without technical expertise, what to track and where, using Lowest Common Ancestor inference over the DOM to generalize a user's selection into a reusable extraction pattern. Preliminary, exploratory testing suggested the approach is broadly workable, with an extraction error rate in the low single digits, though this figure should be read as indicative rather than conclusive given the informal nature of the assessment. The more pressing open questions, going forward, concern scalability under realistic user loads, more rigorous and reproducible accuracy evaluation, and the integration of content-reliability safeguards such as misinformation and stance detection. Addressed together, these would move BeamNet from a working prototype toward a system genuinely ready for broader deployment.

References


Bahuleyan, H., & Vechtomova, O. (2017). UWaterloo at SemEval-2017 Task 8: Detecting stance towards rumours with topic-independent features. https://github.com/HareeshBahuleyan/

Bulus, H. N., Uzun, E., & Doruk, A. (2017). Comparison of string matching algorithms in Web documents. In Proceedings of the International Scientific Conference (UNITECH) (Vol. 2, pp. 279–282).

Chakrabarti, S., Punera, K., & Subramanyam, M. (2002). Accelerated focused crawling through online relevance feedback. In Proceedings of the 11th International Conference on World Wide Web (WWW) (pp. 148–159).

Chakrabarti, S., van den Berg, M., & Dom, B. (1999). Focused crawling: A new approach to topic-specific Web resource discovery. Computer Networks, 31(11–16), 1623–1640.

Diligenti, M., Coetzee, F. M., Lawrence, S., Giles, C. L., & Gori, M. (2000). Focused crawling using context graphs. In Proceedings of the 26th International Conference on Very Large Data Bases (VLDB) (pp. 527–534).

Dong, H., & Hussain, F. K. (2013). SOF: A semi-supervised ontology-learning-based focused crawler. Concurrency and Computation: Practice and Experience, 25(12), 1755–1770.

Ebadi, N., Jozani, M., Choo, K. K. R., & Rad, P. (2022). A memory network information retrieval model for identification of news misinformation. IEEE Transactions on Big Data, 8(5), 1358–1370. https://doi.org/10.1109/TBDATA.2020.3048961

Ferrara, E., De Meo, P., Fiumara, G., & Baumgartner, R. (2014). Web data extraction, applications and techniques: A survey. Knowledge-Based Systems, 70, 301–323. https://doi.org/10.1016/j.knosys.2014.07.007

Gheorghe, M., Mihai, F.-C., & Dârdala, M. (2018). Modern techniques of web scraping for data scientists. Revista Româna de Interac?iune Om-Calculator, 11(1), 45–58.

Grubbs, F. E. (1969). Procedures for detecting outlying observations in samples. Technometrics, 11(1), 1–21.

Ito, T., Kim, Y., & Fukuta, N. (2015). The design and implementation of a personalized news recommendation system. IEEE Computer Society.

Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015). Deep learning applications and challenges in big data analytics. Journal of Big Data, 2(1), Article 1.

Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015). Deep learning applications and challenges in big data analytics. Journal of Big Data, 2(1), 1.

Nusret, H., Namik, B., Üniversitesi, K., Onyedi, B., & Üniversitesi, E. (2017). Comparison of string matching algorithms in web documents. ResearchGate. https://www.researchgate.net/publication/321228533

Pinkerton, B. (1994). Finding what people want: Experiences with the WebCrawler. In Proceedings of the 2nd International World Wide Web Conference (Vol. 94, pp. 17–20).

Salem, H., & Mazzara, M. (2020). Pattern matching-based scraping of news websites. Journal of Physics: Conference Series, 1694(1), Article 012011. https://doi.org/10.1088/1742-6596/1694/1/012011

Shchekotykhin, K., Jannach, D., & Friedrich, G. (2010). XCrawl: A high-recall crawling method for Web mining. Knowledge and Information Systems, 25(2), 303–326.

Uçar, E., Uzun, E., & Tüfekci, P. (2017). A novel algorithm for extracting the user reviews from Web pages. Journal of Information Science, 43(5), 696–712.

Uzun, E. (2020). A novel web scraping approach using the additional information obtained from web pages. IEEE Access, 8, 61726–61740. https://doi.org/10.1109/ACCESS.2020.2984503

Uzun, E., Agun, H. V., & Yerlikaya, T. (2013). A hybrid approach for extracting informative content from Web pages. Information Processing & Management, 49(4), 928–944.

Uzun, E., Bulus, H. N., Doruk, A., & Özhan, E. (2017). Evaluation of Hap, AngleSharp and HtmlDocument in Web content extraction. In Proceedings of the International Scientific Conference (UNITECH) (Vol. 2, pp. 275–278).

Uzun, E., Güner, E. S., Kiliçaslan, Y., Yerlikaya, T., & Agun, H. V. (2014). An effective and efficient Web content extractor for optimizing the crawling process. Software: Practice and Experience, 44(10), 1181–1199.

Wu, Y.-C. (2016). Language independent web news extraction system based on text detection framework. Information Sciences, 342, 132–149.

Wu, Y.-C. (2016). Language independent Web news extraction system based on text detection framework. Information Sciences, 342, 132–149.


Article metrics
View details
0
Downloads
0
Citations
6
Views

View Dimensions


View Plumx


View Altmetric



0
Save
0
Citation
6
View
0
Share