Deep web

src: i0.wp.com

deep web , invisible web , or hidden web is part of the World Wide Web that is not indexed by standard web search engines for reasons. The opposite term with a deep web is the surface web, which can be accessed by anyone who uses the Internet. Computer scientist Michael K. Bergman is credited with coining the term deep web in 2001 as a search index term.

The inner web content is hidden behind the HTTP form, and includes many very common uses such as web mail, online banking, and user-paid services, and are protected by paywall, such as on-demand videos, several magazine and newspaper online, and many again.

Content from deep web can be found and accessed by direct URL or IP address, and may require passwords or other security access pass through public website pages.

Video Deep web

Terminology

The first merger of the terms "deep web" and "dark web" appeared in 2009 when the in-depth web search terminology was discussed along with the illegal activity that occurred in the darknet Freenet.

Since then, use in Silk Road media reporting, many people and media outlets, has been using the Deep Web synonymously with dark web or darknet, a comparison that many have rejected as inaccurate and consequently a source of perpetual confusion. Wired reporters Kim Zetter and Andy Greenberg recommend the term is used in different modes. While the inner web refers to any site that is not accessible through traditional search engines, dark web is a part of the deep web that is intentionally hidden and inaccessible via standard browsers and methods.

Maps Deep web

Un-indexed content

Bergman, in an in-depth web paper published in The Journal of Electronic Publishing, states that Jill Ellsworth used the term "Invisible Web" in 1994 to refer to a website not listed with any search engine. Bergman cites a January 1996 article by Frank Garcia:

These are sites that may be reasonably designed, but they do not bother registering them with any of the search engines. So no one can find it! You are hidden. I call that Web invisible.

Another early use of the term Invisible Web is by Bruce Mount and Matthew B. Koll of the Personal Library Software, in the description of the # 1 Deep Web tool found in a December 1996 press release.

The first use of certain terms web in , now generally accepted, occurs in the 2001 Bergman study mentioned earlier.

src: i.ytimg.com

Indexing method

Methods that prevent webpages from indexing by regular search engines can be categorized as one or more of the following:

1. Contextual web : pages with varying content for different access context (eg client IP address range or previous navigation sequence).
2. Dynamic content : dynamic pages returned in response to queries submitted or accessed only through forms, especially if an open domain input element (such as a text field) is used; Such fields are difficult to navigate without domain knowledge.
3. Limited access content : sites that restrict access to their pages by technical means (for example, using the Robot Exclusion Standard or CAPTCHA, or store-free instructions that prohibit search engines from browsing and making cached copies).
4. Non-HTML/text content : text content encoded in a multimedia file (image or video) or a specific file format that is not handled by search engines.
5. Private Web : sites that require sign-up and login (password-protected resources).
6. Script content : pages that can only be accessed through links generated by JavaScript as well as content downloaded dynamically from Web servers via Flash or Ajax solutions.
7. Software : certain content is intentionally hidden from the Internet, accessible only by special software, such as Tor, I2P, or other darknet software. For example, Tor lets users access websites using the.ion server address anonymously, hiding their IP address.
8. Unlinked content : pages not linked by other pages, which may prevent web crawling programs from accessing the content. This content is referred to as a page without backlinks (also known as links). Additionally, search engines do not always detect all backlinks from web pages being searched.
9. Web archives : Web archive services such as Wayback Machine allow users to view archived versions of web pages over time, including websites that become inaccessible, and are not indexed by search engines like Google.

src: 3.bp.blogspot.com

Content type

While it is not always possible to immediately locate certain web server content so that it can be indexed, a site may potentially be accessed indirectly (due to computer vulnerabilities).

To find content on the web, search engines use web crawlers that follow hyperlinks through known virtual port number protocols. This technique is ideal for finding content on the web surface but is often ineffective in searching for deep web content. For example, this crawler does not try to find dynamic pages that are the result of a database query because of the infinite number of possible queries. It has been noted that this can be (partially) resolved by providing links to query results, but this can inadvertently inflame the popularity for deep web members.

DeepPeep, Intute, Deep Web Technologies, Scirus, and Ahmia.fi are some of the search engines that have deep web access. Intute ran out of funds and is now a temporary static archive in July 2011. Scirus retires at the end of January 2013.

Researchers have explored how deep web can be crawled automatically, including content that is only accessible by special software such as Tor. In 2001, Sriram Raghavan and Hector Garcia-Molina (Stanford Computer Science Department, Stanford University) presented an architectural model for the hidden Web crawler that used the main terms provided by the user or collected from the query interface to request Web forms and crawl the deep Web Content. Alexandros Ntoulas, Petros Zerfos, and Junghoo Cho from UCLA create a hidden Web crawler that automatically generates meaningful queries to be excluded from the search form. Some form request languages ​​(eg DEQUEL) have been proposed that, in addition to issuing queries, it also allows the extraction of structured data from the results page. Another effort is DeepPeep, a project from the University of Utah sponsored by the National Science Foundation, which collects hidden web resources (web forms) in different domains based on novel-focused crawler techniques.

Commercial search engines have begun exploring alternative methods for deep web browsing. The Sitemap Protocol (first developed, and introduced by Google in 2005) and mod oai is a mechanism that allows search engines and other interested parties to find in-web resources on a particular web server. Both mechanisms allow web servers to advertise URLs that are accessible to them, allowing the discovery of resources automatically that are not directly connected to the web surface. Google's web plating system calculates submissions for each HTML form and adds generated HTML pages to Google's search engine index. The resulting results generate a thousand queries per second for deep web content. In this system, the pre-submission calculations are performed using three algorithms:

1. selects input values ​​for text search inputs that accept keywords,
2. identifies an input that accepts only values ​​of a certain type (e.g., date), and
3. selects a small number of combinations of entries that generate matching URLs to be included in the Web search index.

In 2008, to facilitate hidden Tor service users in their access and look for the hidden suffix.onion, Aaron Swartz designed Tor2web - a proxy application that can provide access through a common web browser. Using this app, deep web links appear as a series of random letters followed by a.onion TLD.

src: i.ytimg.com

See also

Dark Web Vs Deep Web

• Deep links
• Gopher protocol
• The DARPA Memex Program

src: i.redd.it

References

src: i.ytimg.com

Further reading

Source of the article : Wikipedia