What Is the Deep Web and How Does It Work? [2024 Update]

From the news stories and articles you see, it’d be easy to assume that the deep web is a shady place filled with criminals, hackers, and all kinds of ne’er-do-wells. But, as with most things, the truth is far more complex and far less sensational than that.

In short, the deep web is no more illicit than the “surface web” or the bits of the internet that you find through Google. In most cases, those scary stories you’ve heard about hacker hideouts and drug smugglers are all referring to the dark web — not the deep web.

Confused? That’s understandable. Hold on tight and join me as we dive into what the deep web actually is and why it’s unfairly received a bad reputation.

The Different Layers of the Web

There are actually 3 distinct layers that make up the internet — the surface web, the deep web, and the dark web. Understanding these 3 layers is key to truly grasping how the internet works.

What Is the Surface Web?

The surface web is the portion of the internet that's accessible to anyone through standard web browsers and search engines. When you use Google to find a website, browse social media, or shop online: that's all on the surface web. This visible layer only makes up about 4% of the entire internet.

I found this from searching "Disney Plus" on Google

Websites on the surface web are “indexed” — that is, they’ve been discovered and cataloged by Google, Bing, or any other surface search engine. For example, to find this page, you might have searched for something related to the deep web on Google. You found us because this page is indexed.

For a page to be indexed, it needs to have a valid link. Most websites do this in two ways. They build what’s called a sitemap, which exists to help search engines automatically navigate the site. Then, for human users, they add new pages to their overall site structure.

There are ways for website owners to keep parts or all of their site from being indexed and shown in search results. They do this by using a file called “robots.txt”, which gives instructions to web crawlers — automated bots that search engines use to catalog and index web pages.

For example, a website could use robots.txt to block crawlers from their login portals, sites under development, or sections with sensitive data. Properly configured robots.txt directives tell crawlers: "Don't go here, don't index this."

What Is the Deep Web?

The deep web comprises pages that aren’t indexed by search engines. The deep web (or deep net) is larger than the surface web you're used to — experts estimate it makes up a staggering 95% or more of the internet.

The vast majority of the deep web consists of pages that you need to log in to, like your email, online banking, streaming services, and more. Your private account pages exist on the deep web.

This page is only accessible through your Disney Plus account

Even public websites utilize the deep web. For instance, an e-commerce site may have a small portion indexed for product listings but a much larger database inventory that’s hidden on the deep web.

While the deep web may sound ominous, the majority of it is simply data and content not meant for public access. It's largely benign, enabling everything from confidential records to password-protected accounts. However, there are also legally dubious areas that host illicit content. So, you should still be careful when diving below the surface web.

What Is the Dark Web?

The dark web describes any page that’s unindexed by search engines and requires specialized software (like Tor) to access. Tor (The Onion Router) is a free network that routes internet traffic through many encrypted relays to anonymize your browsing. This allows you to connect to hidden dark web services that use special “.onion” domains, which don’t work with regular browsers.

Despite its notoriety, the dark web has legitimate uses. Journalists and whistleblowers rely on its anonymity to transmit data and communicate securely away from prying eyes. Political activists use it to organize and spread information freely in oppressive regimes. Even law enforcement and intelligence agencies leverage the dark web.

Facebook and the BBC also have mirror websites

However, a large portion of the dark web is used for illegal purposes that are best avoided. One of the most infamous dark web examples was the Silk Road, which was a market for buying and selling illegal drugs, weapons, hacking tools, pirated content, and more. It was shut down in 2013, and its creator, co-owners, administrators, and some of its sellers were hunted down and arrested. While the dark web provides some anonymity, it still carries major risks if misused.

How Does the Deep Web Work?

To understand how the deep web works, let’s first look at how search engines work. Search tools use crawlers to regularly analyze new or updated web pages. This information is then “indexed” — it’s stored in the search engine’s index that it uses to show you search results.

There are large swaths of data that crawlers can't access and index for a variety of reasons, like:

• The website owner has used a robots.txt file to request crawlers not to index specific pages.
• The page is password-protected or uses CAPTCHA tests.
• The page blocks automated bots.
• The page isn’t linked to by a sitemap or other crawlable link (at the time of writing, crawlers can’t normally use search boxes).

The deep web also covers content that only exists for a specific user. Social media is a perfect illustration — while Facebook's public pages are indexed, your personalized News Feed or private messages with friends are not. These pages get stitched together dynamically every time you log in, combining unique data sets.

This divide between public and private/personalized internet content is crucial for security and functionality reasons. Just imagine if your entire Facebook profile, messages, payment histories, and other sensitive data were openly crawlable, and could be found and read by anyone.

What Is the Deep Web Used For?

Here are just a few examples:

• Online banking portals. These exist behind login credentials that crawlers can’t access, making personal banking data uncrawlable.
• Subscription streaming services. While public streaming homepages like Netflix are indexed, the actual shows/movies and your tailored account homepage exist in unindexed databases. This applies to music services like your Spotify library too.
• Cloud storage. The data and content you store online are usually protected by passwords and encryption, keeping them hidden from search engines.
• Intranet sites. Internal sites for employee communications and documents or student portals are protected from the public web. Most companies even prevent their login page from being indexed for extra security.
• Travel reservation pages. Your specific upcoming flight/hotel reservations with airlines and booking sites are held on dynamic deep web pages, so crawlers won’t be able to index your specific itinerary.
• Webmail inboxes. While you can likely find your email login page through a search engine, individual inboxes aren’t indexed — and neither are the emails you send or receive.
• Online retail order histories. Your order details and individual account areas on e-commerce sites (like Amazon) exist behind password-protected pages. As with most unindexed pages, these are dynamically filled with your personal information.

In addition, investigating the deep web is a massively important part of cybersecurity research. That’s because researchers (and hackers) can use the deep web to find information about you or specific companies. Hackers and penetration testers might use this to send targeted phishing emails (“spear-phishing”) or find hidden pages they can probe for weaknesses.

Deep Web vs. Dark Web: What’s the Difference?

Think of the internet like an iceberg — the surface web is everything you can see, and the deep web is everything below the surface. While the dark web is below the surface too (and, therefore, part of the deep web), it’s not the same thing. Confusing, I know, but let me explain.

If we define the deep web as any website that you can’t find on Google, Bing, or any other “regular” search engine, then the dark web falls under that same definition. However, the dark web has another layer of obfuscation that means regular browsers can’t access it. Going with my earlier metaphor, we can consider this at the very bottom of the iceberg.

Here’s a quick summary of the main differences:

Can You Access the Deep Web?

Yes, and it’s really easy to do so — you probably access the deep web multiple times a day without realizing it.

You don’t need to use a specific browser to visit the deep web. Most of the time, all you need is to log in to any of your personal accounts. Otherwise, you just need to know the specific URL of the website you’re visiting, and you’re good to go.

Is the Deep Web Illegal or Dangerous?

The deep web itself is neither illegal nor inherently dangerous. However, the nature of what you're trying to access on the deep web — and how you go about doing it — determines whether dangers or illegal activities are involved.

The deep web hosts a vast array of legitimate content like email inboxes, online banking portals, subscription services, and more. If you're just logging into accounts you have proper access to, there's no risk in accessing this unindexed portion of the internet.

That said, the deep web can contain illegal or dangerous content though. That can include pages that host malware or are used in phishing scams, which most legitimate search engines will automatically refuse to index.

Additionally, search engines in specific regions may not index websites that are blocked by the local government, as is the case in restrictive countries like Russia or the UAE. Using these websites may be illegal in these jurisdictions.

How to Stay Safe on the Deep Web

When you explore the deep web, remember these essential safety points to keep you and your devices safe:

• Protect your accounts. Use strong, unique passwords for every account you make, and use two-factor authentication (2FA/MFA) where possible. A password manager can also make this easier. That way, if you accidentally enter your account details into a fake login page, you can limit any damage — if not prevent it completely.
• Invest in safety tools. When diving into the unknown, I recommend using a reputable antivirus tool (like Norton 360) and a VPN. The former will quarantine any suspicious files and the latter will encrypt your data and hide your IP address so it can’t be logged by third parties. Identity theft monitoring services can be a good idea, too. Just make sure to use a trusted VPN (free VPNs can often be more dangerous than not using one at all).
• Treat every web page with suspicion. It’s best to assume that everything is a scam unless proven otherwise. You can always check if a page is legitimate by checking the URL closely (look for HTTPS, which is more secure than HTTP) and clicking the lock icon in your address bar to check its security certificate.
• Use a private browser. Hackers and snoops can use “fingerprinting” (assigning an identity to your specific browsing data) to track your movements around the web. Before exploring, open a private browser window that has no plugins or logged-in accounts. You can also opt for private OSs like Tails (“The Amnesic Incognito Live System”).
• Think before you click. Hovering over links will show the URL they’re sending you to. Remember, clicking unknown links could send you to scam pages, disturbing content, or malware. If anything starts downloading, cancel it immediately and run an antivirus scan.
• Check your local laws. Some websites aren’t indexed because they’re banned in your region. Depending on the region, accessing these sites may carry penalties, so I always recommend checking their legal status if you’re in doubt.
• Don’t hack anything. I know it sounds obvious, but unless you’re authorized to access a page hidden behind a login screen, don’t try. This carries heavy repercussions in many countries. The only exception to this rule is if you have explicit permission as part of a bug bounty program or an authorized penetration test.

Editor's Note: Transparency is one of our core values at vpnMentor, so you should know we are in the same ownership group as ExpressVPN. However, this does not affect our review process.

FAQs on the Deep Web

How do I run a deep web search?

Try a private search engine. More popular search engines often exclude results for a variety of reasons. So, you may have a better chance of finding deep web pages on a private search engine, like Ahmia, Torch, DuckDuckGo, or Wolfram Alpha.

What exactly is on the deep web?

Any web page that isn’t automatically indexed by search engines. These can include anything from internal company networks to your online shopping cart. Streaming content and email inboxes are also part of the deep web.

How big is the deep web?

Experts estimate that the deep web comprises 95% of the modern internet. It’s hard to come up with a specific figure, though, as websites and pages are created and deleted every day. But, to give you an idea, 9.7 billion emails are sent daily in the US alone — these aren’t indexed, nor are users’ individual inboxes. And that’s just emails.

TL;DR: The Deep Web Isn’t as Scary as You Might Think

The deep web is the (mostly) benign majority of the internet, and it comprises all the web pages that search engines can’t (or won’t) index. From personal social media news feeds to app settings pages, your e-banking account to paywalled news articles, the deep web is largely made up of useful, private information.

That’s not to say exploring the deep web isn’t risky — it’s always possible to come across a website that hosts malware or a phishing scam. However, it’s still considerably safer to explore than the dark web, which has only a handful of legitimate, lawful web pages. Unfortunately, because so many people (and particularly the media) often conflate the two, the deep web has a bad reputation thanks to its badly behaved sibling.