A few small internet disasters, continued

IPv6 and BGP are often presented as the clean, inevitable future of networking. In practice, both expose the same old problem: a good technical design still has to pass through old equipment, inconsistent implementations, limited budgets, operational risk, and people who would rather not touch a working network.

The following is less a tutorial than a tour of the awkward corners that appear when IPv6 and BGP leave the design document and enter ordinary homes, restaurants, schools, and small businesses.

1. Every household has its own IPv6 problems

IPv6 is supposed to solve the address shortage created by IPv4. IPv4 has only 2^32 addresses—just over four billion—and the protocol was standardized in 1981, when storage, processing power, and the number of connected computers were all difficult to estimate. As Vinton Cerf put it in 2011:

Who the hell knew how much address space we needed?

— Vinton G. Cerf

By the 1990s, it was clear that IPv4 would eventually run out of room. IPv6 expanded the address space to 2^128, which should be sufficient for an extremely long time. What happens after that is another question; perhaps IPv60 will exist by then.

The shortage was not caused only by the size of the address space. IPv4 allocation was uneven, and some regions have a high total allocation but very few addresses per person. The United States benefited from its early role in the development of the internet and from having considerable influence over address distribution. With a relatively small population, it has historically been possible for many users to receive several IPv4 addresses each. In places with that kind of allocation, port forwarding and DMZ functions on home routers are ordinary tools because carrier-grade NAT is not always in the way.

IPv6 was therefore designed with more than address conservation in mind. Its supporters also hoped it would restore the end-to-end principle: if no firewall or other filtering device intervenes, one endpoint should be able to reach another directly by IP address, without NAT or application-layer relays.

That requirement appears in more places than it may seem. Remote access to a home network, BitTorrent and other peer-to-peer applications, P2P games, and direct IPsec tunnels can all benefit from direct connectivity. On IPv4, NAT usually means that endpoints must use a publicly reachable server as a relay or rely on increasingly elaborate NAT traversal techniques.

Some IPv6 enthusiasts consequently oppose NAT in IPv6 almost as a matter of principle. Their ideal is to restore the internet to its original shape. The ideal is understandable, but the operational world is rarely that tidy.

The dream of a global fabric

Network engineers are sometimes said to share a common fantasy: a fully fabric-like network. Imagine one enormous, extremely reliable Layer 3 switch covering the whole planet, with hundreds of billions of ports. Every connected device—from a supercomputer to a child’s smartwatch—would plug directly into one of those ports, whether the physical connection were wired or wireless.

In that imagined network:

• endpoints could communicate directly;
• traffic would be switched at line rate with abundant bandwidth;
• applications would not need to care about the underlying implementation;
• the network would be flexible, highly available, and resilient to failures.

IPv6 naturally fits into that picture. Unfortunately, the real world has old firmware, incompatible interpretations of standards, commercial constraints, and networks that were never designed to be upgraded all at once.

The first conflict: NAT is forbidden, except when it is needed

RFC 4864 lists several reasons IPv6 does not need traditional NAT such as Basic NAT or NAPT:

• the address space is large enough that NAT is not required to conserve addresses;
• NAT is stateful and can interfere with communication;
• many protocols were not designed to operate through NAT.

Those arguments are technically reasonable. The trouble is that rejecting a technology in principle does not eliminate the requirements that led people to use it.

There is no universally accepted equivalent of the IPv4 NAT standards for IPv6, yet IPv6 NAT exists anyway. Vendors including Cisco, Huawei, H3C, and 3Com have implemented NAT66, as has Linux through tools such as iptables. Their implementations are not necessarily identical. When NAT66 is finally needed, these differences can produce obscure failures that are difficult to reproduce or even notice.

NAT66 is not generally treated as a recommended IPv6 deployment pattern, so it has not received the same amount of large-scale testing as more conventional features. That lack of common practice makes the resulting problems more mysterious, not less real.

A less controversial alternative is NPTv6, or IPv6-to-IPv6 Network Prefix Translation, described by RFC 6296. NPTv6 changes the network prefix but does not remap ports. It is stateless and therefore applies beyond TCP and UDP. At the edge of a network, an upstream prefix such as 2001:db8:1001::/48 can be translated to an internal prefix such as fd00::/48, while the remaining bits stay unchanged.

NPTv6 has a standard and is closer to the original idea of address translation, but practical support is still uneven. Older Linux implementations reportedly had compatibility issues with CONNTRACK; mechanisms such as NETMAP may help, although the exact situation depends on the implementation. Software routers also have a performance disadvantage: commercial hardware can accelerate ordinary routing and NAT44, while NPTv6 may fall back to CPU processing. A feature that works in a lab can therefore become a bottleneck in a production router.

The broader lesson is simple: ideals matter, but they have to be reconciled with equipment that people can actually buy and operate.

Android phones and the DHCPv6 problem

Google has historically opposed DHCPv6 on Android and supported SLAAC instead. This creates an immediate compatibility problem for networks that depend on DHCPv6 as their main—or only—method of assigning IPv6 addresses. An Android phone may simply fail to join the IPv6 network.

Before Android supported RDNSS, even a pure IPv6 network could be inconvenient because the phone needed IPv6 DNS information configured separately rather than receiving DNS settings through the IPv4 DHCP process.

SLAAC has attractive properties. It is stateless, and clients can react quickly when a prefix or other network information changes. Managed networks, however, often prefer DHCPv6 because it offers central control, logging, auditing, and a clearer record of which device received which address.

SLAAC does not really hand out an address in the traditional DHCP sense. It advertises a prefix and allows the endpoint to construct its own address. That is convenient, but it can be difficult for an enterprise that needs to identify and manage every device. DHCPv6 provides a stateful service that is more familiar to administrators.

DHCPv6 can also be useful for specific enterprise functions, such as automatically provisioning IP phones through options 120 and 125, or supporting PXE boot through options 60, 66, and 67. It is possible to isolate those functions in separate VLANs, but real networks are not always designed with that much foresight. There are still plenty of organizations where every device shares a single 192.168.0.0/16 network.

Prefix lengths that break assumptions

IPv6 was designed around a familiar division: the first 64 bits represent the prefix and subnet portion, while the last 64 bits serve as the interface identifier. For example, an address such as 2001:db8::1/48 leaves 16 subnet bits between the /48 prefix and the usual /64 interface boundary, allowing 2^16 = 65536 subnets.

Many protocols and operating systems assume that an interface will ultimately receive a /64. But some providers or managed networks hand out prefixes longer than /64, such as /96 or /112. The /127 exception is commonly used for point-to-point links and is comparable to an IPv4 /31, but other long prefixes can cause subtle failures.

The first casualty is often SLAAC. If the prefix is already too long, there may not be enough space for a standard interface identifier. Without DHCPv6 to provide an address by another method, a device may refuse to join the network.

This is probably more common in unusual environments—campus networks, managed enterprise networks, or very small VPS providers—than in ordinary home broadband. That makes it easy to overlook, but the incompatibility is still genuine.

The restaurant Wi-Fi test

Suppose a home network has been configured correctly for IPv6. Prefix delegation works, remote access is protected by a VPN, and the home NAS is reachable from outside. Everything looks excellent.

Then someone goes downstairs to a restaurant, orders lunch, connects to the restaurant’s Wi-Fi, and discovers that mobile payment takes an unusually long time. Later, trying to stream a movie from the NAS, the connection does not work at all.

The reason is often not that IPv6 is inherently slow. It is that IPv6 deployment has to include the networks used by ordinary customers, and the return on investment for those networks is close to zero.

The smoke-covered Wi-Fi router

A small restaurant typically has several characteristics:

• it may use inexpensive residential broadband rather than a business circuit;
• the number of customers varies considerably;
• customers need internet access, but usually not premium performance;
• customers are transient, especially near transport hubs;
• the owner or staff may have little interest in network administration.

Consumer mobile internet expanded rapidly around 2010–2012. Wi-Fi access in restaurants became common around the same period. Wi-Fi 4, or 802.11n, was standardized in 2008. Wi-Fi 5, or 802.11ac, arrived in 2014 and focused on the 5 GHz band; a successor that also supported 2.4 GHz, Wi-Fi 6 or 802.11ax, did not arrive until 2019.

That leaves a large installed base of Wi-Fi 4 routers. They were cheap, widely produced, and good enough for ordinary web access. Rural areas and suburban fringes have extended the life of this equipment, including routers that have passed through several owners or remained in stock for years. If a dusty router is sitting in a restaurant corner under a layer of cooking fumes, assuming that it is an old Wi-Fi 4 device is usually safe.

Wi-Fi generation and IP version are technically separate. A Wi-Fi 4 router can carry IPv6 perfectly well. The problem is that a router manufactured during the Wi-Fi 4 era may never have supported IPv6 in its firmware. Even if a later firmware update added support, IPv6 may still be disabled by default.

IPv6 also offers little obvious benefit to the restaurant:

• an IPv4-only connection still reaches nearly every public service;
• the owner may not know how to configure IPv6;
• customers are visiting to eat and browse short videos, not to hold a financial conference over a carefully managed network.

The situation becomes troublesome when DNS returns both A and AAAA records. Consider the following chain of events:

1. A DNS server returns both IPv4 and IPv6 addresses for a domain.
2. The restaurant router forwards the DNS response but has no IPv6 support and does not remove the AAAA record.
3. The customer’s phone receives the IPv6 address and tries IPv6 first.
4. The router cannot forward IPv6 traffic, so the attempt stalls until it times out.

The payment application then appears to freeze. The customer may blame weak mobile reception or an overcrowded restaurant, both of which are plausible explanations. Few people will identify the real cause as an incomplete dual-stack deployment.

Happy Eyeballs can reduce this problem by attempting IPv4 when IPv6 does not respond, but implementations differ. A device may not fall back quickly enough, so the delay remains visible.

From the restaurant owner’s perspective, customers will not stop visiting because the Wi-Fi lacks IPv6. The business case for upgrading is therefore almost nonexistent.

When the optical modem takes control

Network enthusiasts often dislike provider-controlled optical network terminals. They want the router to handle dialing and routing while the optical modem acts only as a media converter. This gives the customer more control and can sometimes improve performance.

The provider’s perspective is different. If the optical modem performs the connection and remains centrally managed, the provider can distribute IPv6 settings consistently. This is one reason new residential broadband installations commonly use provider-managed dialing by default.

A restaurant may also use residential broadband. If every device connects directly to the optical modem—including its built-in Wi-Fi—the provider’s IPv6 configuration can work as intended. The modem advertises IPv6 prefixes and routes traffic, and customers receive IPv6 connectivity.

Older optical modems, however, may have weak Wi-Fi or no Wi-Fi at all. The natural response is to buy a separate router, connect it, and turn on its wireless network. That immediately recreates the previous problem: no matter how well the optical modem supports IPv6, the separate router can block the entire path if its firmware does not support IPv6.

This may improve over time as more restaurants use the Wi-Fi built into newer optical modems. It is attractive to owners because it removes the cost and maintenance of an additional router. But old equipment remains in service for a long time.

IPv6 and the PMTU black hole

Some users report that everything becomes slightly slower after enabling IPv6. DNS failures are one possible explanation. Another is a Path MTU Discovery black hole.

IPv6 routers are not permitted to fragment packets in transit. If a packet is larger than the next link’s MTU, the router drops it and sends an ICMPv6 message telling the source to adjust its packet size. That design is clean, but it depends on the ICMPv6 message arriving successfully.

Some carrier equipment blocks or loses ICMPv6 messages. The reason may be a security policy, a broken configuration, or ordinary packet loss. Whatever the cause, the source never learns that its packet was too large. The transport protocol then waits for a timeout and retransmits. TCP head-of-line blocking and related effects can make the resulting slowdown even more obvious.

The protocol specification is not necessarily the problem. The problem is that an intermediate network does not reliably carry the control messages that the specification assumes.

Multihoming for small networks

Multihoming means connecting a network to multiple upstream providers, often for load balancing or failover.

A large organization can obtain provider-independent IPv6 address space and an autonomous system number, establish BGP sessions with its carriers, and advertise the same prefixes through multiple links. A small company or household usually cannot take that route. It may not qualify for its own allocation or AS number, may not be able to afford the service, and may find that providers simply refuse to establish BGP sessions. Each broadband connection may also supply a different, dynamically changing prefix.

Even a company that can obtain BGP may reserve it for important services because the circuit is expensive. Ordinary traffic continues to use cheaper broadband.

Several alternatives exist, but none is perfect:

• distribute every provider’s prefix to every endpoint and configure multiple IPv6 addresses, leaving source-address selection to the endpoints;
• keep the backup connection IPv4-only, which avoids some IPv6 problems but is not true IPv6 multihoming;
• use a tunnel provider, accepting additional latency and the risk of addresses that are treated differently by remote networks.

The controversial option is to use NAT66 or NPTv6. NAT44 makes this relatively straightforward in IPv4 because stateful translation can steer flows across different links and remember which path to use for return traffic.

IPv6 has no comparable universal consensus. NAT66 is still treated by many as a violation of the design, and support varies between devices. Home routers and low-end business routers often cannot implement the required policy reliably. NPTv6 is more acceptable in principle, but vendor support is similarly uneven, and some documentation confusingly treats NAT66 and NPTv6 as the same thing.

There may be better solutions today, but for many small networks the practical decision is simpler: avoid the problem entirely. Multihoming is abandoned, and sometimes IPv6 as a whole is disabled.

Why campus networks remain IPv4 islands

Schools and universities may be praised publicly for their IPv6 work while their internal networks still do not support it. A common explanation is that there are too many devices to upgrade consistently. That explanation is not unique to campuses; small businesses face the same problem.

Old equipment and complicated topology

A campus includes classrooms, dormitories, cafeterias, access points, routers, Layer 3 switches, and large numbers of surveillance cameras. Endpoints do not all need native IPv6 support, but the network devices connecting them must be upgraded, configured, and tested. Some need firmware updates; others need replacement.

Institutional purchasing is rarely as simple as ordering new hardware. Public organizations may need tenders, budget approvals, depreciation calculations, and other administrative steps. Even after replacement, every routing design must be recreated for IPv6. A network that already works over IPv4 may need a second routing configuration, including protocols such as OSPFv3, while IPv6 documentation and operational experience remain less abundant.

There is also a less technical obstacle: responsibility. If a migration causes a security incident or an outage, someone must be accountable. That risk often outweighs the perceived benefit of IPv6. Unless an institution has a specific administrative target to meet, there is little incentive to take it on.

Some IPv6 evaluations focus heavily on whether a school website supports IPv6 and how much of the public-facing network is reachable. Such measurements can create an official label without proving that the internal campus network is genuinely dual-stack.

Auditing and authentication

Modern schools and enterprises commonly deploy egress auditing, access auditing, and web portal authentication. A student may need to pay for network access and complete a web login before traffic is allowed.

Now imagine that the routing hardware supports IPv6 and the budget is available. The central question becomes: will any traffic bypass the controls after migration?

In earlier deployments, some campus IPv6 paths reportedly bypassed authentication or avoided traffic accounting. Search results still show examples of this kind of behavior. Routing equipment may be able to carry IPv6, but auditing and access-control systems may not have been upgraded with equal care.

A single overlooked device can allow traffic that should have been blocked. From a technical perspective, the mistake may be minor. From an administrative perspective, it can be enough to shut IPv6 down and assign blame to the migration team. The network may remain IPv4-only simply because it is safer politically.

Some historical cases of IPv6 traffic being effectively “free” may have been intentional. In earlier years, IPv6 services were less common, and the education network sometimes supplied IPv6 capacity alongside purchased IPv4 capacity. A school buying 100 Mbps of education-network IPv4 connectivity might receive another 100 Mbps of IPv6 capacity. If that capacity was handed to the campus network to demonstrate IPv6 support, users could consume IPv6 bandwidth without affecting the IPv4 quota.

That arrangement also helps explain why some campuses deployed IPv6 early through the education network, even while their ordinary commercial uplinks remained IPv4-based.

The education network bottleneck

University websites often advertise a total internet capacity such as 40.6 Gbps or 20.3 Gbps. The fractional-looking figure may represent education-network bandwidth, which is frequently much more expensive than ordinary commercial access.

Historical procurement notices show examples such as:

• University of Science and Technology of China: 3 million yuan for 2.5 Gbps;
• Chengdu University: 1.2 million yuan for 1 Gbps;
• Northwest University: 1.89 million yuan for 2 Gbps;
• Anhui University of Finance and Economics: 360,000 yuan for 200 Mbps;
• Hubei Vocational College of Communications: 300,000 yuan for 300 Mbps.

These are annual prices. They may include more than bandwidth: IPv4 address blocks, IPv6 /48 allocations, and an *.edu.cn domain are also part of the value. Still, the bandwidth itself is limited compared with the number of students and staff sharing it. A few gigabits—or a few hundred megabits—can quickly become a problem when thousands of users compete for access.

Universities therefore often buy cheaper commercial connectivity for everyday traffic. The major carriers are happy to provide it because universities offer stable, large-volume contracts, and providers may help design the internal network as part of the deal.

The education network supported IPv6 early, so initial campus IPv6 traffic often passed through it and then reached the public internet through exchange points. Later, commercial carriers also began offering IPv6, creating a multihoming problem. A university generally cannot establish BGP with all of its providers, and sending all users through one IPv6 path can congest the expensive education connection.

The options are unattractive:

• implement multihoming at considerable cost and complexity;
• use NAT66 as a compromise;
• or shut IPv6 off for internal users and keep it only for externally published services.

The last option is common. A school may bring IPv6 only to the central data center for public-facing services while continuing to use NAT44 for campus users. If IPv4 works, that is often considered sufficient.

The simplest reason: IPv6 is unnecessary

Small businesses have another reason not to deploy IPv6: they do not need it.

Many internal services are accessed by typing an IP address directly, without internal DNS or TLS. A service might be reached at 192.168.3.3:81 because port 80 is already occupied by an office system. Replacing that with an IPv6 address such as 2001:db8::dead:beef:1234:5678 is less convenient. Users must remember the long address, distinguish : from ::, and add square brackets when specifying a port.

Other factors reinforce the decision:

1. A small internal network can fit comfortably inside a fragment of IPv4 space.
2. Existing and newly purchased security devices may still have poor IPv6 support; some require additional licenses to inspect IPv6 traffic.
3. A company may discover that IPv6 lets employees bypass web restrictions because its security equipment only audits IPv4. The owner may respond by disabling IPv6 altogether.
4. The administrator may lack the necessary expertise—or may reasonably decide that a modest salary does not justify wrestling with an unfamiliar migration.

In that environment, IPv6 is not perceived as an improvement. It is another system to configure, monitor, and explain.

When the carrier changes its mind

Some broadband providers treat suspected PCDN users with a familiar combination of measures:

• change the NAT level to NAT4, or symmetric NAT;
• stop assigning IPv6;
• reduce the upload rate to a very low level, such as 5 Mbps.

More aggressive cases involve disconnecting the service entirely. If a line loses its IPv6 address, access to IPv6 content naturally disappears as well.

Even a policy that formally requires providers to leave IPv6 enabled would not necessarily guarantee public IPv6 connectivity. A provider could assign an IPv6 NAT address from fd00::/7 and technically claim that IPv6 is present, while offering no publicly reachable address.

A crippled dedicated circuit

Some providers sell a product that places two services on one fiber connection—for example, a lower-speed “dedicated line” alongside a faster business broadband service. The business broadband component is essentially an enlarged residential connection: more expensive, but with similar routing and behavior.

The dedicated component can be more useful. Although it may still run over PON and have less bandwidth, its routing can be substantially better. In some observations, overseas nodes that became unusably slow on the business broadband connection during the evening remained smooth over the dedicated line. Packet captures showed large numbers of TCP retransmissions on the former, while the latter sustained its nominal speed.

Such a circuit may also provide a fixed, exclusive public IPv6 allocation. Unlike residential broadband, it does not require periodic redialing, so services do not disappear whenever the connection is renegotiated.

The catch is that the cheapest version may provide public IPv6 but no public IPv4. If the goal is to publish a company website or another service, would it be worth paying more for a version that includes IPv4?

For many customers, the answer would be yes. IPv4 remains the most compatible and widely reachable option. That naturally raises the question: if IPv4 works, why bother using IPv6?

The problem becomes more serious outside major urban areas, where old optical modems and routers may still be common. A service exposed only over IPv6 may be unreachable to users whose local networks cannot carry it. Removing public IPv4 therefore reduces the practical reach of the service considerably.

A fixed IPv6-only circuit is not useless, however. Its stable route and predictable link can be valuable for hybrid cloud deployments. A company can connect its own data center to cloud resources through a VPN. The VPN does not care whether its underlying transport is IPv4 or IPv6, nor whether the traffic inside the tunnel is IPv4 or IPv6. If the cloud side can obtain a static IPv6 address cheaply—or even at no additional cost—the combination of a stable address, an uninterrupted link, and better routing can produce a very good site-to-site VPN.

2. Using BGP to watch short videos?

BGP is one of the most widely used wide-area routing protocols. The underlying technology is powerful, but the cost of obtaining it is often more important than the protocol itself.

A minimal BGP configuration might involve an AS number, a router ID, a neighbor address, and the local prefixes to announce. Beyond that, mistakes can cause missing routes—or accidentally turn a machine into a transit point through which multiple carriers happily exchange traffic.

Why use BGP at all?

Home broadband is normally out of the question. In most places, residential customers cannot establish BGP sessions through their ordinary access service.

A conventional enterprise circuit usually works differently. The provider assigns an address, often through PPPoE, IPoE, or static configuration. The address belongs to the provider, which installs a route toward the customer. The customer sets the provider’s router as the default gateway and traffic flows.

A larger allocation works the same way. Suppose a customer leases 198.18.0.0/28. After accounting for the network address, broadcast address, gateway, and other required reservations, 13 addresses may remain usable. One can be placed on the WAN interface, while the others can be used in a NAT pool or assigned directly to servers. If the servers run a hypervisor, the customer can even create multiple VPS instances behind the circuit.

The important point is that these addresses belong to the carrier. Even if a customer configures them locally, routers elsewhere do not automatically know where those prefixes are located. The provider must advertise the route on the customer’s behalf.

This is where dynamic routing enters the picture. A routing protocol allows a router to tell other routers which prefixes it can forward and how desirable the path is. The information propagates through the network, subject to aggregation and policy, until other networks know how to reach the announced addresses.

BGP is one of the protocols used for that purpose and is especially common between autonomous systems. The details of its advantages over protocols such as OSPF depend on the deployment, but the basic reason for using it is straightforward: it lets an organization control and advertise its own prefixes across multiple upstream networks.

The price of BGP

The difficulty is the price. BGP access can be dramatically more expensive than ordinary dedicated internet access.

One carrier price list gives the following illustrative comparison:

• “Dedicated internet access”: 100 Mbps for 17,140 yuan;
• dedicated internet access with BGP routing: 100 Mbps, full transit, for 56,400 yuan.

Adding the BGP option raises the price by more than three times. Even a cheaper non-transit BGP option can cost much more than a conventional connection.

The exact distinction between “non-transit” and “full transit” depends on the provider. One interpretation is that full transit allows traffic belonging to other autonomous systems to pass through the customer’s AS, while non-transit service does not. The terminology should be confirmed with the carrier rather than assumed.

The high price has encouraged a wide range of products marketed with BGP-related names: full-transit BGP, half-transit BGP, static BGP, and services that are not really BGP but use the label because it sounds useful. Some multi-line data centers are examples. Quality, route diversity, and reliability vary considerably.

For a company that is not operating a cloud platform or another network-intensive service, paying several times more each month simply to browse social media is difficult to justify.

Static BGP, or having someone else announce the prefix

A related service is often called static BGP or route origination on behalf of the customer. The customer owns or leases a prefix and asks a carrier to announce it from the carrier’s autonomous system. From the wider internet, the prefix appears to belong to the carrier’s AS. The carrier then supplies an ordinary circuit and installs a static route toward the customer, without requiring a BGP session at the customer site.

If the customer obtains this service from several major carriers and connects all of the circuits to the same edge router, the result can resemble BGP multihoming: users of different networks may reach the same address through different providers. It is considerably cheaper than running full BGP.

The compromise is predictable. The customer does not receive the same degree of control or failure handling as with genuine BGP, and the stability of the arrangement depends heavily on the provider’s static routing and operational policies.

IPv6 and BGP are both capable technologies. The hard part is not proving that they work. The hard part is making them work across old routers, incomplete standards support, inconsistent firmware, expensive circuits, security audits, and networks whose owners have no compelling reason to change them.

That is why the future of networking often arrives unevenly: excellent in a laboratory, useful in a data center, tolerable in a home, and mysteriously slow in the restaurant downstairs.