Chapter 21: Network Troubleshooting

Hardware tools include:

Cable testers, TDRs, OTDRs, certifiers, voltage event recorders, protocol analyzers, cable strippers, multimeters, tone probes/generators, line testers, butt sets, and punchdown tools.

Broken cables

Broken cables don't mean intermittent problems, and they don't slow data down. They make permanent disconnects.
Broken cable could be defined as:
1. Having an open circuit
-where one or more of the wires in a cable simply don't connect from one end of the

How to deal with broken cables

Cable testers can tell you if you have a continuity problem or if a wire map isn't correct.
TDRs (time domain reflectometers) and OTDRs (optical time domain reflectometers) can tell you where the break is on the cable.
TDRs work with copper cables and OTD

Certifiers

Test a cable to make sure that it can handle its rated amount of capacity.
Certifiers require some kind of loopback on the other end of the cable run.
When a cable is not broken, but not moving data the way it should, test it with a certifier; i.e. don't

Optical power meter

Used in fiber optics, uses a high-powered source of light at one end of a run and a calibrated detector at the other end. This measures the amount of light that reaches the detector.

Voltage event recorder

Used to detect fluctuations in power.

Symptoms of heat and power issues

Server rooms that get too hot at certain times of the day
Switches that fail when an AC system kicks on
Etc.

Temperature monitor

Used to detect issues with heat.

Protocol analyzers

Monitor the different protocols running at different layers on the network.
A good protocol analyzer will give you Application, Session, Transport, Network, and Data Link layer information on every frame going through your network.
Can be either hardware

Uses for a protocol analyzer

Is something trying to start a session and not getting an answer?
Is some computer on the network placing confusing information on the network?
Is a rogue DHCP server sending out responses to DHCP requests?

Cable stripper/snip

Helps you make UTP cables. Also need crimpers as well.

Multimeters

Test voltage (both AC & DC), resistance, and continuity. Also a great fallback for testing continuity.

Tone probes & Tone generators

They only have one job: to help you locate a particular cable.
You will never use a tone probe without a tone generator.

Butt sets

Used to tap into a 66- or 110-block to see if a particular line is working.

Line testers

Simple devices used to check the integrity of telephone wiring.
Use to check a twisted pair line to see if it is good, dead, reverse wired, or if there is AC voltage on the line.

Punchdown tools

Put UTP wires into 66- and 110-blocks.

traceroute/tracert command

The tracert command used in Windows that sends only ICMP packets. Traceroute in UNIX/Linux/Cisco can send either ICMP packets or UDP packets, but sends UDP by default. Because many routers block ICMP packets, if you run tracert from a Windows PC and it fa

traceroute with IPv6

By default is IPv4, but can use a switch to use IPv6.
Windows:
tracert -6
UNIX/Linux:
traceroute6
traceroute -6 (some variations of Linux)

ipconfig/ifconfig/ip commands

ipconfig (Windows)
ifconfig (UNIX)
ip (linux)
The commands tell you almost anything you want to know about a particular computer's IP settings. Just typing 'ipconfig' returns basic info, while using 'ipconfig /all' give detailed information such as DNS se

arp command

Address Resolution Protocol that is used to resolve IP addresses to MAC addresses. When a computer learns various MAC addresses on its LAN, it jots them down in the ARP table.
Enables you to view and change the ARP table on a computer.
Example using the c

ping command

Ping command uses ICMP (Internet Control Message Protocol) packets to query by IP or by name. Works across routers, so it's generally the first tool used to check if a system is reachable. But since many devices block ICMP packets, a failed ping doesn't a

arping command

Can be used when 'ping' doesn't work since it uses ARP frames instead of ICMP. BUT, since it uses frames and not packets, arping doesn't cross routers; can only be used in a broadcast domain.
Not supported in Windows, but can be used in UNIX/Linux systems

pathping command

A Microsoft only utility that combines functions of ping and tracert with some additional functions.

Connectivity software

Utilities like ping and traceroute.

nslookup/dig commands

nslookup (all OSs) and dig (UNIX/Linux) commands are used to diagnose DNS problems.
dig can be used to query all MX records for example from a host.

hostname command

It returns with the host name of the computer you are on.

mtr (My Traceroute) command

Is dynamic in that it keeps running the traceroute. Windows doesn't support mtr.

route command

Gives you the capability to display and edit the local system's routing table.
To show the routing table:
route print
netstat -r

nbtstat command

Windows-only program that is the command line equivalent to Window's My Network Places or Network icon.
Always run it with a switch. Most useful one is -n which shows the local NetBIOS names.

netstat and ss commands

Netstat displays information on the current state of all of your running IP processes; which sessions are active and can also provide statistics based on ports or protocols (TCP, UDP, etc.)
Using 'netstat' without switches shows only the current sessions.

Packet sniffer

Packet sniffer, protocol analyzer, or packet analyzer. All of these names define a tool that intercepts and logs network packets. Can be hardware or software.
Most popular is Wireshark.
But if your interface has no GUI installed, such as a server, you wou

Port scanners

A program that probes the ports on another system, logging the state of the scanned ports.
Used to look for unintentionally opened ports that might make a system vulnerable to attack.
Most popular (and free) port scanner is Nmap, Windows users also use An

Throughput testers

Enable you to measure the data flow in a network, such as several speed test sites online.

Looking glass sites

Remote servers accessible within a browser that contain common collections of diagnostic tools such as ping and traceroute, plus some Border Gateway Protocol (BGP) query tools. Used to perform things such as ping or traceroute from a location outside of t

Troubleshooting process

1. Identify the problem
2. Establish a theory of probable cause
3. Test the theory to determine the cause
4. Establish a plan of action to resolve the problem and identify potential effects
5. Implement the solution or escalate if necessary
6. Verify full

Troubleshooting step 1: Identify the problem

Getting to the true problem, not just what someone tells you it is.
Gather information, duplicate the problem if possible, question users, identify symptoms, determine if anything hass changed, approach multiple problems individually.

Troubleshooting step 2: Establish a theory of probable cause

Question the obvious
Consider multiple approaches: top-to-bottom/bottom-to-top OSI model, divide and conquer

Troubleshooting step 3: Test the theory to determine cause

Once a theory is confirmed, determine next steps to resolve the problem.
If theory is not confirmed, reestablish a new theory or escalate.

Troubleshooting step 4: Establish a plan of action and identify potential side effects

If the plan is complex, write down the steps.
Plan for and mitigate potential side effects.

Troubleshooting step 5: Implement the solution or escalate as necessary.

Once you think you've fixed the problem, you should try to make it happen again. If it does reoccur, then you know you haven't fixed it.

Troubleshooting step 6: Verify full system functionality and implement preventative measures

If you've replaced a NIC in a server, then it has a different MAC address that could affect other things, such as logon security controls or your network management and inventory software.
Implement preventative measures to avoid a repeat of the problem.

Troubleshooting step 7: Document findings, actions, outcomes

Vital for two reasons:
1. you're creating a support database to serve as a knowledge base for future reference, enabling everyone on the support team to identify new problems as they arise and know how to deal with them quickly without having to duplicate

Hands-on problems

Refers to things that you can fix at the workstation, work area, or server. Include physical problems and configuration problems.
Examples:
-power failure or power anomalies
-hardware failure
-EMI or RFI
-transceivers can go bad
-interface errors (wall ja

LAN problems

Incorrect configuration of any number of options in devices can stop a device from accessing resources over a LAN.
An example would be having duplicate IP addresses on the same network.

Server misconfigurations

Can block all or some access to resources on a LAN.
Misconfigured DHCP settings on a host above can cause problems, but limited to the host.
If these settings are misconfigured on the DHCP server, many more machines and people can be affected.
A misconfig

Adding VLANs

Example:
Bill divides a switch of 24 ports into 4 VLANs of 6 ports each, but mistakenly put 7 ports on VLAN 1 and 5 ports for VLAN 2 when setting it up.
Cable placement error is when a cable meant for one port is incorrectly plugged into a different port

Link aggregation problems

Link aggregation (AKA NIC teaming) is a way to scale up Ethernet speeds by using multiple NICs in tandem to increase bandwidth in smaller increments.
Two link aggregation protocols:
The IEEE 802.3ad spec Link Aggregation Control Protocol (LACP)
Cisco prop

NIC teaming redundancy

You can team two NICs as one logical unit, but set them up with one NIC as the primary--live--and the second as the hot spare--standby. If the first NIC goes down, all traffice will automatically flow through the second NIC.
Key here is that multicast tra

Router problems

Router configuration issues can be many, such as specifying the wrong routing protocol or misconfiguring the right protocol.
An ACL (access control list) might include addresses to block that shouldn't be blocked or allow access to network resources for n

MTU mismatch

When your network's packets are so large that they must be fragmented to fit into your ISP's packets.
Path MTU discovery solved this by sending fixed-sized ICMP packets that wouldn't be fragmented until the ping is successful, then automatically sets the

Appliance problems

Routers contain many features such as routing, NAT, switching, and intrusion detection system, firewall, etc. These complex boxes, such as Cisco's Adaptive Security Appliance (ASA) are called network appliances.
One common problem with appliances is techn

Company security policy

Can set throttling policies that can define the maximum amount of bandwidth per day a department could use.
Then add a blocking policy, so that if anyone goes over that limit, the company will block all traffic of that type for a certain amount of time.
T

Escalating problems

Examples include things out of your scope of duty or knowledge, such as:
-broadcast storming
-switching loops
-routing problems
-routing loops
-proxy ARP

Broadcast storm

Result of one or more devices sending a nonstop flurry of broadcast frames on the network.
First sign is when every computer on the broadcast domain suddenly can't connect to the rest of the network. Every actvity light on every node is solidly on. Comput

Switching loops

AKA bridging loop.
When you connect and configure multiple switches together in such a way that causes a circular path to appear. Switching loops are rare because all switches use the Spanning Tree Protocol (STP), but they do happen.
Symptoms:
Identical t

Routing loops

Occurs when interconnected routers loop traffic, causing the routers to respond slowly or not at all. The old RIP dynamic routing protocol could have issues with routing loops because of its slow convergence time, but most routing loops are caused by stat

Split horizon issue

When a router learns a route through a certain interface, it will not communicate that route out the same interface.
Was an issue with RIP when routers would report their hops away. Router C might tell Router D that it was two hops away from Router A. Rou

Proxy ARP

The process of making remotely connected computers truly act as though they are on the same LAN as local computers. Proxy ARP is done in a number of different ways, with a VPN as the classic example.
If a laptop connects to a network via a VPN, that compu

End-to-end connectivity

Refers to connecting users with essential resources within a smaller network, such as a LAN or private WAN.

IEEE 1905.1

Creates a hybrid networking standard with the goal of flexibly integrating a few types of wired and wireless networking technologies, including WIFI, Ethernet, MoCA, and the powerline standard defined by IEEE 1905, allowing a network to span all four tech

MoCA

Multimedia over Coax.
Provides Ethernet access through your existing coaxial cabling (even when it's being used for video), and is the technology used by Verizon's FiOS product to provide video, phone, and Internet service.

HomePlug (IEEE 1901)

AKA HomePlug HD-PLC.
Provides high speed home networking through a building's existing power infrastructure.
Also called:
Ethernet over power (EoP)
Broadband over power lines (BPL)
Ethernet over power line

nVoy

nVoy certified networking equipment will create a single network for devices spread across Ethernet, WIFI, MoCA, and HomePlug connections.

HDMI 1.4

This standard also creates HDMI Ethernet Channel (HEC), or Ethernet enabled HDMI ports that combine video, audio, and data on a single cable.
Can allow two-way communication and connection-sharing between a TV and set-top boxes.