Routing issues, slow network applications, DNS resolution problems -- a network administrator has to deal with a host of network nuisances on a daily basis. How do you survive when you're constantly under the gun to fix the problems? Like any other professional, you need a solid set of tools.
Not surprisingly, plenty of options exist in the open source camp. Excellent open source software tools are available to help you keep a close watch over your network, as well as meet many other needs of the busy network manager. From monitoring, troubleshooting, and security analysis tools to utilities for keeping track of IP allocations, passwords, and router configurations, here are my top 10 picks of the most essential open source tools for our network admin toolbox -- all free for the downloading.
This is by no means an exhaustive list of open source networking utilities available, and I've merely touched on their capabilities. Are there other free open source tools that you use regularly but we didn't list here? Leave a comment and let us know!
Dig
DNS problems plague us all, and they're easily overlooked when troubleshooting, so you need a reliable tool that provides detailed information about how users' DNS queries are being resolved. Why not use the tool made by the
Internet Systems Consortium, the same group that produces the BIND DNS server software running the majority of DNS servers worldwide? That tool is Dig.
At the heart of it, Dig is a command-line utility that performs DNS queries. That alone is helpful, but Dig can also tell you most everything about the queries and replies -- you'll sometimes need that extra information to determine why you're getting a strange reply from a DNS server. The default output of Dig provides you with all the data you'll require for troubleshooting: reply/error codes from the server, flags used in the query, a reiteration of your query, the answer to your query, how long the query took, which server it received the reply from, and how much data it received in the reply. Dig can be quite useful when you're trying to diagnose slow network applications, by determining how long it takes a computer to get DNS resolution for the application server's domain name.
Dig can ask for a typical name query, replying with an IP address when you give it a domain name. You can also do a reverse lookup: By using the -x switch and giving it an IP address, Dig it will return the corresponding domain name for that IP address. The -t switch lets you specify the type of query you're making, so you can ask for mail server records (MX), name server records (NS), text records (TXT), and more.
If you are sporadically getting incorrect replies to your DNS queries, it's possible that one of your DNS servers has a different set of DNS records than the others. With Dig, you can run the same query against each of your DNS servers to find out which one is providing the erroneous replies. Just give Dig the DNS server's address with the @ symbol in front:
dig @4.2.2.2 www.yourdomain.com
Are you troubleshooting DNS problems with servers that use transaction signatures? Dig lets you specify a TSIG key to use for your queries. Dig also lets you tailor IPv6-only queries to help you troubleshoot IPv6-specific problems.
Dig is a part of the client utilities of the
BIND project. It is not generally installed by default, but is readily available on all Unix, Linux, and BSD variants, including Mac OS X. A Windows version is available too.
Nmap
Carrie Moss used it in "The Matrix Reloaded." Crackers, hackers, and network admins alike rely on it, and every networking consultant better have
Nmap installed on his or her computer. Nmap is available for nearly every platform imaginable and is amazingly useful as a network and security analysis tool.
Nmap is a lightweight security scanner that's heavy on utility. Nmap can perform tasks as simple as a ping sweep to see which IP addresses are active and responding, as well as carry off complex scripts to scan your systems for known vulnerabilities. Another fun feature of Nmap is the ability to analyze the reply packets it receives from a host to determine which OS the host is running.
Nmap is most commonly used to see which services or ports are open or available on a host. It supports both TCP and UDP scanning. You can give it a single host to scan or a CIDR (Classless Inter-Domain Routing) block or an entire list of hosts and networks from a file. A dizzying range of options allows you to specify which types of packets to send out and to see which hosts are susceptible to various remote attacks. Additionally, Nmap provides several options to bypass firewalls and other network filters that would otherwise block your scans.
Nmap also includes the
Nmap Scripting Engine (NSE), which combines custom scripts with existing Nmap functionality to perform more specific discovery and attack analyses than Nmap does by itself. Fyodor and David Fifield gave an excellent talk and demonstration on the NSE at the Black Hat conference in Las Vegas last year. In the demonstration, Fyodor showed the results of Nmap scans against Microsoft company computers that used some of the NSE's MS RPC discovery scripts. The scripts used rpcinfo to gather info such as share names and usernames from the Windows computers. There are 177
NSE scripts available from Nmap.org as of this writing, and because they are user contributed, the list of NSE scripts is expanding at an amazing pace.
If you're a longtime user of Nmap but haven't kept up with Nmap news and releases, you'll want to check out the
Zenmap GUI's new network topology feature, which lets you create an interactive network map based on information gathered by Nmap. The map begins with localhost at the center and displays all discovered hosts in concentric rings around it, the rings indicating the number of hops away the hosts are. From there you can shift the focus to another host or get more info by clicking a host's icon in the map. The shape of the icon refers to the type of device, and the size indicates the number of open ports.
All this makes Nmap perfect for checking on IP address usage, scanning for security vulnerabilities, and ensuring your firewalls and routers are operating properly.
KeePass
Oh the passwords! How many passwords do we have for all the various servers, switches, routers, and other network gear we have to manage? And when we have to change a password, we must be sure to notify all of the other people who have access to that equipment. A good password management system can save valuable time and spare you a lot of hassle.
Enter
KeePass, an encrypted database program to store all of your usernames, passwords, access URLs, and more. You can restrict access to the KeePass database with a password, a key file, or both. The password database is encrypted with either AES or the Twofish encryption algorithm, and not as one contiguous file but in 256-bit chunks -- decrypting a single piece of data nets a cracker little or no useful data. Plus KeePass encrypts all the data in its database, not just the passwords, so your usernames, URLs, and other notes are safe as well.
You can create groups for password records to help organize the info if you have a lot of passwords to track. Groups can have subgroups, subgroups can have subgroups, and so on. A search function helps you quickly find the password record you need.
How do you share the KeePass database with coworkers who are running Mac OS X or some other version of Unix? No problem. KeePass is ported to Windows, Mac OS X, various Linuxes, and popular mobile phone platforms, including iPhone, Android, and BlackBerry. Because the KeePass database is stored in a single file, it's easy to distribute among your NOC team. Did I mention that KeePass is portable, needing no installation on Windows or Mac OS X? You can carry it with you on a USB stick or download it to a computer without leaving any unwanted registry entries or library files.
Already using another password manager? KeePass can import your existing password database in formats from a variety of programs such as Password Keeper, CodeWalletPro, and Password Agent. Other file formats are supported through KeePass plug-ins.
IPplan
We network admins must keep track of which IP addresses we have, which are in use, and which are available to be allocated to the systems administrators who always want yet another IP address for their servers. Are you tracking your IP addresses in an Excel spreadsheet? Well, stop! Get the benefits of using a real database with
IPplan.
IPplan is not a generic spreadsheet or database application. It is tailored to tracking IP addresses, so it understands and enforces CIDR blocks. Each address record has support for contact info, hardware, DNS name, location, description, MAC address, NAT address tracking, and a generic "additional information" field. You can also upload a file to attach to the IP address record.
IP address records are organized into subnets (CIDR blocks), which are assigned to customers or autonomous systems. Subnets are further organized into network areas or ranges (supernets) within the customer hierarchy. Because IPplan is designed for use by ISPs, it works well for organizing enterprise addresses for companies with multiple locations and complex networks that use multiple AS (Autonomous System) numbers. If you decide to rework the subnets on your network, no problem. IPplan handles changes easily via its split and merge subnet functions, allowing you to split and merge subnets without losing any data.
You can import your existing IP allocation data via a CSV file upload. Alternatively, you can use an XML file created by
Nmap to import addresses, or you can define the subnets yourself and have IPplan automatically fill in the DNS names for you. IPplan can import the DNS info via a zone transfer from your DNS servers.
You can set up multiple logins, so the entire network admin team has access to the IPplan database. You could even give the system administrators access to IPplan and require them to request IP addresses from you via the Web GUI.
IPplan is more than just an IP address tracking database. As you can see, you can also use IPplan to manage your DNS records, and IPplan implements "triggers" that allow you to run custom scripts based on actions taken in the Web interface.
Tcpdump and Wireshark
When things get really hairy and you can't figure out what's going wrong on your network, it's time to pull out
Tcpdump. This utility lets you capture the network traffic on a network card and view the packets and frames in real time.
If you're wondering why a browser can't find the Web server, you can fire up Tcpdump and see what's happening. Is the computer sending out DNS queries? Is it receiving a valid reply from the correct DNS server? By viewing the query and reply packets with Tcpdump, you can determine if the DNS server is replying with NXDomain for what should be a perfectly valid domain name or if the user changed the DNS server settings because he thinks that Google's DNS servers "must be faster" than your company's own servers. Or maybe the DNS queries and replies are fine, but the remote Web server is not responding. Then you would see the HTTP request packet leave the computer, but no replies from the Web server.
Tcpdump is a great tool by itself, but pair it with
Wireshark, and you have an unbeatable system for troubleshooting network application issues. You can save your Tcpdump packet captures to files and open them in
Wireshark for easier analysis. Wireshark gives you a GUI to examine Tcpdump captures and sort the data for more thorough analysis. You can compare time stamps on individual packets to see how long it's taking for a reply to be returned after a request has been made. And if you've synced the system clocks on client and server computers, you can see how long it takes for packets to travel between the two.
If you have a slow internal Web application, you can use Tcpdump and Wireshark to locate the bottleneck. If you see a long delay in the DNS lookup requests and replies but the actual HTTP requests and replies are fast, then you know the trouble lies with the DNS system or the network links to the DNS servers. If the DNS process is working normally, then you'll want to examine how long it takes for client requests to reach the server and how long it takes for the server to reply back. Wherever your network problem lies, Tcpdump and Wireshark can help you put your finger on it.
Books have been written about Tcpdump and Wireshark. Read them and learn all about these two utilities. You'll certainly improve your network troubleshooting game.
RANCID
We've all had that horrible sinking feeling in the pit of our stomachs when we've copied and pasted a new config into a router or switch and it stops responding. Did I remember to back up my old config before I uploaded the new one? How late will I be staying up tonight to fix this mess?
RANCID (Really Awesome New Cisco confIg Differ) is a versioning system for your switch and router configs. It uses either CVS or Subversion to store each new version of your configuration files. As it gathers and stores the configs for each of your devices, it runs a diff against the previous version to see what, if any, changes have been made. When it detects a change, it sends out an email with the details of that change to an address of your choosing. With RANCID, you'll know whenever a change has been made by your NOC team.
Because RANCID runs via a crontab entry, you can control how often it logs in and checks your configurations. If you are a stable shop and rarely make changes, you might have RANCID check once a day. If you are a more dynamic NOC and make changes frequently, you can set RANCID to check hourly or as often as is appropriate for your company.
One of the neat features of RANCID is that it includes a looking-glass server. You can take a quick peek at all the routes in your organization and search for any elements that are out of sorts when you suspect a routing problem on your network.
RANCID supports gear from most of the big networking vendors, including Cisco, HP ProCurve, Juniper, Foundry, and several others. It is known to work on Linux, BSDs, Mac OS X, and Solaris.
OpenNMS and Cacti
OpenNMS has a place in every enterprise. It's a
highly scalable network monitoring system that is completely open source software. A single server can monitor hundreds of thousands of network interfaces and produce nice graphs for metrics such as bandwidth usage, CPU, memory, and more.
You can set thresholds that indicate when a device is busy or down and receive a notification via email, SMS, IM, and so on. Of course you can have separate logins for each of your NOC team, and you can set up an on-call schedule so that notifications go only to on-duty team members. OpenNMS also has an escalation handler, so if the level-one NOC techs don't take care of an issue right away, an engineer or manager can be notified to oversee issue resolution.
The
Cacti graphing solution makes a good complement to OpenNMS. Although OpenNMS has the same graphing capabilities, Cacti's more intuitive Web UI allows nontechnical staff to build and manage collections of graphs that are interesting to them. For example, you could configure
Cacti to graph data from your (SNMP-capable) HVAC controllers, and your facility maintenance team members could log in to Cacti and build custom views that display only the data they need to see. If one is watching fan rotation speed and another is tracking electrical power draw, they wouldn't have to view each other's data.
You can organize Cacti's graphs into trees, similar to the old Microsoft file system viewers used to display files in a directory structure. And with individual logins for each staff member, everyone gets their own view settings saved under their login.
My TraceRoute
My TraceRoute (MTR) is not quite as useful as it once was. MTR relies on ICMP packets to judge network latency -- and ICMP are the first packets modern routers will drop in favor of more important data traffic when they get too busy. However, I still find MTR a great tool for troubleshooting network links that traverse multiple routers. Specify a destination, and MTR shows you a list of routers that your traffic passes through on the way (as well as the destination itself) and the results of a continuous ping to those routers.
MTR updates the statistics of the pings as it runs, so you can see which routers are slow to respond or which are dropping a significant number of ping requests. The results include the percentage of lost packets, the response times from each router (average, best, and worst), and the standard deviations for those times. How many times have you heard a user complaining "the Internet is slow," only to discover that the problem is a particular website or provider upstream from your office? MTR is a great way to see whether there really is a problem and to get a quick idea of where the problem resides.
One of MTR's more commonly used command-line options is -n, which stops MTR from doing reverse DNS lookups on the IP addresses of the routers it pings. This is handy when you're having DNS problems and don't want to wait for the lookups to timeout. Another useful option is -r, which issues a single summary report after running a certain number of pings (specified by the -c option) to each router. This can be used with scripts to build regular reports to be printed, emailed, or even inserted into a Web page.
PHP Weathermap
Sometimes you want the 10,000-foot overview of your network traffic.
PHP Weathermap provides exactly that. It shows a logical map of your routers and the links between them, using different colors to indicate how busy each link is. PHP Weathermap is a good complement to an application such as
Cacti or
OpenNMS.
PHP Weathermap does not handle its own data collection, so you'll need to pair it with another application such as Cacti, OpenNMS, MRTG, or RRDtool. PHP Weathermap has a plug-in that helps it integrate into Cacti, with some options available in the Cacti preference panels.
You will need to edit the map config files to adjust the appearance of your maps. You can do this by hand, though PHP Weathermap offers a GUI editor that runs within a Web browser. You can use the editor to create your nodes (routers and switches) and links, as well as produce a functioning map.
When you're ready for manual tweaking, you can add custom background images to the map and insert custom icons for your routers. You can also add subnodes, which allow you to display more information within the router's icon, such as CPU or memory usage. You can also insert parallel links or bonded links between routers.
Through further tweaking of config files, you can fine-tune the placement of router icons and the map's legend. If you have a more complicated map with plenty of router icons, you can create curved link lines between your router icons to help keep the map readable and less cluttered or cramped.
Top free open source tools for network admins: NtopNeed an sFlow or NetFlow collector to get a thorough look at your data traffic flows?
Ntop will take NetFlow or sFlow data from popular switches and routers and display it in a Web GUI, complete with clickable links that take you to details about particular hosts or protocols or to actual conversations and flows.
There are interesting features to Ntop's output, such as identifying workstation users by their email addresses and (passively) detecting the operating systems of network hosts through packet analysis. Ntop will break down traffic volume on a per-protocol basis, helpful for comparing the traffic your network actually has to what you
think it should have.
Ntop can give you a list of IP protocols, sortable by protocol type, as well as lists of traffic sources and destinations. Ntop also creates a matrix table of IP traffic, so you can see who is talking to whom and how much data is being passed between the two. Of course, data is not as useful if you cannot sort it. Ntop allows you to sort on both the traffic source and the destination.
Platforms supported include Windows and all major Unix types, such as Linux, the BSDs, Solaris, and Mac OS X. Although resource usage will vary by network size and the configuration options that you choose, Ntop should be very light on modern workstation and laptop hardware. Ntop's Web UI accepts multiple HTTP usernames and passwords, so each member of your NOC team can have individual access, and you can force Ntop to work with HTTPS.
Ntop supports a wide range of network protocol types, IP protocol types, and even network media types. There is support for several VoIP protocols, including Cisco SCCP, Asterisk's IAX protocol, and of course SIP. Ntop can even do protocol decodes on most common IP protocols. If you want the information for later retrieval and analysis, Ntop can record its network traffic data to RRD-style files on a disk.