Archive

Posts Tagged ‘Networking’

Using wget in interesting ways

February 17, 2016 Leave a comment

Make a web request for a web-page and all its resources in order to display correctly, but delete everything immediately after being downloaded:

wget -H -p -e robots=off --delete-after http://www.google.com 
  • -H [or] --span-hosts
  • -p [or] --page-requisites
  • -e robots=off [or] --execute
  • --delete-after

Other useful options

  • --no-dns-cache [...Turn off caching of DNS lookups]
  • --no-cache [...to disable server-side cache so as to always get the latest page]

If you want to store the pages then the -E and -K directives may be of use

wget -E -H -K -p -e robots=off http://www.google.com 
  • -E [or] --adjust-extension
  • -K [or] --backup-converted

If the web-server that you are fetching pages from blocks automated web-requests based on the user-agent, you can fool it with the following directive:

  • -U [or] --user-agent=""

If you don’t want to use the –user-agent option you can create a .wgetrc file in the home directory such that wget will always use the pre-configured user-agent

Example ./wgetrc

### Sample Wget initialization file .wgetrc by http://www.askapache.com
## Local settings (for a user to set in his $HOME/.wgetrc).  It is
## *highly* undesirable to put these settings in the global file, since
## they are potentially dangerous to "normal" users.
##
## Even when setting up your own ~/.wgetrc, you should know what you
## are doing before doing so.
header = Accept-Language: en-us,en;q=0.5
header = Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
header = Connection: keep-alive
user_agent = Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36
referer = /
robots = off</pre>

References:

Setting up DNS Tunneling with iodine on Ubuntu 14.04, 15.10

January 18, 2016 Leave a comment

So I thought setting up DNS Tunneling was as easy as getting the server software running, then getting the client software running and once the tunnel is set up we’re good to go.

Easier said than done. The tunnel is set up but the routing also needs to be set up such that traffic goes through the tunnel. Ahaaa … yes that’s where things got tricky.

Summarily, the steps that need to be done are:

Phase I: Getting the DNS server and domain configurations done

Here you’ll need a domain name and access to the authoritative name server. Alternatively you can use a free domain name service that can provide you with the capability of configuring the A resource record (assigning a domain name to an IP) and the NS resource record (assigning the name or IP of the authoritative name server).

(Afraid.org is a nice free service that allows you to make use of free public subdomains and modify the necessary DNS records for this experiment)

Phase II: Getting the Tunneling ready

  1. Set up your network with an internal client machine, a firewall that locks down the internal machine and a machine that has a (static) public address
  2. Download the tunneling software (in my case iodine)
  3. Install (Make, Make install, Make test etc)
  4. Run the server
  5. Run the client

Phase III: Getting the routing of traffic through the tunnel and the forwarding at the server side

  1. Set up forwarding of traffic received on the tunnel to go out the server’s physical interface
  2. Create a route in the routing table of the client machine that tells the machine to go out through the normal interface to the normal gateway when looking for the usual/normal DNS server (So only DNS traffic to this server has a “default route”)
  3. Remove the original Default Gateway, and replace it with with the IP of the DNS Tunneling server within the tunnel

Now let’s get straight into the mix of how to configure this …

Step-by-Step Configuration:

The network set up:

  • A Client machine within a private/internal LAN: 192.168.XX.XX
  • A pfSense firewall with 2 interfaces (one to the internal LAN, that is the gateway 192.168.XX.1) and the other on a public network with a public IP.
  • A server with a public address ZZZ.ZZZ.ZZZ.ZZZ  (lets make it 123.123.123.123 for illustrative purposes)

The diagram below illustrates the set-up.

 

Phase I: Getting the DNS server and domain configurations done

Sign up at afraid.org. Either create a new domain or make use of a publicly usable shared sub-domain. Set the A record to be the IP address of your iodine server e.g. 123.123.123.123, and set the name to be something that you’ll remember e.g. test.ignorelist.com. Set the NS record name (testns.ignorelist.com) to point to the domain name (test.ignorelist.com)

That is:

test.ignorelist.com - A - 123.123.123.123
testns.ignorelist.com - NS - test.ignorelist.com

In effect both the (authoritative) name server and the domain itself are at the same IP, though the domain name actually doesn’t seem to be used visibly (to my knowledge).

Phase II: Getting the Tunneling Ready:

Get the latest version of the iodine repository to your server machine. Assuming you’re on Ubuntu for both the client and the server, you’ll need to install git first:

sudo apt-get install git

Clone the latest version of iodine from it’s github repository into whichever directory you please. I chose to clone it to the desktop:

cd Desktop
git clone https://github.com/yarrick/iodine.git

Build, install and test the source:

NB: While doing the “make” “make install” and “make test”, it may complain that it’s missing some C header files (eg. zlib.h and check.h), so you should install ‘zlib1g-dev’ and ‘check’ packages/libraries

sudo apt-get install zlib1g-dev check

Then:

cd iodine
make
make install

Run the tests:

make test

If you’ve so far you’ve set this up on the server machine, then you need to repeat the process on the client machine so as to get the iodine package installed also on the client.

CHECK POINT: At this point you should have iodine installed on both your client and server machines.

Now we can run them so as to get the tunnel set up  (To be honest, I haven’t read deeply in into the man pages for the details of the  parameters /options available for different capabilities, so i’ll just note down what worked for me)

Run the server:

 sudo iodined -c -D 10.0.0.1 testns.ignorelist.com 

It may ask you for a password, which i  used Password01. Alternatively you could use the -P directive and assign it a password directly in the command.

 sudo iodined -c -P Password01 -D 10.0.0.1 testns.ignorelist.com 

Run the client:

sudo iodine -P Password01 testns.ignorelist.com 

At this point the tunnel should connect with the server having an IP of 10.0.0.1 and the client having an IP of 10.0.0.2. A new tun/tap interface (dns0) should also appear on both the client and the server ipconfig output. There should also be some “keep-alive” pings being logged on the server side terminal.

From the client, ping the server at 10.0.0.2:

ping 10.0.0.2 

If the ping succeeds then we’ve made some good progress in setting up the tunnel, … but there’s still more to go.

Phase III: Getting the routing of traffic through the tunnel and the forwarding at the server side

This is done in order to get all the traffic routed from the client machine through the DNS tunnel (dns0) to the server physical interface and onward to the requested resource, and that subsequent responses received on the server side physical interface come back through the tunnel server and are forwarded back through the DNS tunnel to the client.

NB: These steps are only necessary if you want to get ‘all’ traffic from the client passing through the tunnel. If that’s not your desire, then you can skip Phase III.

Setting up IP forwarding on the SERVER SIDE involves changing the ip_forward flag and setting some iptables (ip_tables) rules

 sudo bash -c 'echo 1 > /proc/sys/net/ipv4/ip_forward' 

Check that the flag has been set to 1

cat /proc/sys/net/ipv4/ip_forward 

(many times this ip_forward flag does not persist through reboots)

Set up the NAT rules in iptables to NAT traffic from the dns0 interface to the eth0 interface and vice versa:

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
sudo iptables -A FORWARD -i eth0 -o dns0 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -i dns0 -o eth0 -j ACCEPT

If you were unlucky like me, something was wrong with iptables and it could not be found despite the package being installed

 sudo modprobe iptables 

or

 sudo modprobe ip_tables 

should sort out the problem. If they don’t then you might have to check that the kernel object exists and if it doesn’t then an update of the the kernel package as follows, then running modprobe might help

 /lib/modules/'uname -r'/kernel/net/ipv4/netfilter/iptables.ko 

or

 /lib/modules/'uname -r'/kernel/net/ipv4/netfilter/ip_tables.ko 

Re-installing/updating the kernel image

sudo apt-get install linux-image-$(uname -r)

Run modprobe iptables again as seen above

Now for the tricky part (CLIENT SIDE):

Fix the routing table on the CLIENT machine such that DNS traffic goes legitimately to the local DNS server that usually does domain look-ups on your client’s behalf, out through the original default gateway

sudo route add -host [IP-Address of local DNS server] gw [original default gateway] 

e.g:

sudo route add -host 123.123.123.10 gw 192.168.1.1 

i.e. adding a route for a host (in this case, the DNS host machine), rather than a network via the normal gateway

The next 2 steps need to be done quickly (if there is a sizeable delay, for some reason the dns0 interface disappears and you won’t be able to add it as the actual default gateway)

Remove the original default gateway:

sudo route del default

Replace the default gateway with the tunnel device interface (dns0)

sudo route add default dev dns0 gw 10.0.0.1 

Read as “add default gateway as 10.0.0.1 through device dns0”

At this point you should be able to ping 10.0.0.1 from the client and get a successful response indicating that the tunnel is active and connected.

To test that HTTP traffic is being tunneled over the DNS tunnel, open up Wireshark on the client machine and start a capture on eth0. If you open any webpage e.g. http://www.google.com you should see a flurry of DNS traffic that also corresponds with the verbose output visible on the server terminal

Some gotchas that might trip you up:

Check that the dns0 interfaces are up on both client and servers and that they are pingable between the 2. Sometimes the dns0 interface on the client could disappear unnannounced and you might have to start the client side of the tunnel again and do the routing set up again. Restart the server first for good measure.

Check that the ip_forward flag on the server side is still set to 1

How to: Get Virtualbox hooked up to your Wireless LAN

April 29, 2013 Leave a comment

Totally forgot about this “How-to” …. Will update this later …

Virtual LAN over Wireless LAN: Get the Wireless Routing Going in Virtualbox

September 24, 2012 Leave a comment

Virtualbox definitely has it’s fair share of networking chaos that makes it rather difficult to quickly set up the networking options and get it right the first time. The various networking modes that it gives you are tailor made for very particular scenarios with very precise conditions that must be met – and not to mention, some very annoying exceptional circumstances to boot. It’s a big jumble and if you need a good primer to get you at least somewhere on the way, I suggest that you read this:

https://blogs.oracle.com/fatbloke/entry/networking_in_virtualbox1 [Very nice and easy read. With pics 🙂 …]

and, this:

http://www.virtualbox.org/manual/ch06.html [The manual itself – with almost all the gory details explained.]

To cut the long story short, I needed to test some security tools in a wireless environment where I need 2 virtual machines, the host machines and any other machine on the host machine (wireless) network to all be in communication at the same time, i.e, more or less on the same LAN, or at most one hop away through a single router.  I need to easily (and seamlessly) switch between communication with the virtual machines, the host itself and physical machines on the host’s LAN – and also get to the Internet. The physical LAN is actually a Wireless LAN, and well the Virtual LAN, is … well … virtual (Physical or Wireless, it’s hard to tell). Is that all too much to ask (even as an early Christmas present) from the almighty Virtualbox?

Well, it seems it’s more difficult that it actually seems. None of Virtualbox’s networking modes seems to support this scenario out of the box (No pun intended 😛 ). I’m pretty sure there are many people who are looking for this kind of thing to work: Virtual machines to talk to each other, the host and other devices on the host’s WLAN. The closest thing is using NAT with port-forwarding, but i didn’t like that solution because it’s rather limiting in terms of getting several things on several different ports to get through without having to manually configure the ports.

Something that could have been close to the ideal situation would have been if Virtualbox could support bridging with the physical host’s Wireless interface. (There are probably several other “almost” solutions, but none working yet).

The main problem it seems is that Virtualbox – on it’s own – does not support actual routing – or at least from the little testing done, it doesn’t seem to do any serious routing – whether in NAT, bridged adapter mode, host-only adapter or internal network mode. Routing between the physical LAN that the host is on

The Solution:

What worked for me was the luck that my Virtualbox environment was actually running on a Windows 7 host that supports a rather nifty tool called the Microsoft Virtual Miniport Adapter – something that seems to have hived off from a previous Virtual Router project. The idea is that it allows the Windows 7 user machine to easily act as a Wireless Router… and perhaps share the Internet connection also.

So the thing is that I shared the Internet Connection of the host machine (and thereby the connection to the Host Machine’s Wireless LAN, Wired Lan, etc) with the Microsoft Virtual Miniport Adapter (i.e Virtual Router). This creates a bridge between the physical wired/wireless LAN to this “Virtual Wireless Router.”

(Now comes the beautiful part …)

In Virtualbox, if one uses the Bridged Adapter mode and bridges the virtual machines to the respective “Microsoft Virtual Miniport Adapter”/ Virtual Router … Hey presto! Magic happens! Suddenly the virtual machines are able to route traffic towards the host itself as well as towards any other machine on the host’s network … and onwards to the Internet – if the host network is connected to the Internet.

Suddenly, the problem that Virtualbox cannot (as of  yet: Sept 2012) support bridging to the host’s Wireless Adapter, is suddenly alleviated. All the pent up tension is relieved  … absolute carthasis, and a sense of relaxation follows. It works!

🙂 A step-by-step “How-to” guide will follow in the next post. 🙂