Linux kernel TCP smoothed-RTT estimation

Recently I decided to look under the hood to see how exactly srtt is calculated in Linux. Actual (Exponentially Weighted Moving Average) srtt calculation is a rather straight-forward part but what goes in as input to that calculation under various scenarios is interesting and very important in getting correct rtt estimate.

Also useful to note the difference between Linux and FreeBSD in this regard. Linux doesn’t trust tcp packet Timestamps option provided value whenever possible as middle-boxes can meddle with it.

Basic algorithm is:
For non-retransmitted packets, use saved packet send timestamp and ack arrival time.
For retransmitted packets, use timestamp option and if that’s not enabled, rtt is not calculated for such packets.

Let’s look at the code. I am using net-next.
When a TCP sender sends packets, it has to wait for acks for those packets before throwing them away. It stores them in a queue called ‘retransmission queue’.
When sent packets get acked, tcp_clean_rtx_queue() gets called to clear those packets from the retransmission queue.

A few useful variables in that function are:
seq_rtt_us – uses first packet from ackd range
ca_rtt_us – uses last packet from ackd range (mainly used for congestion control)
sack_rtt_us – uses sacked ack
tcp_mstamp is a tcp_sock member which represents timestamp of most recent packet received/sent. It gets updated by tcp_mstamp_refresh().

For a clean ack (not sack), seq_rtt_us = ca_rtt_us (as there is no range)

If such a clean is also for a non-retransmitted packet,
[sourcecode language=”c”]seq_rtt_us = tcp_stamp_us_delta(tp->tcp_mstamp, first_ackt);[/sourcecode]

and for a sack which is again for a non-retransmitted packet,
[sourcecode language=”c”]sack_rtt_us = tcp_stamp_us_delta(tp->tcp_mstamp, sack->first_sackt);[/sourcecode]

Code that updates sack→first_sackt is in tcp_sacktag_one() where it gets populated when the sack is for a non-retransmitted packet.

tcp_stamp_us_delta() gets the difference with timestamp that the stack maintains.

Now tcp_ack_update_rtt() gets called which starts out with:
[sourcecode language=”c”]
/* Prefer RTT measured from ACK’s timing to TS-ECR. This is because
* broken middle-boxes or peers may corrupt TS-ECR fields. But
* Karn’s algorithm forbids taking RTT if some retransmitted data
* is acked (RFC6298).
*/
if (seq_rtt_us < 0)
seq_rtt_us = sack_rtt_us;
[/sourcecode]

For acks acking retransmitted packets, seq_rtt_us would be -ve.
But if there is a SACK timestamp from a non-retransmitted packet, it would use that as it carries valid and useful timestamps.

Then it takes TS-opt provided timestamps only if seq_rtt_us is -ve.
[sourcecode language=”c”]
if (seq_rtt_us < 0 && tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
flag & FLAG_ACKED) {
u32 delta = tcp_time_stamp(tp) – tp->rx_opt.rcv_tsecr;
u32 delta_us = delta * (USEC_PER_SEC / TCP_TS_HZ);

seq_rtt_us = ca_rtt_us = delta_us;
}
[/sourcecode]

By this point, there is seq_rtt_us that can be fed into tcp_rtt_estimator() that’d generate smoothed-RTT (which is more or less based on SIGCOMM 88 paper by Van Jacobson).

Bhyve setup for tcp testing

Here is how I test simple FreeBSD tcp changes with dummynet on bhyve. I’ve already wrote down how I do dummynet so I’ll focus on bhyve part.

Caution: Handbook entry on bhyve is the true source. Please refer to it for exact information. This post is super quick and may contain not-entierly-correct things. Also, I am lazy and all this config is what I am using, you may need to tweak a bit here and there.

Setup:
I’ll create 3 bhyve guests: client, router and server:

client            router            server
192.168.1.227     192.168.1.228     192.168.1.229 
10.10.10.10   10.10.10.11
                  10.10.11.11   10.10.11.10

Here, 192.* addresses are for ssh and 10.* are for guests to be able to communicate within themselves.

First, create tap interfaces needed for all bhyve guests:

client has tap0 (ssh), tap1
router has tap2 (ssh), tap3, tap4
server has tap5 (ssh), tap6

ifconfig tap0 create
ifconfig tap1 create
ifconfig tap2 create
ifconfig tap3 create
ifconfig tap4 create
ifconfig tap5 create
ifconfig tap6 create

Now create bridge interfaces for the communication.

bridge0 contains re0, tap0, tap2, tap5
bridge1 contains tap1, tap3
bridge2 contains tap4, tap6

re0 is host interface here.

ifconfig bridge0 create
ifconfig bridge0 addm re0 addm tap0 addm tap2 addm tap5
ifconfig bridge0 up
ifconfig bridge1 create
ifconfig bridge1 addm tap1 addm tap3
ifconfig bridge1 up
ifconfig bridge2 create
ifconfig bridge2 addm tap4 addm tap6
ifconfig bridge2 up

bridge0 would help connect all guests mgmt interfaces to re0 (host interface) so they all can reach out and for us to be able to ssh into them.

bridge1 connects client to router and bridge2 connects router to server.

Now, let’s create VMs.

truncate -s 10G client.img
truncate -s 10G router.img
truncate -s 10G server.img

Setup/install VMs:

sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap0 -t tap1 -d client.img -i -I iso client
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap2 -t tap3 -t tap4 -d router.img -i -I iso router
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap5 -t tap6 -d server.img -i -I iso server

Here, ‘iso’ is the path to iso image that you want to install with and last arguments – client, router,server – are VM names.

Start the VMs:

sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap0 -t tap1 -d client.img client
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap2 -t tap3 -t tap4 -d router.img router
sh /usr/share/examples/bhyve/vmrun.sh -c 2 -m 2048M -t tap5 -t tap6 -d server.img server

Stop a VM:

bhyvectl --force-poweroff --vm=

To setup networking, you’d need following in rc.conf files:

client:
ifconfig_vtnet0="inet 192.168.1.227 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
ifconfig_vtnet1="inet 10.10.10.10 netmask 255.255.255.0"
static_routes="inet1"
route_inet1="-host 10.10.11.10 10.10.10.11"

router:
ifconfig_vtnet0="inet 192.168.1.228 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
ifconfig_vtnet1="inet 10.10.10.11 netmask 255.255.255.0"
ifconfig_vtnet2="inet 10.10.11.11 netmask 255.255.255.0"

server:
ifconfig_vtnet0="inet 192.168.1.229 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
ifconfig_vtnet1="inet 10.10.11.10 netmask 255.255.255.0"
static_routes="inet1"
route_inet1="-host 10.10.10.10 10.10.11.11"

static route entries make sure routes are setup correctly for client and server to communicate with each other.

router would also need following in /etc/sysctl.conf to be able to pass traffic between client and server.

net.inet.ip.forwarding=1

Try pinging client from server or the other way around to make sure networking is working:

root@server:~ # ping 10.10.10.10
PING 10.10.10.10 (10.10.10.10): 56 data bytes
64 bytes from 10.10.10.10: icmp_seq=0 ttl=63 time=0.718 ms
64 bytes from 10.10.10.10: icmp_seq=1 ttl=63 time=0.999 ms
64 bytes from 10.10.10.10: icmp_seq=2 ttl=63 time=0.553 ms
64 bytes from 10.10.10.10: icmp_seq=3 ttl=63 time=0.553 ms
^C
--- 10.10.10.10 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.553/0.706/0.999/0.182 ms

Working networking setup on the guest looks something like this:

root@server:~ # ifconfig
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether xx:xx:xx:xx:xx:xx
        inet 192.168.1.229 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T 
        status: active
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether xx:xx:xx:xx:xx:xx
        inet 10.10.11.10 netmask 0xffffff00 broadcast 10.10.11.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T 
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo

Working networking setup on the host looks something like this:

tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26035
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26035
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26093
tap3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26093
tap4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 26093
tap5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 25977
tap6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000 ether xx:xx:xx:xx:xx:xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect
status: active
groups: tap
Opened by PID 25977
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xx:xx:xx:xx:xx:xx
inet 192.168.1.224 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=1 groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 13 priority 128 path cost 2000000
member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000000
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000000
member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 20000
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xx:xx:xx:xx:xx:xx
nd6 options=1 groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 11 priority 128 path cost 2000000
member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000000
bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xx:xx:xx:xx:xx:xx
nd6 options=1 groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 14 priority 128 path cost 2000000
member: tap4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 12 priority 128 path cost 2000000

Drop a packet

A few months back when I started looking into improving FreeBSD TCP’s response to packet loss, I looked around for traffic simulators which can do deterministic packet drop for me.

I had used dummynet(4) before so I thought of using it but the problem is that it only provided probabilistic drops. You can specify dropping 10% of the total packets, for example. I came across dpd work from CAIA, Swinburne University but it was written for FreeBSD7 and I couldn’t port it forward to FreeBSD11 with reasonable time/efforts as ipfw/dummynet has changed quite a bit.

So I decided to hack dummynet to provide me deterministic drops. Here is the patch: drop.patch
(Yes, it’s a hack and it needs polishing.)

Here is how I use it:
Setup:

client              dummynet          server
10.10.10.10  <--->  10.10.10.11
                    10.10.11.11 <---> 10.10.11.12

Both client and server need their routing tables setup correctly so that they can reach each other.

Dummynet node is the traffic shaping node here. We need to enable forwarding between interfaces:

sysctl net.inet.ip.forwarding=1

We need to setup links (called ‘pipes’) and their parameters on dummynet node like this:

# ipfw add pipe 100 ip from 10.10.11.12 to 10.10.10.10 out 
# ipfw add pipe 101 ip from 10.10.10.10 to 10.10.11.12 out
# ipfw pipe 100 config mask proto TCP src-ip 10.10.11.12 dst-ip 10.10.10.10 pls 3,4,5 plsr 7
# ipfw pipe 101 config mask proto TCP src-ip 10.10.10.10 dst-ip 10.10.11.12

‘pls 3,4,5 plsr 7’ – is the new configuration that the patch provides here.
pls : packet loss sequence
plsr : repeat frequency for the loss pattern

In the example above, it configures the pipe 100 to drop 3rd, 4th and 5th packet and repeat this pattern at every 7 packets going from server to client. So it’d also drop 10th, 11th and 12th packets and so on and so forth.

Side note: delay, bw and queue depth are other very useful parameters that you can set for the link to simulate however you want the link to behave. For example: ‘delay 5ms bw 40Mbps queue 50Kbytes’ would create a link with 10ms RTT, 40Mbps bandwidth with 50Kbytes worth of queue depth/capacity. Queue depth is usually decided based on BDP (bandwidth delay product) of the link. Dummynet drops packets once the limit is reached.

For simulations, I run a lighttpd web-server on the server which serves different sized objects and I request them via curl or wget from the client. I have tcpdump running on any/all of four interfaces involved to observe traffic and I can see specified packets getting dropped by dummynet.
sysctl net.inet.ip.dummynet.io_pkt_drop is incremented with each packet that dummynet drops.

Future work:
* Work on getting this patch committed into FreeBSD-head.
* sysctl net.inet.ip.dummynet.io_pkt_drop increments on any type of loss (which includes queue overflow and any other random error) so I am planning to add a more specific counter to show explicitly dropped packets only.
* I’ve (unsuccessfully) tried adding deterministic delay to dummynet so that we can delay specific packet(s) which can be useful in simulating link delays and also in debugging any delay-based congestion control algorithms. Turns out it’s trickier that I thought. I’d like to resume working on it as time permits.

Improving FreeBSD’s transport layer

FreeBSD network stack is quite stable but lacks some of the improvements/features available in other OSes.

A bunch of us have started an effort to try and identify current problems to improve transport layer (TCP, UDP, SCTP and others) for FreeBSD:

Transport Protocols wiki

Traditionally, freebsd-net has been the mailing list where networking problems get discussed but some have complained it to be too spammy and too focused on NIC drivers related issues. So a new mailing list has been created to specifically talk about transport level protocols: transport@

We’ve also started creating a list of TCP related RFCs and their support for FreeBSD to have a single point of reference.

Plan is to have a coordinated effort to improve TCP, UDP, etc.. so if you are interested in any of those protocols, please join the mailing list and help FreeBSD.

Build just one FreeBSD port

I usually build my own packages with poudriere but it’s not fun to do on tiny boxes so I just do ‘pkg install ‘ on them and use upstream packages. One downside is, that package is build with default options. I recently ran into a situation where I wanted to change some options for just a single port.

Now, what is the minimal set of things in /usr/ports/ that I need to checkout to be able to config/build just one port?
Turns out to be:


/usr/ports/Mk
/usr/ports/Keywords
/usr/ports/Templates
/usr/ports/ports-mgmt/dialog4ports


And the port I want to actually build. Now I can ‘make config’, change the options and build/install that port without checking out entire ports tree.

FreeBSD on PicoStation M2HP

After tplink, it was time to play with picostation m2hp.

I’ve learned a bunch of things from this experience and this entire bring up would have been impossible without the help from loos@ and adrian@

First of all, a usb-ttl serial adapter is needed to connect to the board:

Here,
black – GND
white – TX
green – RX

When you power it on, the boot up sequence looks like this:

U-Boot 1.1.4.2-s564 (Jul 19 2012 - 10:41:56)

Board: Ubiquiti Networks XM board (rev 1.0 e302)
DRAM: 32 MB
Flash: 8 MB
PCIe WLAN Module found (#1).
Net: eth0, eth1
Hit any key to stop autoboot: 0


Here is where you should top the autoboot and it will drop into uboot>.

ar7240> printenv
bootdelay=1
baudrate=115200
ethaddr=00:15:6d:0d:00:00
mtdids=nor0=ar7240-nor0
partition=nor0,0
mtddevnum=0
mtddevname=u-boot
filesize=10000
fileaddr=81000000
serverip=192.168.1.254
ethact=eth0
mtdparts=mtdparts=ar7240-nor0:256k(u-boot),64k(u-boot-env),1024k(kernel),5760k(rootfs),256k(cfg),64k(EEPROM)
bootcmd=bootm 0x9f050000
bootargs=console=tty0 root=31:03 rootfstype=squashfs init=/init
ipaddr=192.168.1.20
stdin=serial
stdout=serial
stderr=serial

Environment size: 452/65532 bytes
ar7240>


Again, mtdparts an important piece here to see how uboot expects the image layout:

mtdparts=ar7240-nor0:256k(u-boot),64k(u-boot-env),1024k(kernel),5760k(rootfs),256k(cfg),64k(EEPROM)


I picked up a working openwrt image and tried to load it.
My setup looks like this:

Black power adapter has 2 cables going to it:
o POE (yellow)- power over ethernet – which connects to the board
o LAN (green)- which connects to my working router

Laptop act as a tftp server here which is also connected to the router (via gray cable). This way laptop and board are both in the same network.
laptop has a tftp server running and I’ve assigned 192.168.1.254 to that network interface (em0 in my case) with “ifconfig alias”. This is because as you can see in uboot’s printenv, uboot expects the tftp server to be running at that address.
The board will obviously act as a tftp client.

Now, to transfer the image, after generating the image (which we will look into in a bit), copy that into /tftpboot on the server.

This is how you put board into “urescue” mode:

Board: Ubiquiti Networks XM board (rev 1.0 e302)
DRAM: 32 MB
Flash: 8 MB
PCIe WLAN Module found (#1).
Net: eth0, eth1
Hit any key to stop autoboot: 0
ar7240> ping 192.168.1.254
Using eth0 device
host 192.168.1.254 is alive
ar7240> urescue
Setting default IP 192.168.1.20
Starting TFTP server...
Using eth0 (192.168.1.20), address: 0x81000000
Waiting for connection: /


Now, board is waiting for image. Send it from the tftp server:

% tftp 192.168.1.20
tftp> bin
tftp> put PICOSTATION_M2HP.initial.img
Sent 5994346 bytes during 15.2 seconds in 11709 blocks
tftp>


board receiving the image:

Receiving file from 192.168.1.18:18401
Received 6040751 bytes
Firmware Version: XM.ar7240.FreeBSD


Alright this was the basics of how to upload a valid image onto the board. Fun part is to generate a *valid* image that board will accept.
I tried a bunch of different kenrconf’s available in the freebsd-wifi-build project but board was not accepting the generated images. After accepting the image, it used to fail like this:

Firmware check failed! (-2)

This error was coming from uboot which was proprietary and I could not find this error in any opensourced version of uboot.

General consensus about the reasons for this error was:
– bad layout of the firmware image
– wrong header (something that this uboot is not liking)

loos@ showed me a trick to look at images with hexdum. After looking at openwrt’s working image by “hexdump -c image”, I found out that uboot was expecting something along the lines of “XS2.ar7240.FreeBSD” as version string in its header. If it does not see “ar7240” in the header, it would fail the check.

So, after all that and with ray@ suggestion of using lzma’ed kernel, I could boot up the board which later failed at mounting the rootfs:

U-Boot 1.1.4.2-s594 (Dec 5 2012 - 15:23:07)

jmg@ suspected that /dev/map/rootfs.uncompress doesn’t exist for some reason as error: 19 is ENODEV and asked me to enable debugging in g_uncompress.c
I did not find anything useful after enabling debugs:

flash/spi0.uncompress: media sectorsize 512, mediasize 8388608
flash/spi0.uncompress: no CLOOP magic
map/u-boot.uncompress: media sectorsize 512, mediasize 262144
map/u-boot.uncompress: no CLOOP magic
map/u-boot-env.uncompress: media sectorsize 512, mediasize 65536
map/u-boot-env.uncompress: no CLOOP magic
map/kernel.uncompress: media sectorsize 512, mediasize 1048576
map/kernel.uncompress: no CLOOP magic
map/rootfs.uncompress: media sectorsize 512, mediasize 6684672
map/rootfs.uncompress: no CLOOP magic
map/cfg.uncompress: media sectorsize 512, mediasize 262144
map/cfg.uncompress: no CLOOP magic
map/eeprom.uncompress: media sectorsize 512, mediasize 65536
map/eeprom.uncompress: no CLOOP magic

mountroot>


Later loos@ caught that the problem was the rootfs start point in the hints file.

Specially we had to specify:
hint.map.3.start=0x130000

Because mtdparts specify 0x130000 as rootfs offset.

After this bring up part was done:

U-Boot 1.1.4.2-s594 (Dec 5 2012 - 15:23:07)

login: root
No home directory.
Logging in with home = "/".
# uname -a
FreeBSD freebsd-wifi-build 10.0-CURRENT FreeBSD 10.0-CURRENT #6 r254676M: Tue Aug 27 11:11:27 UTC 2013 [email protected]:/usr/home/hirenp/work/freebsd/head/obj/mipseb/mips.mips/usr/home/hirenp/work/freebsd/head/src/sys/AP91 mips
# df -k
Filesystem 1024-blocks Used Avail Capacity Mounted on
/dev/map/rootfs.uncompress 21851 20633 -529 103% /
devfs 1 1 0 100% /dev
/dev/md0 828 8 756 1% /tmp
/dev/md1 828 56 708 7% /var
/dev/md2 828 436 328 57% /etc
# pciconf -lv
none0@pci0:0:0:0: class=0x028000 card=0xe3020777 chip=0x002a168c rev=0x01 hdr=0x00
vendor = 'Atheros Communications Inc.'
device = 'AR928X Wireless Network Adapter (PCI-Express)'
class = network
# ifconfig
arge0: flags=8943 metric 0 mtu 1500
options=80000
ether 12:00:00:18:3c:02
media: Ethernet autoselect (100baseTX )
status: active
arge1: flags=8802 metric 0 mtu 1500
ether 12:00:00:18:3c:03
media: Ethernet 1000baseT
status: active
lo0: flags=8049 metric 0 mtu 16384
options=600003
inet 127.0.0.1 netmask 0xff000000
bridge0: flags=8843 metric 0 mtu 1500
ether d2:c4:a8:63:0e:57
inet 192.168.1.20 netmask 0xffffff00 broadcast 192.168.1.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: arge0 flags=143
ifmaxaddr 0 port 5 priority 128 path cost 200000
#


Next up was figuring out networking.

I loaded all needed modules:

# kldstat
Id Refs Address Size Name
1 22 0x80050000 2d8f10 kernel
2 1 0xc0081000 1078 if_ath_pci.ko
3 1 0xc0083000 11e4bc if_ath.ko
4 4 0xc01a2000 606fc wlan.ko
5 2 0xc0203000 4924 bridgestp.ko
6 1 0xc0208000 6d90 if_bridge.ko
7 1 0xc020f000 a5e0 random.ko
8 1 0xc021a000 3ac wlan_xauth.ko
9 1 0xc021b000 27e4 wlan_tkip.ko
#


But if loading if_ath_pci failed in:

ath0: at device 0.0 on pci0
ath0: ath_pci_attach: EEPROM firmware @ 0x8046c000
[ath]: default pwr offset: -5 dBm != EEPROM pwr offset: 0 dBm; curves will be adjusted.
ath0: ath_getchannels: unable to collect channel list from hal, status 12
device_attach: ath0 attach returned 22


Turning up debugging level:

# sysctl hw.ath.hal.debug=25600000000
hw.ath.hal.debug: 0 -> 2147483647

device_attach: ath0 attach returned 22


Error here was: isEepromValid: invalid regulatory domain/country code 0x2a
What it meant was that hal regdomain needed to be updated with country code 0x2a

Just to hack that up for time being, adrian@ suggested to set it to 0x0 which worked.


Index: dev/ath/ath_hal/ah_regdomain.c
===================================================================
--- dev/ath/ath_hal/ah_regdomain.c (revision 255320)
+++ dev/ath/ath_hal/ah_regdomain.c (working copy)
@@ -169,6 +169,10 @@
if (regDomainPairs[i].regDmnEnum == rd)
return AH_TRUE;
}
+ if (rd == 42) {
+ rd = 0;
+ return AH_TRUE;
+ }
HALDEBUG(ah, HAL_DEBUG_REGDOMAIN,
"%s: invalid regulatory domain/country code 0x%x\n", __func__, rd);
return AH_FALSE;


The kernconf and hints file are committed as r255086.

This is how you can generate the image:

$ cd ~
$ mkdir -p work/freebsd/head
$ cd work/freebsd/head
$ svn checkout svn://svn.freebsd.org/base/head src
$ svn checkout http://freebsd-wifi-build.googlecode.com/svn/trunk/ build
$ cd build/programs/ubnt-mkfwimage
$ make
$ su
# make install
# exit
$ cd ~/work/freebsd/head/src
$ ../build/build/bin/build picostation_m2hp buildworld buildkernel
$ su
# mkdir /tftpboot
# ../build/build/bin/build picostation_m2hp installworld installkernel distribution mfsroot fsimage uboot ubnt
# exit


On successful completion, resulting image PICOSTATION_M2HP.initial.img will be in /tftpboot.

My entire effort/rant has been also recorded at freebsd-embedded@

Making TL-WR1043ND do networking

This is a followup post to Installing FreeBSD on TP-Link TL-WR1043ND

I’ve learned a few very basic networking things while trying to make this work.

After having FreeBSD run on the board, next thing was to make it do networking. How Adrian’s scripts work is, it has configuration in /etc/cfg/


# pwd
/etc/cfg
# ls
hostapd.wlan0.conf manifest rc.conf
#


Here,
manifest file has list of files to be stored in flash on “cfg_save”. Basic workflow is that you make changes you want and do “cfg_save” which writes things to flash and then “reboot”. So router comes up with the saved settings.

To create a wifi network for let’s say wlan0, we need to create a hostapd file for it:

# cat hostapd.wlan0.conf
interface=wlan0
driver=bsd
ssid=STRCDR_11N
wpa=3
wpa_key_mgmt=WPA-PSK
wpa_passphrase=NotAdminORPassword
wpa_pairwise=CCMP TKIP
ctrl_interface=/var/run/hostapd


As this is a new file and we want to preserve it across reboots, we should add an entry for this file into manifest file. This is how manifest file looks:

# cat manifest
etc/cfg/manifest
etc/master.passwd
etc/group
etc/cfg/rc.conf
etc/cfg/hostapd.wlan0.conf


rc.conf is where you specify your networking configuration:

# cat rc.conf
# Set the default system hostname
system_hostname="freebsd-wifi-build"

# Create a bridge, flip on an IPv4 static address
netif_bridge0_type="bridge"
netif_bridge0_addrtype="static"
netif_bridge0_descr="default"
netif_bridge0_name="bridge0"
# These are bridge members w/ STP enabled
netif_bridge0_members_stp="arge0"
# These are bridge members w/ STP disabled
netif_bridge0_members="arge0 wlan0"
netif_bridge0_ipv4_address="192.168.1.20"
netif_bridge0_ipv4_netmask="255.255.255.0"
netif_wlan0_enable="YES"
netif_wlan0_type="wifi"
netif_wlan0_wifi_mode="hostap"
netif_wlan0_descr="default"
netif_wlan0_addrtype="none"
netif_wlan0_name="wlan0"
netif_wlan0_wifi_parent="ath0"
netif_wlan0_wifi_createargs1="country US regdomain FCC3"
netif_wlan0_wifi_createargs2="channel 1:ht/20 up"
netif_wlan0_wifi_hostapd_enable="yes"
netif_wlan0_wifi_hostapd_conf="/etc/cfg/hostapd.wlan0.conf"


A few things to notice here:
* arge0 is the ethernet interface
* wlan0 is the wifi interface
* bridge0 is the bridge interface connecting all of it together. It does the bridging of ethernet and wifi interfaces to send and receive bits (and bytes).

After making changes, to save them do:

# cfg_save
*** Storing configuration files from /etc/cfg/manifest -> /dev/map/cfg..
etc/cfg/manifest
etc/master.passwd
etc/group
etc/cfg/rc.conf
etc/cfg/hostapd.wlan0.conf
8 blocks
dd: /dev/map/cfg: end of device
0+2 records in
1+0 records out
65536 bytes transferred in 0.479322 secs (136726 bytes/sec)


And “reboot” the router. When it comes back, it _should_ have everything ready and setup which was not the case with my setup.

This is how interfaces looked:

# ifconfig
arge0: flags=8943 metric 0 mtu 1500
ether 64:70:02:cd:94:56
inet6 fe80::6670:2ff:fecd:9456%arge0 prefixlen 64 scopeid 0x6
nd6 options=21
media: Ethernet 1000baseT
status: active
arge1: flags=8802 metric 0 mtu 1500
ether 64:70:02:cd:94:57
nd6 options=21
media: Ethernet 100baseTX
status: active
ath0: flags=8843 metric 0 mtu 2290
ether 00:19:e0:66:66:68
nd6 options=21
media: IEEE 802.11 Wireless Ethernet autoselect mode 11ng
status: running
lo0: flags=8049 metric 0 mtu 16384
options=600003
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
inet 127.0.0.1 netmask 0xff000000
nd6 options=21
wlan0: flags=8943 metric 0 mtu 1500
ether 00:19:e0:66:66:68
inet6 fe80::219:e0ff:fe66:6668%wlan0 prefixlen 64 scopeid 0xa
nd6 options=21
media: IEEE 802.11 Wireless Ethernet autoselect mode 11ng
status: running
ssid STRCDR_11N channel 1 (2412 MHz 11g ht/20) bssid 00:19:e0:66:66:68
regdomain FCC3 country US ecm authmode WPA1+WPA2/802.11i
privacy MIXED deftxkey 3 TKIP 2:128-bit TKIP 3:128-bit txpower 30
scanvalid 60 protmode CTS ampdulimit 64k ampdudensity 8 shortgi wme
burst dtimperiod 1 -dfs
bridge0: flags=8843 metric 0 mtu 1500
ether 96:ce:f3:c8:09:b4
inet 192.168.1.20 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=21
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: wlan0 flags=143
ifmaxaddr 0 port 10 priority 128 path cost 33333
member: arge0 flags=143
ifmaxaddr 0 port 6 priority 128 path cost 2000000
#


As you can see everything looked sane and I could see it advertizing wifi network but when a client tries to associate, it would get stuck on “Obtaining IP address” and even if you assign static IP, client does not receive any packets.

We then looked at “etherswitchcfg” which provides ethernet connectivity for the board:

# etherswitchcfg
port0:
vlangroup: 1
media: Ethernet autoselect (100baseTX )
status: active
port1:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port2:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port3:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port4:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port5:
vlangroup: 0
media: Ethernet 1000baseT
status: active
vlangroup0:
vlan: 1
members 1,2,3,4,5
vlangroup1:
vlan: 2
members 0,5t
#


Now, as you can see the ethernet cable in WAN (blue) port shows up as port0 and it is in vlangroup1. But vlangroup1 is in vlan 2 and it is tagged. Moving ethernet cable from WAN port to one of the LAN ports fixed the problem.


# etherswitchcfg
port0:
vlangroup: 1
media: Ethernet autoselect (none)
status: no carrier
port1:
vlangroup: 0
media: Ethernet autoselect (100baseTX )
status: active
port2:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port3:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port4:
vlangroup: 0
media: Ethernet autoselect (none)
status: no carrier
port5:
vlangroup: 0
media: Ethernet 1000baseT
status: active
vlangroup0:
vlan: 1
members 1,2,3,4,5
vlangroup1:
vlan: 2
members 0,5t
#


Now a client could associate and get an IP via dhcp from the incoming ethernet connection on the router. (No dhcpd is needed/running on the router itself).

To look at connected clients:

# ifconfig wlan0 list sta
ADDR AID CHAN RATE RSSI IDLE TXSEQ RXSEQ CAPS FLAG
08:11:96:f9:b2:ec 1 1 144M 38.5 0 4012 27616 EPS AQEHTRS RSN HTCAP WME


To debug, we can look at individual ports with “tcpdump -ni “.

Verbose debugging from ath wifi stack can be obtained by:

# wlandebug -i wlan1 +assoc
net.wlan.1.debug: 0x0 => 0x800000


Just as a side-note, hostapd proc is using the configuration we provided and is running:

# ps awwux | grep hostapd
root 176 0.0 6.6 13104 2176 - Ss 6:56PM 0:00.13 hostapd -P /var/run/hostapd.wlan0.pid -B /etc/cfg/hostapd.wlan0.conf


Again a *HUGE* Thanks to Mr FreeBSD Wifi (Adrian Chadd) for all the build scripts and help with the bring-up.

Installing FreeBSD on TP-Link TL-WR1043ND

I decided to play with some wireless router and put FreeBSD on it. Adrian suggested TL-WR1043ND so I bought it.

I’ve used Adrian Chadd’s scripts do this. All the commands for flashing is also from him. A *huge* thanks to him.

I’ve also referenced OpenWRT page a lot.

Serial connection was the first thing I had to determine. Figured out serial pinouts from the OpenWRT page. Did my first soldering job:

Got usb to ttl serial cable to connect to the tx, rx and gnd pins.

After having serial connection setup, next thing was to load FreeBSD on it. (It comes with preloaded linux on it.) I grabbed image building tool from Adrian Chadd’s scripts and generated an image out of freebsd-head suitable for this board: TP-WN1043ND.factory.bin

Now, what you do is, plug usb end of the serial cable into laptop which FreeBSD presents as a cuaUN device. (Check your dmesg o/p to know value of “N”)
Connect to it via:

#cu -s 115200 -l /dev/cuaU0


Now power on the router and interrupt the boot process by typing “tpl” to drop into “uboot” prompt.
uboot looks like this:

ar7100> printenv
bootargs=console=ttyS0,115200 root=31:02 rootfstype=jffs2 init=/sbin/init mtdparts=ar9100-nor0:128k(u-boot),1024k(kernel),4096k(rootfs
),64k(art)
bootcmd=bootm 0xbf020000
bootdelay=1
baudrate=115200
ethaddr=00:1D:0F:11:22:33
ipaddr=192.168.0.2
serverip=192.168.0.5
stdin=serial
stdout=serial
stderr=serial
ethact=eth0

Environment size: 317/131068 bytes


Now starts the fleshing of firmware part. The first step below is *VERY* important…

# This erases the flash between uboot and the firmware configuration area. Whatever you do, don't mistype this!
erase 0xbf020000 +7c0000


Now, we should setup the tftp part so that we can get image we prepared from tftpserver (laptop) to tftpclient (router board). uboot is very specific about the IP addresses of both server and client. As per “printenv”, server IP *must* be 192.168.0.5 so assign that IP to the ethernet interface (via ifconfig alias). Run a network cable from laptop to one if the LAN ports of the router. (Original instructions suggest using the WAN port which did not work for me (because of which I wasted 2 frustrating weeks debugging)). Start tftpserver on laptop and copy prepared image in respective location (/tftpboot by default)

Let’s transfer/copy the image to router via tftp into RAM:

ar7100> tftpboot 0x81000000 TP-WN1043ND.factory.bin
dup 1 speed 1000
Using eth0 device
TFTP from server 192.168.0.5; our IP address is 192.168.0.2
Filename 'TP-WN1043ND.factory.bin'.
Load address: 0x81000000
Loading: T T T T T T T T #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############################
done
Bytes transferred = 8126464 (7c0000 hex)


Next step is to copy firmware image from RAM to the flash.

cp.b 0x81000000 0xbf020000 0x7c0000


Let’s boot the damn thing

bootm 0xbf020000
## Booting image at bf020000 ...
Uncompressing Kernel Image ... OK

# df -k
Filesystem 1024-blocks Used Avail Capacity Mounted on
map/rootfs.uzip 17695 16769 -489 103% /
devfs 1 1 0 100% /dev
/dev/md0 828 8 756 1% /tmp
/dev/md1 828 56 708 7% /var
/dev/md2 828 436 328 57% /etc
#


Next up is _actually_ using this thing: creating wireless network and other fun stuff.

Resize partitions and zpool on the fly

This is how my disk layout looked before the change:

# gpart show -l
=> 63 250069617 ada0 MBR (119G)
63 250069617 1 (null) [active] (119G)

# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zroot 54.5G 39.1G 15.4G 71% 1.00x ONLINE -


Above you can see only ada0s1a (55G) was being used by my zpool: zroot.
I wanted to extend it to use the ada0s1b (56G) too.

1) Set autoexpand to “on” on zroot

# zpool get autoexpand zroot
NAME PROPERTY VALUE SOURCE
zroot autoexpand off default

# zpool get autoexpand zroot
NAME PROPERTY VALUE SOURCE
zroot autoexpand on local


2) Tell geom: “I know what I am doing”

# sysctl kern.geom.debugflags=16
kern.geom.debugflags: 0 -> 16


3) Delete the unused partition

# gpart delete -i 2 ada0s1
ada0s1b deleted
# gpart show
=> 63 250069617 ada0 MBR (119G)
63 250069617 1 freebsd [active] (119G)

=> 0 250069617 ada0s1 BSD (119G)
0 115343360 1 freebsd-zfs (55G)
115343360 117440512 - free - (56G)
232783872 17285745 4 freebsd-swap (8.2G)


4) Resize the partition in use to include the unused space

# gpart resize -s 232783872 -i 1 ada0s1
ada0s1a resized

errors: No known data errors


5) Make zfs realize the fact that partition has been changed and make zpool
use the new partition which is actually the same one (ada0s1a).

# zpool online -e zroot ada0s1a ada0s1a
# zpool status
pool: zroot
state: ONLINE
scan: none requested
config:

errors: No known data errors


6) Look at the success (SIZE is now 110G)

# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zroot 110G 39.1G 71.4G 35% 1.00x ONLINE -


sshfs on FreeBSD

With sshfs you can mount remote filesystem locally via ssh. It uses FUSE underneath.
Very quick and convenient.

On server:

You need to enable sftp from /etc/ssh/sshd_config.
Uncomment following line if it’s commented:
Subsystem sftp /usr/libexec/sftp-server

And restart sshd.

On client:

Create a mountpoint:
mkdir mount_shared

sshfs user@server:/home/user/share mount_shared/

I am using 9.0 stable on server and 10.0 current on client.
“sysctl vfs.usermount=1” is not needed as suggested by others.

sshfs_debug is useful to get debugs.