Archive for the ‘IT’ Category

Modifying Android APKs which does self certificate checking

2016/07/13

When I modify an Android app which is not mine, I use apktool to unpack the APK, modify the smali, use apktool to pack it and finally sign it using my own key. This works for most of the Apps. However, some Apps explicitly perform certificate checking in their code. The following code snippet illustrates such checking:

PackageManager pm = myContext.getPackageManager();
PackageInfo info = pm.getPackageInfo("my.package.name", PackageManager.GET_SIGNATURES);
String expectedSig = "308203a5...";
if (!expectedSig.equals(info.signatures[0].toCharsString()) {
  quit();
}

One way to fix this is by removing the “if” block, but it’s difficult to locate all such if blocks, especially in smali. A more convenient way is to hook the PackageManager.getPackageInfo() API and change info.signatures to the original APK’s signature. The following code snippet illustrates such hooking:

PackageManager pm = myContext.getPackageManager();
PackageInfo info = PatchSignature.getPackageInfo(pm, "my.package.name", PackageManager.GET_SIGNATURES);
...
public class PatchSignature {
  public static PackageInfo getPackageInfo (PackageManager pm, String packageName, int flags) throws PackageManager.NameNotFoundException
  {
    PackageInfo info = pm.getPackageInfo(packageName, flags);
    if ("my.package.name".equals(packageName) &&
        info.signatures != null && info.signatures.length > 0 &&
        info.signatures[0] != null)
      info.signatures[0] = new Signature("308203b6...");
    return info;
  }
}

Three things need to be done here:

  1. Change all PackageManager.getPackageInfo() to PatchSignature.getPackageInfo().
  2. Add class PatchSignature.
  3. Find out the correct signature.

It is very easy to locate and replace all PackageManager.getPackageInfo() calls in the smali code. All we need to do is replacing:

invoke-virtual {PARAM1, PARAM2, PARAM3}, Landroid/content/pm/PackageManager;->getPackageInfo(Ljava/lang/String;I)Landroid/content/pm/PackageInfo;

with

invoke-static {PARAM1, PARAM2, PARAM3}, LPatchSignature;->getPackageInfo(Landroid/content/pm/PackageManager;Ljava/lang/String;I)Landroid/content/pm/PackageInfo;

The smali code for PatchSignature can be easily obtained by writing a simple app with the java code and disassemble it. Now we have only one problem remaining, how to obtain the original certificate? I.e. what should we put in new Signature("...")?

Here’s what I found the most convenient way:

openssl pkcs7 -inform DER -print_certs -in unpacked-app-using-apktool/original/META-INF/CERT.RSA | openssl x509 -inform PEM -outform DER | xxd -p

Why NUS-to-MyRepublic latency is so high?

2013/08/16

I have switched from starhub cable to MyRepublic fibre. I noticed the lag when SSH from home to NUS, so I tried to figure it out.

Here is a comparison of the round trip time (RTT) from MyRepublic to various of places I can think of. The measurement spams about one and half days. 5 ping packets is sent to all targets every 20 minutes. Sorry for the limited coloring of the plot. The legend is sorted from low latency on the top to high latency on the bottom. So singnet is the fastest and berkeley is the slowest. NUS, having an average RTT of 189ms, ranks among the slowest three.

Image

Here is a summary in the following table. I got geographic location from http://www.iplocation.net.

host geo. location RTT
http://www.singnet.com.sg Singapore 4.4
speedtest.starhub.com Singapore 5.1
http://www.sutd.edu.sg Singapore 6.4
google dns California 11.7
http://www.sina.com.cn Guangdong 42.4
http://www.tsinghua.edu.cn Beijing 81.4
http://www.fudan.edu.cn Shanghai 144.6
http://www.youku.com Beijing 159.6
http://www.nus.edu.sg Singapore 188.8
http://www.mit.edu Massachusetts 191.3
http://www.berkeley.edu California 223.5

Then I traceroute from MyRepublic to NUS.

[myrepublic]$ traceroute www.nus.edu.sg
traceroute to www.nus.edu.sg (137.132.21.27), 30 hops max, 60 byte packets
 1  192.168.0.2 (192.168.0.2)  0.212 ms  0.412 ms  0.495 ms
 2  172.24.0.1 (172.24.0.1)  2.737 ms  2.736 ms  2.910 ms
 3  192.168.255.2 (192.168.255.2)  6.822 ms  6.837 ms  6.771 ms
 4  203.117.132.9 (203.117.132.9)  4.172 ms  4.296 ms  4.154 ms
 5  anatll11.starhub.net.sg (203.118.7.34)  4.809 ms  4.861 ms  4.764 ms
 6  203.116.10.186 (203.116.10.186)  189.093 ms  188.329 ms  188.602 ms
 7  xe-5-2-0.cc-rtr1.gigapop.nus.edu.sg (202.51.240.34)  177.459 ms  178.062 ms  178.173 ms
 8  nus-gw1.gigapop.nus.edu.sg (202.51.241.14)  178.442 ms  178.512 ms  178.423 ms
 9  * * *

The route seems OK in the sense that every intermediate host are in Singapore. It’s just the RTT which isn’t right. Could the return route be the problem? Then I traceroute from NUS to MyRepublic. Note that 103.6.151.113 is in MyRepublic.

[nus]$ traceroute 103.6.151.113
traceroute to 103.6.151.113 (103.6.151.113), 30 hops max, 60 byte packets
 1  137.132.83.6 (137.132.83.6)  0.339 ms  0.370 ms  0.458 ms
 2  core-e-m.comp.nus.edu.sg (172.18.176.137)  0.315 ms  0.376 ms  0.437 ms
 3  core-j-211.comp.nus.edu.sg (172.18.176.34)  1.216 ms  1.115 ms  1.114 ms
 4  core-k-211.comp.nus.edu.sg (172.18.176.35)  1.231 ms  1.294 ms  1.142 ms
 5  187-9.priv.nus.edu.sg (172.18.187.9)  1.234 ms  1.330 ms  1.580 ms
 6  21-74.priv.nus.edu.sg (172.18.21.74)  1.763 ms  1.420 ms  1.421 ms
 7  border1-cc-l3-po110.priv.nus.edu.sg (172.18.20.102)  1.934 ms  1.985 ms  2.174 ms
 8  border-cc-m1.nus.edu.sg (137.132.3.130)  2.553 ms  3.706 ms  4.776 ms
 9  xe-5-3-0.cc-rtr1.gigapop.nus.edu.sg (202.51.241.13)  4.627 ms  4.120 ms  4.155 ms
10  ge-1-1-0.la-rtr1.gigapop.nus.edu.sg (202.51.240.193)  212.824 ms  212.802 ms  233.477 ms
11  he.net.coresite.com (206.223.143.122)  194.115 ms  214.906 ms  194.099 ms
12  10gigabitethernet2-1.core1.lax1.he.net (72.52.92.121)  192.053 ms  191.509 ms  199.569 ms
13  pacnet.10gigabitethernet2-3.core1.lax1.he.net (216.218.223.206)  191.899 ms  193.382 ms  193.344 ms
14  te0-2-0-0.gw1.lax3.asianetcom.net (61.14.158.48)  188.949 ms  190.592 ms  189.575 ms
15  te0-3-0-1.wr1.sin0.asianetcom.net (61.14.158.49)  204.286 ms  193.958 ms  197.660 ms
16  * gi4-0-0.gw2.sin3.asianetcom.net (61.14.157.134)  205.614 ms  204.824 ms
17  te0-0-0-4.gw3.sin3.asianetcom.net (202.147.32.101)  191.739 ms  191.265 ms  193.485 ms
18  MYR-0001.asianetcom.net (203.192.154.154)  227.445 ms  188.560 ms  227.428 ms
19  10GE-103-6-148-30.myrepublic.com.sg (103.6.148.30)  240.193 ms  240.195 ms  240.204 ms
20  * * *
21  103-6-151-113.myrepublic.com.sg (103.6.151.113)  192.013 ms  193.590 ms  193.314 ms
22  103-6-151-113.myrepublic.com.sg (103.6.151.113)  203.320 ms  195.278 ms  191.638 ms

Notice that in hop 10-11, it goes from NUS to coresite.com, which is in California. It then goes to asianetcom.net, which is in Tokyo. Then it comes back to Singapore. Why it routes like that is beyond my knowledge.

Besides this, another interesting question is that why google dns is so fast. The distance between Singapore and San Francisco is 13572.66 KM. The RTT of light is 90.5ms, so no one can do better than that. But google can do 11.7ms!

It turns out that 8.8.4.4 is an anycast address which means it routes to one of the google DNS servers distributed across the world. Try yourself traceroute to 8.8.4.4!

Experiment on TCP Hole Punching

2013/01/31

I recently need to find a way to connect to a subversion server behind a NAT. I used to tunnel through a SSH server with public IP. It worked perfectly, but recently I lost access to the server. So I want to try TCP hole punching.

It’s not hard to find related resource online. I followed the approach described in the paper “Peer-to-Peer Communication Across Network Address Translators”. The basic idea is to let both peers do connect and listen on the same port. If the internet gateway sees an outgoing SYN packet to X, the gateway will allow subsequent packets from X. As a result, at least one of the SYN packet should punch trough the NAT.

Before this, we need to know the external IP and port of both peers. Fortunately, most NAT implementations always map the same internal IP/port to the same external IP/port. It’s known as “independent mapping”. Even better, most NAT will use the same external port as the internal port if it’s not occupied. It’s known as “port preserving”. To know the external IP/port, we can connect to a third server and let it tell us, just like STUN.

So I implemented the idea in Ford’s paper.

#include <stdio.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <netdb.h>

#define DIE(format,...) do {perror(NULL); printf(format, ##__VA_ARGS__); exit(1);} while(0)

int say_something (int sock)
{
	char buff[256];
	int len, flags;

	flags = fcntl(sock, F_GETFL);
	flags = flags & (~ O_NONBLOCK);
	if (fcntl(sock, F_SETFL, flags))
		DIE("fcntl() failed\n");

	snprintf(buff, sizeof(buff), "Hello. I'm %d", getpid());
	printf("sending %s\n", buff);
	if (send(sock, buff, strlen(buff) + 1, 0) != strlen(buff) + 1)
		DIE("send() failed\n");

	len = recv(sock, buff, sizeof(buff), 0);
	if (len <= 0)
		DIE("recv() failed\n");
	printf("received %s\n", buff);

	return 0;
}

// TODO address type, length...
int getaddr (struct sockaddr *addr, const char *host, const char *port)
{
	struct addrinfo hints, *res;

	memset(&hints, 0, sizeof(hints));
	hints.ai_family = AF_INET;
	hints.ai_socktype = SOCK_STREAM;
	hints.ai_protocol = 0;
	hints.ai_flags = AI_PASSIVE;

	if (getaddrinfo(host, port, &hints, &res))
		return -1;

	if (res == NULL)
		return -1;

	memcpy(addr, res->ai_addr, res->ai_addrlen);
	freeaddrinfo(res);
	return 0;
}

int main (int argc, char *argv[])
{
	int ssock, csock;
	struct sockaddr_in local_addr, remote_addr;
	fd_set rfds, wfds;
	struct timeval tv;
	int i;
	socklen_t len;

	if (argc != 4) {
		printf("Usage: %s localport remotehost remoteport\n", argv[0]);
		exit(0);
	}

	if (getaddr((struct sockaddr *)&local_addr, NULL, argv[1]))
		DIE("getaddr() failed\n");
	if (getaddr((struct sockaddr *)&remote_addr, argv[2], argv[3]))
		DIE("getaddr() failed\n");

	if ((ssock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0)
		DIE("socket() failed\n");
	if ((csock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0)
		DIE("socket() failed\n");

	i = 1;
	if (setsockopt(ssock, SOL_SOCKET, SO_REUSEADDR, &i, sizeof(int)))
		DIE("setsockopt() failed\n");
	if (setsockopt(csock, SOL_SOCKET, SO_REUSEADDR, &i, sizeof(i)))
		DIE("setsockopt() failed\n");

	if (bind(ssock, (const struct sockaddr *)&local_addr, sizeof(local_addr)))
		DIE("bind() failed\n");
	if (bind(csock, (const struct sockaddr *)&local_addr, sizeof(local_addr)))
		DIE("bind() failed\n");

	if (fork()) {
		close(csock);

		if (listen(ssock, 1))
			DIE("listen() failed\n");
		while (1) {
			len = sizeof(remote_addr);
			i = accept(ssock, (struct sockaddr *)&remote_addr, &len);
			if (i < 0) {
				perror("accept() failed.");
			} else {
				printf("accept() succeed.");
				return say_something(i);
			}
		}
	} else {
		close(ssock);
		srandom(getpid());

		for (i = 0; i < 3; i ++) {
			if (connect(csock, (const struct sockaddr *)&remote_addr, sizeof(remote_addr))) {
				int sleeptime = random() * 1000000.0 / RAND_MAX + 1000000.0;
				sleeptime = sleeptime << i;
				perror("connect() failed");
				if (i < 2) {
					printf("sleeping for %.2f sec to retry\n", sleeptime / 1000000.0);
					usleep(sleeptime);
				}
			} else {
				printf("connect() succeed");
				return say_something(csock);
			}
		}
		return 1;
	}
}

It worked. host1 and host2 have external IP 1.1.1.1 and 2.2.2.2 respectively. Both NAT preserve ports so if host1 binds on port 30000, the external port is also 30000.

host1$ ./biconn 30000 2.2.2.2 20000
connect() failed: Connection timed out
sleeping for 1.13 sec to retry
connect() succeed: Connection timed out
sending Hello. I'm 8151
received Hello. I'm 6629
host2$ ./biconn 20000 1.1.1.1 30000
connect() failed: Connection refused
sleeping for 1.68 sec to retry
connect() succeed: Connection refused
sending Hello. I'm 6629
received Hello. I'm 8151

I noticed an unexpected behaviour. accept() never succeeded in either peer. connect() succeed in both peers.

Is it possible for two peers to symmetrically “connect()” to each other? Is question is not related to NAT. The answer is yes. Find any computer networks text book and look for the TCP state diagram. It’s possible to go from the SYN_SENT state to the SYN_RECV state by receiving a SYN packet. Someone has asked the question before.

So I wondered if I can remove the listen() part in the code, and use only one socket in each peer. A problem with the previous approach (as mentioned here) is that it’s not possible to bind additional sockets on the port after listen().

So I did the second experiment. It’s much cleaner.

#include <stdio.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/select.h>
#include <netinet/in.h>

void die (const char *msg)
{
	perror(msg);
	exit(1);
}

int main (int argc, char *argv[])
{
	int sock;
	struct sockaddr_in addr;
	char buff[256];

	if (argc != 4) {
		printf("Usage: %s localport remotehost remoteport\n", argv[0]);
		exit(0);
	}

	sock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
	if (sock < 0)
		die("socket() failed");

	memset(&addr, 0, sizeof(addr));
	addr.sin_family = AF_INET;
	addr.sin_addr.s_addr = htonl(INADDR_ANY);
	addr.sin_port = htons(atoi(argv[1]));
	if (bind(sock, (const struct sockaddr *)&addr, sizeof(addr)))
		die("bind() failed\n");

	memset(&addr, 0, sizeof(addr));
	addr.sin_family = AF_INET;
	addr.sin_addr.s_addr = inet_addr(argv[2]);
	addr.sin_port = htons(atoi(argv[3]));

	while (connect(sock, (const struct sockaddr *)&addr, sizeof(addr))) {
		if (errno != ETIMEDOUT) {
			perror("connect() failed. retry in 2 sec.");
			sleep(2);
		} else {
			perror("connect() failed.");
		}
	}

	snprintf(buff, sizeof(buff), "Hi, I'm %d.", getpid());
	printf("sending \"%s\"\n", buff);
	if (send(sock, buff, strlen(buff) + 1, 0) != strlen(buff) + 1)
		die("send() failed.");

	if (recv(sock, buff, sizeof(buff), 0) <= 0)
		die("recv() failed.");
	printf("received \"%s\"\n", buff);

	return 0;
}

It works. I wonder what’s the reason of doing listen(). Does it related to the way connection tracking is implemented in different type of NAT? Or does it related to the way TCP is implemented in different OS?

host1$ ./biconn1 20000 2.2.2.2 30000
connect() failed. retry in 2 sec.: Connection refused
sending "Hi, I'm 6566."
received "Hi, I'm 7600."
host2$ ./biconn1 30000 1.1.1.1 20000
connect() failed. retry in 2 sec.: Connection refused
connect() failed.: Connection timed out
connect() failed.: Connection timed out
sending "Hi, I'm 7600."
received "Hi, I'm 6566."

My objective is to connect to my subversion server in a NAT. Now, I still need a publicly accessible server to coordinate the hole punching. It basically works like this: In the subversion server I run a program with persistent connection to the public server. When I want to connect from outside, I can contact the public server, which then notifies my program in the subversion server. Then I can launch the TCP hole punching and get a TCP connection, which can then be used to tunnel the subversion connection.

Without possessing a public accessible server, other mechanisms can be used. I can think of the following mechanisms:

  • Online forum: Post the client’s external IP/port in a forum and have a program running in the subversion server to periodically check the forum.
  • DHT, e.g. the mainline bittorrent DHT: The server randomly generates a infohash, and “announce” itself to be downloading this infohash. The server then periodically queries for peers on the infohash. To do hole punching, the client also announces itself to be downloading it. The server sees a new peer joining, then both parties can do hole punching. The limitation is that two peers cannot exchange port information, thus they need to predetermine a particular port.
  • IRC bot
  • Public SIP registrar: It’s a bit overkill, but quite related, and well supported (plenty public servers and libraries).

I’m not sure if there is any existing tool for this purpose. Before IPv6 getting well established, there are going to be more and more servers behind NAT, so this is going to be handy. Please leave a comment if you know any.

Strange problem of Singapore ICA’s SAVE system

2011/06/28

I was trying to apply for visa yesterday. After I clicked the “Proceed to submit” button, I always get a blank page. Cleared cookie/cache, still same problem. I thought it’s because too many people applying visa. I tried after mid-night, still same problem. I tried school’s computer today, also same. Tried Firefox, failed at even an earlier step.

Tried my laptop and it succeed. I figured out that the reason of failure is probably adding the site in trusted sites. The web system requires turning off popup blocker for http://www.psi.gov.sg and http://www.enets.com.sg. I did that, and in addition I added both sites as trusted site. This unnecessary step turned out to cause the failure. I guess the reason is probably that when http://www.psi.gov.sg is trusted while singpass site is not, they are in different security zones, and there is problem in their handshake.

I’m a bit curious on why Firefox fails, so I checked the error console. The reason turned out to be the use of location.href(newurl), which Firefox considers as a property, not a method. If I manually types the url into the address bar, I can at least get to the singpass login page, which is one step further than the IE’s problem.

How to steal iPhone ringtone from iTunes shop?

2010/04/09

In iTunes shop, all musics and ringtones have 30 seconds preview. Ringtones are always less than 30 seconds. That means the ringtone previews are always in full length. If we can hear it, we can download it (for free, of course). This article is to share on how to download ringtones and add to your iPhone for free. It works for all ringtones in iTune Store. The main intention is to use this as an example to illustrate common practices in network hacking.

First, get a network log of iPhone’s ringtone preview traffic. To do this, setup a sniff-able wifi environment. This might be difficult for some people, but I have an existing environment. All my internet traffic goes through my linux router, so I simply run tcpdump there. I use wireshark to analyze the saved dump file.

When I play a ringtone preview, I see the this request: http://a1778.phobos.apple.com/us/r1000/031/Music/9c/22/14/mzi.wphahwgb.aac.p.m4p. (Lucky it’s not https. If it’s https, I have to try self signed certificate and see if it can pass the check.) Quickly do a direct wget. I get error 403 forbidden. First reaction is user agent. Change UA to iPhone. succeed.

$ wget -U 'Apple iPhone OS v3.1.3 CoreMedia v1.0.0.7E18' 'http://a1778.phobos.apple.com/us/r1000/031/Music/9c/22/14/mzi.wphahwgb.aac.p.m4p'
--2010-04-09 01:53:43-- http://a1778.phobos.apple.com/us/r1000/031/Music/9c/22/14/mzi.wphahwgb.aac.p.m4p
Resolving a1778.phobos.apple.com... 124.155.222.67, 124.155.222.58
Connecting to a1778.phobos.apple.com|124.155.222.67|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 493781 (482K) [text/plain]
Saving to: “mzi.wphahwgb.aac.p.m4p”

100%[=============================================================>] 493,781 256K/s in 1.9s

2010-04-09 01:53:45 (256 KB/s) - “mzi.wphahwgb.aac.p.m4p” saved [493781/493781]

Feed the m4p to a media player. It plays.

Up to here, we can already download the ringtone. Just follow previous steps and get the m4p url. It’s not very convenient though, as we have to use iphone and sniff to get the url. I want to get rid of the iphone step. I want to have a script to download a ringtone given it’s name, or have a script to download top 100 ringtone of a given genre.

For the top 100 script, I found out the url of top ringtone listing by genre to be something like http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewTop?selected-tab-index=0&top-ten-m=1%27%3B1&genreId=8004. However, the viewTop page requires sign-in. There are two ways to deal with sign-in. The hardworking way is to figure out the sign-in protocol and implement it. Usually it requires posting user id and password and get a session id. The dirty way is to sniff and get the session id, but we can only use the session before it expires. I’m not going into details about this. Here is the wget command to download the page. It’s a bit long because of those special X-apple-* headers. You can’t use it because 1. the session has expired; and 2. I have modified some of those IDs for my privacy. The page is an xml containing titles, artists, purchasing information, preview-url (the most important one for us), user ratings, etc.

wget -O - --header 'X-Apple-Store-Front: 143441-1,2' --header 'X-Apple-Partner: origin.0' --header 'X-Apple-Connection-Type: WiFi' --header 'X-Apple-Cuid: 068c5db16ca2b6956f7d582690613b68' --header 'X-Apple-Software-Cuid: 6a26ef98bfc6b1ef6f00694e61735a64' --header 'X-Dsid: 1369530585' --header 'X-Apple-Client-Application: WiFi-Music' --header 'Cookie: mz_at0=xQQUAABxlwAABABLsXbOCow79QEJcf6OqeR9C9ya+U87hxY=; mzf_in=180805; X-Dsid=1369530585; a=A2dAjgAAABtjAlRWMEsHtXFZZzAvdWlodAFxSTs5Ak1yOTjaSG9lYmtLdWcjKgsQAAdAJ5BiPTt=; Pod=18; s_cvp35b=%5B%5B%27google%253A%2520organic%27%2C%271369276708021%27%5D%2C%5B%27192.168.0.1%253A8000%27%2C%274278433477967%27%5D%5D; s_vi=[CS]v1|25C941528801054F-70001710E0178F3F[CE]; s_vnum_sg=ch%3Dip%26vn%3D1%3B; s_vnum_us=ch%3Dlegal%26vn%3D1%3Bch%3Dwebapps%26vn%3D3%3Bch%3Dip%26vn%3D2%3Bch%3Ddeveloper%26vn%3D1%3B' -U 'iTunes-iPhone/3.1.3 (2)' 'http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewTop?selected-tab-index=0&top-ten-m=1%27%3B1&genreId=8004'

There are many articles teaching how to add ringtones to iphone. I briefly describe here.

  1. Rename to .m4r
  2. Import (drag) to iTunes. It should appear under the ringtone directory in iTunes. Note: Don’t manually manage music and ringtone. Add to iTunes and sync. I tried the first way and failed miserably. I hate iTunes.
  3. You may want to change the metadata. Alternatively, before importing to iTunes, you can use opensource tools like mp4tags from libmp4v2 to change metadata. I prefer mp4tags, because it works in command line so that I can run in batch.
  4. Sync
  5. You should see the new ringtone in your iPhone.

So is it possible for apple to prevent this? I can think of a few solutions, but none of them work well.

  1. Do not provide preview. Customers won’t be happy.
  2. Add noise to preview. Shorten it to 10 seconds. Customers won’t be so happy.
  3. Use https or a custom protocol. “If we can hear it, we can download it.” It only makes hackers taking longer time. But, hey, hackers are the group of people having least money and most time.

Convert videos from iPhone using ffmpeg on Fedora 11

2010/03/11

ffmpeg in fedora 11 doesn’t buildin faac library, which encodes AAC. I need to build my own ffmpeg from source.

  1. yum install lame-devel xvidcore-devel x264-devel faad2-devel faac-devel gsm-devel dirac-devel libogg-devel libtheora-devel speex-devel libvorbis-devel openjpeg-devel liboil-devel schroedinger-devel libraw1394-devel libdc1394-devel bzip2-devel alsa-lib-devel xorg-x11-proto-devel libXau-devel libxcb-devel libXdmcp-devel libX11-devel libvdpau-devel libXext-devel libXv-devel libXvMC-devel
    Some packages are in rpmfusion. You know what you need to do.
  2. download ffmpeg source and extract. I downloaded the latest version 0.5.1.
  3. ./configure --arch=pentium4 --enable-bzlib --enable-libdc1394 --enable-libdirac --enable-libfaad --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-vdpau --enable-x11grab --enable-avfilter --enable-avfilter-lavf --enable-postproc --enable-swscale --enable-pthreads --enable-gpl --disable-stripping --cpu=pentium4 --enable-nonfree --enable-libfaac --prefix=/home/atp/install/ffmpeg-0.5.1
    I followed the configuration of ffmpeg from rpmfusion. The only changes made are:

    • --enable-nonfree --enable-libfaac
    • --prefix=/home/atp/install/ffmpeg-0.5.1(I never install my build using root.)
    • change i586 to pentium4 and removed some gcc options I don’t understand.
    • remove --disable-mmx2 --disable-sse --disable-ssse3 --disable-yasm
    • change to static build
  4. make
    make install

to be continued…

Noise problem with iTunes optimization

2010/03/10

I noticed severe image noise after I transferred my 320×480 photos to my iPhone. This is probably to do with the so called “optimization” done by iTunes.

Below is my original image:

Original Image

Original Image

Below is the “processed” image: (How did I get it? Select the image in iPhone and send email.)

Processed Image

Processed Image

Notice the added noise and slightly increased saturation. I tried to google to find out a way to disable the processing. No luck.

I tried to run process monitor on iTunes and found out the optimization is done by iTunesPhotoProcessor.exe. The processed image was saved into a .ithmb file. After a couple of hours, I couldn’t figure out a way to prevent the optimization.

Here is another attempt: Below is a comparison of the two JPEG header information:
ExifTool Version Number : 8.00
File Name : original.jpg
Directory : .
File Size : 41 kB
File Modification Date/Time : 2010:03:09 23:16:14+08:00
File Type : JPEG
MIME Type : image/jpeg
JFIF Version : 1.02
Resolution Unit : None
X Resolution : 100
Y Resolution : 100
Quality : 80%
DCT Encode Version : 100
APP14 Flags 0 : [14], Encoded with Blend=1 downsampling
APP14 Flags 1 : (none)
Color Transform : YCbCr
Image Width : 320
Image Height : 480
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:4:4 (1 1)
Image Size : 320x480

ExifTool Version Number : 8.00
File Name : processed.jpg
Directory : .
File Size : 55 kB
File Modification Date/Time : 2010:03:09 08:25:10+08:00
File Type : JPEG
MIME Type : image/jpeg
JFIF Version : 1.01
Resolution Unit : None
X Resolution : 1
Y Resolution : 1
Image Width : 320
Image Height : 480
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2)
Image Size : 320x480

The most suspicious differences are X&Y Resolution and YCbCr. Could any of these be the culprit? For example, I can generate an image with the same parameter as the processed image and hope iTunes will skip the processing. I haven’t tried this method yet…

C Container Library

2009/10/05

A container library is a data structure library for containing data. Common examples are stack, hash-table, tree and queue. Container libraries for C++ and Java are standardized by the Standard Template Library and the Java Collections Framework. However, C programs such as the Linux kernel, GTK/GLib, Apache httpd, usually implement their own modules for individual projects. There are a few generic C container libraries which I will discuss later. For some reason, none of them don’t get much attention, not to mention getting standardized. Browsing through the latest Fedora and Ubuntu packages, I don’t find any C container related library. (If you know any, please let me know by leaving a comment here.)

Before discussing individual existing c container libraries, I will give way to categorize them by memory management.

  • user-managed
    User of the library manages the container’s data structure memory. Usually the container data structure is put together with the data. The most notable example is the Linux kernel linked list.

    struct student {
       int student_id;
       /* This is the container DS. */
       struct list_head list;
    };
    
    struct list_head *pos;
    list_for_each(pos, &head) {
       /* list_entry() is just pointer arithmetic */
       struct student *stu = list_entry(pos, struct student, list);
       printf("%d\n", stu->student_id);
    }
    

    The main advantage of this type is memory efficiency, because container DS struct list_head is allocated together with data DS struct student.

  • lib-managed
    TO BE CONTINUED…
  • immortal

Why I don’t like C++

2009/06/19

I love C and Java, but I don’t like C++. C++ gives you lots of new stuff on top of C, but programming language isn’t supermarket, the more the better. Programming language shouldn’t go ad-hoc or evolution. It should go intelligent design.

C++ gives classes, inheritances, information encapsulation … lots of nice OO stuff. But on the other hand, it allows pointer manipulation. WTF! It’s like establishing a comprehensive, wonderful law, but the last rule says “You can break all the previous rules.”

C allows passing parameters by value or pointer. C++ introduces pass-by-reference which is semantically the same as pass-by-pointer but syntactically different. Being able to do the same thing in a thousand ways is not a plus for programing languages, I’d vote it to be a minus.

The only feature of C++ I like is variable declaration between statements, especially “for (int i=0 …”.