Ever since we started out with SoccerProject the forum has been powered by phpBB. I don't recall the version we started with, but for the last months we were using phpBB v2.0.18.

Allthough I added several modifications, we suffered more and more from spambots during the last few weeks. So we decided to migrate to the latest version : phpBB v3.0.2

Unfortunately that did not go as expected and as you know from the SoccerProject gameMessages we even lost some data in the process. So what went wrong ?

We used the default converter from phpBB, but apparently that one doesn't handle utf-8 MySQL-tables as it should. This resulted in a lot of converted forumposts to be unreadable, especially in the Czech and other foreign languages sections of our forum.

From some other websites, eg. this one, we learned what the problem was, but an exact simple problem solution was hard to find. At least we didn't find any. So we decided on the following solution :

1. Export phpBB2 database
2. Manually make sure all tables were defined as being utf-8
3. Import in phpBB3 database

Looks very simple, but none of the articles we found on the subject put it like this. This solution worked out for us, and I hope it will be useful to someone else in the future, because I know how hard it is to work without decent info..

By the way, the converter is not really user-friendly. ReesinG (he did all the work this time) told me he had to click every three minutes to keep the process going.... Just a warning for those who want to get started, it'll be a long evening ;-)

I had some problems setting it up at first. It would shutdown when I tried to activate my account. After research and testing I discovered it was because of the theme I was using on Wordpress.com. I had to switch back to the default theme, activate on the iPhone, then I could switch back to the theme I was using.

Hopefully the developers will get this fixed in the next release, but until then, that is a workaround. Hopefully this will help many of you who searched like I did.

Once running, the software is awesome and you can't beat the price, FREE!

So try it out and let me know what you think.

I googled for it and found a blog page of http://jeongsw.tistory.com/283.

And I put the following line in ~/.bash_aliases:

alias unzip="unzip -O CP949"an,

The reason that the file name were broken was Windows uses CP949 (which is similar to EUC-KR) and my Ubuntu box uses UTF-8 for encoding and decoding. (CP stands for Code Page)

The option "-O" lets the user specify the encoding to use.

몇일 전에 zip 파일을  받았는데, 압축을 푸니 파일 이름의 한글이 다깨져서 읽을 수가 없었다.

구글에서 검색해서 관련된 블로그 글(http://jeongsw.tistory.com/283)을 찾았다. 그리고 내 홈디렉토리의 .bash_aliases에 다음 줄을 집어 넣었다.

alias unzip="unzip -O CP949"

파일명이 깨진 이유는 윈도우에서는 EUC-KR과 비슷한 CP949를 사용하는데 내 우분투는 유니코드인 UTF-8을 인코딩으로 사용하기 때문이다. (CP는 코드 페이지의 약자)

-O 옵션은 사용자가 사용할 인코딩을 지정할 수 있도록 해 준다.

The objective of my effort was to display the contents of the XML record into a web page. I was surprised to notice that the UTF-8 encoding of the characters is lost in this process. The culprit turned out to be XStream! I "assumed" that any standard XML parser would support the now ubiquitous encoding UTF-8. Guess what?

The default parser used by XStream does not preserve document encoding, unless an explicit UTF-8 encoded InputStream or Reader. Although the fix was as simple as specifying the encoding for the input steam as UTF-8 (a 4-character fix!!), it was quite an annoyance to chase this down - a day before our application went live :-)

I Had a package from England arriving yesterday (Wednesday) so I called in that I'd be late, I have to admit that as much as I intended to start working at 10am, I found myself making toast and eggs, having a coffee while I answered a few (work-related) calls, and then I sort of sauntered over to the computer and sat down.

Ramp was having problems with UTF8<->16<->32 conversions so about 6pm I took a break and then knocked up my own UTF8 <-> wchar_t conversion module, nice and simple and a fair bit more efficient than the Unicode.org versions (I plan on resubmitting my changes).

I flicked the switch that enables the Chinese callsigns, and theoretically opens the door for 10 character callsigns (sorry, gamename changes will cost $13.37).

And then I decided to test it the smart way, and put together a test harness. It took several hours getting all the components to work across all 3 platforms... But finally, I had my build:

(It's not quite so blurry full sized)

Ok, so let's see what happens if...

[sourcecode language='sql']mysql> SET @id = (call wwii.f_get_player_id('kfs1')) ;
Query OK, 1 row affected (0.01 sec)
mysql> UPDATE wwii_player
        SET callsign = 'kfsone-san'
        WHERE playerid = @id ;
Query OK, 1 row affected (0.01 sec)
mysql> CALL wwii.p_add_friend(@id, "龠") ;
Query OK, 1 row affected (0.01 sec)[/sourcecode]

Well, that's just the database... Lets log in... The IRC management window catches my eye:

^BetaChat> [CRS:SYS] KFSONE-SAN has logged in

Oooh.

Yes, I'm in! So lets try some commands:

That's me trying /who, /m kfsone-san, /all, /3 blah, /crs blah and finally, /who. Ok.

Ok, so the client is still not quite sure what to do with the UTF8 characters it receives, but... It's still using the old Unicode routines that Ramp was having a hell of a time with. I can hook him up with my new, improved routines and hopefully we'll have through-and-through Unicode working in the next day or so.

Which means it's almost time for me to dive in and finish off the instancing code, although at some point I have to bite the bullet and test a login with a Chinese-character game name. But not, I suspect, after I get up again in 3 hours.

There's this thing called a Byte Order Marker (BOM) that text editors use, apparently to remind themselves of the file's UTF-8 encoding. I'm pretty sure it's useless, because UTF-8 doesn't actually have a variable byte order to keep track of, but there you go.

Basically, it's 3 bytes that the text editor inserts at the beginning of a text file, and then hides from you. It might look like a plain text file, but it's actually got 3 hidden bytes for no good reason. When you try to run it through the Ruby 1.8 interpreter, it'll see 3 invalid characters on Line 1 and throw an error right away.

This sort of error message is pretty unhelpful, especially when you appear to have nothing at all on Line 1. You might enable visible whitespace: still nothing. You might try opening it in another text editor or IDE: you will likely still not see the problem, as the only program I've tried so far that doesn't hide the BOM is NetBeans.

SciTE has two different UTF-8 encoding settings: UTF-8 and UTF-8 Cookie. In theory, the plain UTF-8 setting uses a Byte Order Marker, while UTF-8 Cookie setting doesn't. In practice, the choice doesn't seem to affect whether or not the Ruby interpreter chokes on the file, at least not with Ruby 1.8.

With 1.9 I've still had problems one or two times, but of the kind that could be fixed by closing the text editor, opening the file in NetBeans, removing the BOM, and restarting the text editor.

It's not perfect, but at least it works now, even if it's very slightly buggy.