mgreene, Posted: Jul 26, 2008 4:10:57 am:
Attachment: mkmsgf.zip (Downloaded 68 times)
I needed to learn some about mkmsgf so I looked at the pascal source and came up with the
attached. I am only going to do a couple more things. Any tips or comments?
Mike
MikeG
http://www.mgreene.org/wikka
http://mikeos2.blogspot.com/
mkmsgf
Re: mkmsgf
valerius, Posted: Jul 30, 2008 10:26:41 am:
Hello, Mike!
Sorry for not responding quickly -- we were a little busy. I wrote a letter to Yuri, so, I think,
he'll will answer soon. (I did not saw him for about a week). I think, he can say more about
mkmsgf than me. As for me, your work is made very accurately, a good example to follow,
my sources are not made so thorough. So, I like such a style.
BTW, I downloaded your os2ldr sources from svn and tested with my FreeLdr and with JFS microfsd.
I tried IBM's os2ldr as well your and Pasha's os2ldr. With your os2ldr version I found a bug in my
mu_Open function, it returned wrong file size. -- I think, It was because in your loader registers
ES and DS were not equal, and in IBM's and Pasha' loaders they're equal (I assumed that mu_Open
return value is placed in es:[di], but in reality it was placed in ds:[si]). So, what is interesting,
the error appeared only with your version of os2ldr. I corrected it, so thanks for a testcase
What is also interesting -- you said you're testing os2ldr in MS VirtualPC and Bochs, but when I tried
it in Innotek VPC for OS/2, all were OK, but with Bochs (a version with compiled-in debugger from
Dmitry Froloff' site -- http://froloff.homeip.net) it does not work, because of Bochs does not support
i686 instructions, only i386 ones ;( -- Maybe, OS/2 build of Bochs is too old. And I can't test it on my
old i486 notebook But anyway, good start!
WBR,
valery
do not use windows, try doors first
Hello, Mike!
Sorry for not responding quickly -- we were a little busy. I wrote a letter to Yuri, so, I think,
he'll will answer soon. (I did not saw him for about a week). I think, he can say more about
mkmsgf than me. As for me, your work is made very accurately, a good example to follow,
my sources are not made so thorough. So, I like such a style.
BTW, I downloaded your os2ldr sources from svn and tested with my FreeLdr and with JFS microfsd.
I tried IBM's os2ldr as well your and Pasha's os2ldr. With your os2ldr version I found a bug in my
mu_Open function, it returned wrong file size. -- I think, It was because in your loader registers
ES and DS were not equal, and in IBM's and Pasha' loaders they're equal (I assumed that mu_Open
return value is placed in es:[di], but in reality it was placed in ds:[si]). So, what is interesting,
the error appeared only with your version of os2ldr. I corrected it, so thanks for a testcase
What is also interesting -- you said you're testing os2ldr in MS VirtualPC and Bochs, but when I tried
it in Innotek VPC for OS/2, all were OK, but with Bochs (a version with compiled-in debugger from
Dmitry Froloff' site -- http://froloff.homeip.net) it does not work, because of Bochs does not support
i686 instructions, only i386 ones ;( -- Maybe, OS/2 build of Bochs is too old. And I can't test it on my
old i486 notebook But anyway, good start!
WBR,
valery
do not use windows, try doors first
Re: mkmsgf
mgreene, Posted: Jul 30, 2008 11:50:44 pm:
No problem, it has been a busy week here too. I put a final version on hobbes
with a correction that Doug Bissett found.
My style is good? Thanks, but I got beat up about it when I submitted stuff to the
Open Watcom group. Michal would crap on me so I learned to follow the style
Guide (http://www.openwatcom.org/index.php/style_Guide) the best I could
and since I am so forgetful I comment the hell out of everything.
Hmmm… the wrong file size. Good for me! I thought I was loosing my mind because
it would come up different at times. I did not know if it was my problem, the uFSD,
or version of OW I was using. I keep a copy of OW 1.7.1 Stable installed but usually
check Peter’s build server once a week and download a 1.8 version without errors
(http://www.openwatcom.org/index.php/Build_Servers).
So, how I am testing stuff:
I have a system that boots eCS, Gentoo, and XP. Under XP I use MS VirtualPC 2007
since they give it away free.
1. On the XP box I am able to make and install eCS RC4 to a VirtualPC drive then install
the extras for accessing a directory on the XP box.
2. Next, I boot with the eCS CDROM and make a VirtualPC drive, format it bootableJFS,
and sysinstx it. (this I call the work drive) I make a profile to boot this drive.
3. I boot the full VirtualPC eCS profile (from #1) and make the work drive the 2nd drive
and set a XP directory to be mounted as a network drive.
So, profile 1 looks like this when booted:
eCS Install (drive 1) ======> Work Drive (drive 2)
|
------- Z: (xp directory mounted as VirtualPC drive)
If I compile on XP I copy loader to Z:, start profile 1 and copy to Work Drive, save state
of running eCS profile, and then boot profile 2 which boots with my loader. Because I save
state of the VirtualPC eCS install, I can compile and test the loader in about 2 minutes. I
don’t like XP, but this makes things very fast.
Boch I run under linux because I found an old 100meg hard drive I could image fast. This is
very good for tracing as you know. I use Bochs 2.3.6 that I compiled on Gentoo. I asked Dmitry
on IRC if he could update OS/2 Boch but he did not seem excited about doing it.
I think if I can get an install of VirtualBox on OS/2 to work like VirtualPC and an updated Boch
all work could be done under OS/2 which I would much rather use.
MikeG
http://www.mgreene.org/wikka
http://mikeos2.blogspot.com/
No problem, it has been a busy week here too. I put a final version on hobbes
with a correction that Doug Bissett found.
My style is good? Thanks, but I got beat up about it when I submitted stuff to the
Open Watcom group. Michal would crap on me so I learned to follow the style
Guide (http://www.openwatcom.org/index.php/style_Guide) the best I could
and since I am so forgetful I comment the hell out of everything.
Hmmm… the wrong file size. Good for me! I thought I was loosing my mind because
it would come up different at times. I did not know if it was my problem, the uFSD,
or version of OW I was using. I keep a copy of OW 1.7.1 Stable installed but usually
check Peter’s build server once a week and download a 1.8 version without errors
(http://www.openwatcom.org/index.php/Build_Servers).
So, how I am testing stuff:
I have a system that boots eCS, Gentoo, and XP. Under XP I use MS VirtualPC 2007
since they give it away free.
1. On the XP box I am able to make and install eCS RC4 to a VirtualPC drive then install
the extras for accessing a directory on the XP box.
2. Next, I boot with the eCS CDROM and make a VirtualPC drive, format it bootableJFS,
and sysinstx it. (this I call the work drive) I make a profile to boot this drive.
3. I boot the full VirtualPC eCS profile (from #1) and make the work drive the 2nd drive
and set a XP directory to be mounted as a network drive.
So, profile 1 looks like this when booted:
eCS Install (drive 1) ======> Work Drive (drive 2)
|
------- Z: (xp directory mounted as VirtualPC drive)
If I compile on XP I copy loader to Z:, start profile 1 and copy to Work Drive, save state
of running eCS profile, and then boot profile 2 which boots with my loader. Because I save
state of the VirtualPC eCS install, I can compile and test the loader in about 2 minutes. I
don’t like XP, but this makes things very fast.
Boch I run under linux because I found an old 100meg hard drive I could image fast. This is
very good for tracing as you know. I use Bochs 2.3.6 that I compiled on Gentoo. I asked Dmitry
on IRC if he could update OS/2 Boch but he did not seem excited about doing it.
I think if I can get an install of VirtualBox on OS/2 to work like VirtualPC and an updated Boch
all work could be done under OS/2 which I would much rather use.
MikeG
http://www.mgreene.org/wikka
http://mikeos2.blogspot.com/
Re: mkmsgf
valerius, Posted: Jul 31, 2008 4:06:19 am:
>My style is good? Thanks, but I got beat up about it when I submitted stuff to
the Open Watcom group. Michal >would crap on me so I learned to follow the style
Guide (http://www.openwatcom.org/index.php/style_Guide) >the best I could and
since I am so forgetful I comment the hell out of everything.
The style itself is good, but it can interfere with other projects' style.
According testing cases -- I use bochs for testing/debugging -- I have four emulators under
OS/2 -- bochs and qemu from Dmitry Froloff site (qemu is good for debugging to commport
without debug terminal attached -- it has serial console and even parallel console accessed
by pressing Ctrl-Alt-2[-3]. I could, for example, test Merlin kernel with its kernel debugger,
and switching between serial console and main screen is very easy. -- It is yet unofficial and
not released yet, but if you interested, I can give you a link (and, I myself tried to port qemu
to OS/2, I compiled it, but it is non-working now, I plan to finish that port because Froloff
included only qemu-system-x86 and not other emulators and utilities, and Froloff's version
didn't updated for a long time)) and VPC and VBOX. VPC is very fast. VBOX is also very fast,
it inherits qemu functionality. But what is bad, it don't support raw inages, only compressed ones.
With bochs and qemu (and even, with VPC, but not with vbox ;() I can use raw images. It is useful
for quick testing. You can make raw HDD image with partitions on it -- boot with needed OS and
fdisk and format. After that you can exchange files on host system and VM very quickly, using
mtools (if the drive is formatted in FAT, of course). -- mcopy command can be included in
script or makefile, so loader can be built and copied to HDD image at once. With JFS and
HPFS it is not so fast because you must boot into eCS and copy manually. But in our old versions
of loader we used ext2fs. -- We decided to use ext2_os2.ifs and its uFSD because it was the only
uFSD available in sources. This uFSD is somewhat big to fit into ext2 bootblock -- ext2 bootblock
takes only 2 sectors (1024 bytes) and ext2 uFSD was 15 KBytes long. So, Matthieu Willm, the
author of ext2_os2, decided to use LILO linux loader for starting uFSD instead of linux kernel (and
because it was long time ago, the current LILO versions don't work, and LILO and ext2 must be
taken from very old version of Linux, like RedHat 5.0 and Slackware 3.0 (I have them)). In the
case of ext2 we could not use mtools (as mtools are for FAT only), but there are ltools (utilities
for accessing ext2 and (readonly) reiserfs from DOS). I started to port them, but again not
finished ;( My attempts are yet in osfree\tools\ltools directory in osFree sources.
And other posibility for fast exchanging files with ext2 HDD image: I started small Linux
installation from 10 Mb HDD image, along with test ext2 HDDimage; and in bash profile I wrote
a command for mounting, copying and unmounting ext2 test partition. -- Actually, I used 3 HDD
images -- first one with Linux, 2nd one with test ext2 partition, and 3rd one with FAT filesystem.
I built a loader, copied it automatically to FAT image, then qemu was started with 3 mentioned
images. The ext2 image was mounted automatically, then loader was copied to ext2 partition
from the FAT one, then ext2 partition was unmounted, then Linux was shut down. And after a
reboot without Linux a loader was ready for testing. The testing is very convenient with bochs.
-- I used bochs for debvugging with breakpoints, tracing etc. -- Even bootsectors were debugged
such way.
And also, I made a simple REXX script genhdd.cmd (in osfreetoolsbin directory) which can
generate a full HDD image with several primary partitions from partition images. The MBR PT
fields were generated automatically. The partition image with ext2 filesystem was generated
with genext2fs program, which we ported from UNIX. So, the goal was to automate generation
of full HDD image from a set of files (like mkisofs does) and automatically write MBR and
bootsector to point to uFSD, which must load a loader. But again, we did not finished it. It was
assumed that ext2fs uFSD is non-fragmented. And if it was so, all may be working. But in reality
it didn't, because uFSD _was_ fragmented. So, it almost worked, with exception of a
fragmented uFSD support.
And the last time, when we switched to GRUB filesystem support, I decided to use uFSD
which is based on GRUB ext2 driver, it was more compact than the one from ext2_os2. And
I decided no more use ext2 because GRUB contains iso9660 (cdfs) driver, and bootable iso
image is much simpler to generate.
So, now I compile the loader first, and second, I launch mkiso.cmd script, which launches
mkisofs with proper parameters, and the ISO image is copied to images directory. All can be
done in one click/keypress
I can suggest you to use my method of testing -- no need in booting into eCS etc.
WBR,
valery
do not use windows, try doors first
>My style is good? Thanks, but I got beat up about it when I submitted stuff to
the Open Watcom group. Michal >would crap on me so I learned to follow the style
Guide (http://www.openwatcom.org/index.php/style_Guide) >the best I could and
since I am so forgetful I comment the hell out of everything.
The style itself is good, but it can interfere with other projects' style.
According testing cases -- I use bochs for testing/debugging -- I have four emulators under
OS/2 -- bochs and qemu from Dmitry Froloff site (qemu is good for debugging to commport
without debug terminal attached -- it has serial console and even parallel console accessed
by pressing Ctrl-Alt-2[-3]. I could, for example, test Merlin kernel with its kernel debugger,
and switching between serial console and main screen is very easy. -- It is yet unofficial and
not released yet, but if you interested, I can give you a link (and, I myself tried to port qemu
to OS/2, I compiled it, but it is non-working now, I plan to finish that port because Froloff
included only qemu-system-x86 and not other emulators and utilities, and Froloff's version
didn't updated for a long time)) and VPC and VBOX. VPC is very fast. VBOX is also very fast,
it inherits qemu functionality. But what is bad, it don't support raw inages, only compressed ones.
With bochs and qemu (and even, with VPC, but not with vbox ;() I can use raw images. It is useful
for quick testing. You can make raw HDD image with partitions on it -- boot with needed OS and
fdisk and format. After that you can exchange files on host system and VM very quickly, using
mtools (if the drive is formatted in FAT, of course). -- mcopy command can be included in
script or makefile, so loader can be built and copied to HDD image at once. With JFS and
HPFS it is not so fast because you must boot into eCS and copy manually. But in our old versions
of loader we used ext2fs. -- We decided to use ext2_os2.ifs and its uFSD because it was the only
uFSD available in sources. This uFSD is somewhat big to fit into ext2 bootblock -- ext2 bootblock
takes only 2 sectors (1024 bytes) and ext2 uFSD was 15 KBytes long. So, Matthieu Willm, the
author of ext2_os2, decided to use LILO linux loader for starting uFSD instead of linux kernel (and
because it was long time ago, the current LILO versions don't work, and LILO and ext2 must be
taken from very old version of Linux, like RedHat 5.0 and Slackware 3.0 (I have them)). In the
case of ext2 we could not use mtools (as mtools are for FAT only), but there are ltools (utilities
for accessing ext2 and (readonly) reiserfs from DOS). I started to port them, but again not
finished ;( My attempts are yet in osfree\tools\ltools directory in osFree sources.
And other posibility for fast exchanging files with ext2 HDD image: I started small Linux
installation from 10 Mb HDD image, along with test ext2 HDDimage; and in bash profile I wrote
a command for mounting, copying and unmounting ext2 test partition. -- Actually, I used 3 HDD
images -- first one with Linux, 2nd one with test ext2 partition, and 3rd one with FAT filesystem.
I built a loader, copied it automatically to FAT image, then qemu was started with 3 mentioned
images. The ext2 image was mounted automatically, then loader was copied to ext2 partition
from the FAT one, then ext2 partition was unmounted, then Linux was shut down. And after a
reboot without Linux a loader was ready for testing. The testing is very convenient with bochs.
-- I used bochs for debvugging with breakpoints, tracing etc. -- Even bootsectors were debugged
such way.
And also, I made a simple REXX script genhdd.cmd (in osfreetoolsbin directory) which can
generate a full HDD image with several primary partitions from partition images. The MBR PT
fields were generated automatically. The partition image with ext2 filesystem was generated
with genext2fs program, which we ported from UNIX. So, the goal was to automate generation
of full HDD image from a set of files (like mkisofs does) and automatically write MBR and
bootsector to point to uFSD, which must load a loader. But again, we did not finished it. It was
assumed that ext2fs uFSD is non-fragmented. And if it was so, all may be working. But in reality
it didn't, because uFSD _was_ fragmented. So, it almost worked, with exception of a
fragmented uFSD support.
And the last time, when we switched to GRUB filesystem support, I decided to use uFSD
which is based on GRUB ext2 driver, it was more compact than the one from ext2_os2. And
I decided no more use ext2 because GRUB contains iso9660 (cdfs) driver, and bootable iso
image is much simpler to generate.
So, now I compile the loader first, and second, I launch mkiso.cmd script, which launches
mkisofs with proper parameters, and the ISO image is copied to images directory. All can be
done in one click/keypress
I can suggest you to use my method of testing -- no need in booting into eCS etc.
WBR,
valery
do not use windows, try doors first
Re: mkmsgf
prokushev, Posted: Aug 05, 2008 11:32:41 am:
Hello.
Looked into mkmsgf. First question is header structure:
Starting from pointed line some problems starts. According to different sources and message
files dumps can be country info or pointer to country info. Size or byte (as in your structure)
or word. In first case it is SBCS/DBCS flag in second - pointer to countryinfo. Next country info
structure (16 bit in your case, 32bit in my case) also is a question. In one files it is codepage
number in another - it is pointer to MultipageBlock (according to my sources).
So, most actual message header structure (only end of structure) is:
During msg file generation last structure (as in my sources) can be easely used. For reading (or
message decoding) some more solid logic required to detect which structure really used. Your
structure is older and used only for single-language message file (IIRC). For multi-language files
(many DBCS systems uses multi-language files) last structure must be used. Also standard oso001h.msg
file (At least in Russian OS/2) as in my sources.
Also, dword indexes required for long message files like oso001h.msg. And support for response files
also required.
But in general clone is fine.
wbr,
Yuri
Hello.
Looked into mkmsgf. First question is header structure:
Code: Select all
typedef struct _MSGHEADER {
uint8_t magic[8];
uint8_t identifier[3];
uint16_t msgnumber;
uint16_t firstmsgnumber;
int8_t offset16bit;
uint16_t version;
uint16_t indextaboffset;
int8_t countryinfo; <- here problems starts
int16_t nextcoutryinfo;
uint8_t reserved[8];
} MSGHEADER, *PMSGHEADER;
files dumps can be country info or pointer to country info. Size or byte (as in your structure)
or word. In first case it is SBCS/DBCS flag in second - pointer to countryinfo. Next country info
structure (16 bit in your case, 32bit in my case) also is a question. In one files it is codepage
number in another - it is pointer to MultipageBlock (according to my sources).
So, most actual message header structure (only end of structure) is:
Code: Select all
int16_t countryinfo;
int32_t nextcoutryinfo;
uint8_t reserved[5];
message decoding) some more solid logic required to detect which structure really used. Your
structure is older and used only for single-language message file (IIRC). For multi-language files
(many DBCS systems uses multi-language files) last structure must be used. Also standard oso001h.msg
file (At least in Russian OS/2) as in my sources.
Also, dword indexes required for long message files like oso001h.msg. And support for response files
also required.
But in general clone is fine.
wbr,
Yuri
Re: mkmsgf
mgreene, Posted: Aug 17, 2008 2:35:38 am:
Ok, I did find different versions of the structure but your point makes sense. I just
wanted something simple in an attempt to learn the format and spent more time on
it than I wanted to. I no nothing about DBCS so that wasn't and issue for me.
MikeG
http://www.mgreene.org/wikka
http://mikeos2.blogspot.com/
Ok, I did find different versions of the structure but your point makes sense. I just
wanted something simple in an attempt to learn the format and spent more time on
it than I wanted to. I no nothing about DBCS so that wasn't and issue for me.
MikeG
http://www.mgreene.org/wikka
http://mikeos2.blogspot.com/
Re: mkmsgf
prokushev, Posted: Aug 17, 2008 11:23:10 am:
Well. This question not DBCS related, but mulyilanguage related.
wbr,
Yuri
Well. This question not DBCS related, but mulyilanguage related.
wbr,
Yuri