Live Fast, Code Hard, Die Young

Posts tagged ‘OS’

Kernel high or low?

One question that I bumped into when starting in OS development is where to put the kernel in memory. You have to think about where things such as applications and data will end up so that they don’t collide with your precious kernel. After all, the kernel will be present at all times and you don’t want it to be in the way so where do you place it?

Well, the simple solution is the one that GRUB gives you which is to load it at the 1MB mark. In fact, GRUB won’t let you load anything below 1MB, probably because that area is littered with reserved areas for things such as the screen memory, BIOS data and so on. Loading the kernel at 1MB is also quite safe because PC computers of today most likely have more than 1MB of memory. (I remember my 486 had 16MB RAM and that was back in 1994 so it should be safe!)

Virtual address space
But PC’s have something called virtual memory mapping, which means that you can map virtual addresses to physical addresses. Even if the kernel is loaded at 1MB, we can "pretend" that it is placed elsewhere using this memory mapping technique. This is where the question of "high or low" comes into the picture. We can map the kernel to address zero if we want. Or we can put it high up at the 2GB mark. This is possible even if the machine does not actually have 2GB of memory, simply because we map virtual address space to physical ones.

Which solution is the best? Well, let’s say we put the kernel at address zero. Where can we put the applications then? We need to reserve some space for kernel data and book keeping records so maybe we can place programs at 512MB? This means that the kernel gets the first 512MB of address space and applications get the rest.

This would be fine if applications did not use static linking. Let me explain…

Relocation vs static linking
Static linking means that all addresses used in an application are "hard coded" to a certain base address. The application MUST be loaded to this base address or it will crash. This does not sound very smart, but it’s really convenient when you pair it with virtual addressing. All applications can get the same address range (without colliding) so it’s really easy to load them to a certain base address.

The alternative is to use a relocation table in the executable which can be used to move it to any base address. However, this comes at a performance penalty as you have to go through all address pointers and relocate them to the new base address. This technique is commonly used for dynamic libraries that are loaded by applications and may collide with other libraries in memory.

I remember that relocation tables were used on the Amiga because it did not have any virtual memory mapping (at least not the first models). In order to load many programs at the same time they had to be relocatable. It’s not that relocation is evil, but I guess static linking is used extensively for performance reasons nowadays.

The problem…and solution
So back to my question – put kernel high or low?

Well, if our kernel sits at address zero and we compile and link programs so that they run from the 512MB mark in memory we may eventually get into trouble. Why? Because we have no space for expanding the kernel. If our kernel needs to grow in size to 1GB we are in deep trouble because then all applications have to be recompiled to a new base address!

Clearly this is bad…We can either require all apps to use relocation tables and pay the penalty, or we can choose another way.

The solution that both Windows and Linux employs is to put the kernel high up where it won’t interfere with the apps. Windows puts the kernel at 2GB (or 3GB with a special registry hack). This gives applications a nice 0-2GB address space to play with without having to touch the sensitive kernel private parts. It also makes it easy to switch to a 64 bit architecture without having to recompile all programs. On a 64 bit system the kernel is simply recompiled at a new location way way high up in memory, and applications still live at address zero but have lots more space to play with.

Placing the kernel high is the way to go in my opinion. Not all hobby OS devs agree on this, but that’s the way I will go. In my next post I will explain how it is done, quite easily.

Moving to C++

I started writing my OS kernel a few weeks ago using a combination of assembler and C. This seemed to be the easiest way to get started, and it sure was. C is very simplistic and easy to compile and link. However, just recently I realized that I really miss the object oriented way of structuring the code. This gets apparent once you reach a certain size of the project where it it’s just not a few files anymore. You start thinking in terms of classes and namespaces and what not, and suddenly C is no longer that convenient. Sure, you can accomplish most object oriented stuff in C as well but it requires lots of discipline and manual bookkeeping. These things you get naturally in C++…

So today I set out to upgrade my project to use C++ instead. I will of course still use assembler for the low level functionality, but most of the kernel will be fully object oriented from now on.

Compiling C++
The problem with using C++ is that you need to take care of some additional dependencies that GCC requires. Normally this is satisfied with the standard libraries for the system, but as we effectively ARE the system we can’t rely on that. We need to implement everything on our own. No problems though, it’s not that hard.

After some fiddling and reading of guides, mainly the C++ guide in the OSDev wiki, I managed to get my new C++ kernel up and running. These are the GCC options I use for compilation:

gcc -Wall -O -fstrength-reduce -finline-functions -nostdinc -fno-builtin -fno-rtti -fno-exceptions -I./include -c main.cpp

These options tell GCC to do some optimizations (not too much though because we need the code to be safe and robust). I also disable some standard libs, turn off the run time type information and exceptions. This makes my code lean and mean, just like a kernel should be.

I also had to define a couple of functions that GCC needs for special situations:

/* Required to support pure virtual functions */
extern "C" void __cxa_pure_virtual()
{
    // This is called if a function could not be found
}
 
extern "C" void __gxx_personality_v0()
{
    // Called after an exception occured
}

The first one is called if you try to call a pure virtual function and it has not been implemented. In practice this should never happen unless there is something wrong with the compiler. In other words, you will get a compile time error, not a runtime one. But I guess GCC must be safe and not trust code blindly.

The second function is a bit of a mystery. I have specified that I don’t want any exception handling, but GCC still wants a function for unwinding the stack when an exception is thrown. I won’t be using try/catch so this should never be called either.

start.asm
My kernel is booted from a small piece of assembler code that takes care of setting up some basic structures and enable paging. I will go through it later to keep this post from getting too long.

However, what I wanted to point out is that I updated the startup code so that it initializes all static constructors and destructors as explained in the C++ bare bones article.

Linking
The last part that was a bit tricky was the linking. I’m not an expert in LD linker scripts, but I managed to put it all together and it works now. For those of you interested in how it looks, here it is in full:

OUTPUT_FORMAT("binary")
virt = 0xC0100000;
phys = 0x00100000;
SECTIONS
{
  .text virt : AT(phys) {
    codevirt = .;
    code = LOADADDR(.text);
    *(.text)
    *(.gnu.linkonce.t.*)    /* C++ classes */
    *(.rodata*)
    *(.gnu.linkonce.r.*)    /* Read only (not used?) */
    . = ALIGN(4096);
  }
 
  .data : AT(phys + (datavirt - codevirt))
  {
    datavirt = .;
    *(.data)
    
    start_ctors = .;
    *(.ctor*)
    end_ctors = .;
    
    start_dtors = .;
    *(.dtor*)
    end_dtors = .;
 
    *(.eh_frame)
    . = ALIGN(4096);
  }
 
  .bss : AT(phys + (bssvirt - codevirt))
  {
    bssvirt = .;
    bss = LOADADDR(.bss);
    *(.bss)
    *(COMMON)
    *(.gnu.linkonce.b.*)
    /*. = ALIGN(4096);*/
  }
 
  endvirt = .;
  end = LOADADDR(.bss) + (endvirt - bssvirt);
}

I will go through all this in a later post, when I explain my kernel memory layout.

Booting with GRUB

The first thing an operating system does is to boot the kernel using a boot loader. When you develop your own OS you have two options. Either you write your own boot loader or you use one of the existing ones available.

I’ve decided to use the GRUB boot loader to launch my OS kernel. Although it would definitely be an interesting task to write my own boot loader, I think it’s a bit of a time sink to build something that works on all types of PC BIOS out there and is also capable of booting several different operating systems. Someday I may do it, but not today. 🙂

Boot image
GRUB needs to be installed in the boot sector of the device you are booting from. This can be for example your hard drive, a CD or a floppy disc. Installing a custom boot sector can be a bit of a chore if you run on Windows because you need third party tools. Instead of going this route I snatched a pre-made GRUB image from the web of a FAT formatted disk with GRUB in the boot sector.

Now I could either install this image on a floppy disc but it would be slow and painful to run things from a real physical floppy drive, so I installed an excellent tool called Virtual Floppy Drive (VFD) to get a virtual drive instead. With this tool I can mount the image and read/write to it like it was a normal disc. Bochs can also boot from it without any problems.

Bochs
I have chosen to mount my image as a B: drive (as I already have an A: drive) using the following .bat file:

D:\Projects\Tools\VFD\vfd.exe install
D:\Projects\Tools\VFD\vfd.exe start
D:\Projects\Tools\VFD\vfd.exe link B
D:\Projects\Tools\VFD\vfd.exe open "D:\Projects\Current\PontOS\bootimage\PontOS.img"

(On Vista you need to run this as administrator.)

Further on you need to modify the Bochs config to boot from drive B: like this:

floppya: 1_44=b:, status=inserted
boot: floppy

Configure GRUB
GRUB has a lot of functionality and I won’t go through it all. However, the first thing to note is that it is split up in more than one part. The boot sector is only 512 bytes and just contains a small loader that loads the rest of GRUB. This is placed under \boot\grub\ on the floppy disc in the files "stage1" and "stage2". There is also a file called "menu.lst" there which contains the menu alternatives that GRUB will show when booting the machine.

title   PontOS
        root    (fd0)
        kernel  /kernel.bin

This is what "menu.lst" contains for my OS right now. The first line is the text to show for the menu alternative and then there are a couple of lines telling GRUB what to do. "root" tells it what drive to boot from, which is the first floppy drive (I have configured Bochs so that floppy 0 is the B: drive on my computer). The next command is "kernel" which makes GRUB load and launch a kernel boot image which can be found at "/kernel.bin", i.e. in the root directory of the disc.

All I have to do now is to copy my compiled kernel to the root of the floppy and launch Bochs and it will boot the OS. In fact, I’ve put the copy command in my build script so that it always puts it on the virtual drive ready for testing.

Simple printf

Today has been a busy day so not much coding has happened in PontOS. However, I implemented my own version of a printf() function for printing strings with formatting codes. This is highly useful for debugging purposes. To be able to write out numbers in decimal and hexadecimal formats I also had to implement an "integer to string" function. It made me feel really nostalgic, as it reminded me of the numerous implementations I made in assembler on Amiga and PC when I was an über nerdy teenager… 🙂

Tools for OS development

So what kind of tools do you use when writing an operating system?

Well, before I delve deeper into that discussion let me talk about where I come from tech-wise. I admit that I’m a Microsoft junkie. I’ve been using Microsoft tools and technologies since back in 1995 when I wrote my first C++ programs for Windows. Before that I used TASM/TLINK and wrote everything in x86 assembler in a primitive DOS environment and compiled it all with a good old BAT file…

I really like Visual Studio and would love to use it for OS development. After all, you get quite spoiled with the feature rich IDE, integrated debugging, intellisense and what not, but unfortunately it doesn’t cut the mustard when it comes to OS development. At least not the compiler/linker that comes with it. It is simply not suited for that task since it makes too many assumptions about what to include in the final executable. When you create your own OS there is no runtime library to rely on. You have to write all code yourself, including primitive functions such as memcpy() and strlen().

Instead I had to make a journey back in time to my old DOS days and the BAT file. This time however, instead of using TASM/TLINK, I picked out some tools based on advice from the OS dev community.

Compiler and linker
First of all you need a good compiler and linker toolset. I decided early on to develop in C since it is a high level language that still gives you raw access to low level structures and memory pointers. C++ is a viable alternative that I may investigate later on, but for now I use C.

GCC is a widely used C/C++ compiler with plenty of settings. It can produce highly optimized, tight code without unnecessary junk tying it to a specific OS.

Along with GCC I use the LD linker which takes the object files that the compiler generates and puts it all together in one executable file. The good thing about this linker is that it can create many different types of executables and even raw binary images which is very handy when starting out in kernel development.

These tools are readily available on Unix systems but what about Windows? Well, you can either use Cygwin but this seemed quite messy to me. I’m not interested in getting a Unix shell in Windows…Instead I believe the best option is to use DJGPP which is a collection of Unix tools that have been ported to Windows. This way you get both GCC and LD and some other handy utilities that work perfectly from the Windows command line. You can choose exactly which tools you need from the DJGPP toolset and installing is just a matter of unpacking the archives and updating the PATH environment variable.

I should also warn you not to use MinGW. This was the first choice I evaluated but it turned out to be a real mess with GCC and LD not supporting all the stuff I needed to compile and link correctly.

Assembler
When you write an OS kernel you will inevitably need an assembler. Some things simply cannot be done in C because you need precise control over the stack and CPU registers. It is also more convenient to write longer pieces of code in a dedicated assembler file instead of writing messy inline assembler in C which is a total pain.

The GNU toolset contains an assembler (GAS) but I don’t like its weird syntax. Instead I have chosen NASM which has a similar syntax to TASM that I used before. It is open source and works really well.

Emulator
Next up you need an emulator to run the OS in (yeah, you didn’t think I was going to keep rebooting my own system over and over again a million times while testing did you?)

There are quite a few emulators to choose among so you can just pick one that suits you. However, you should think about whether you need an emulator or a virtualization software. An emulator can emulate things that your current computer does not have, like multi core CPUs and exotic hardware, while virtualization programs often just simulate a "copy" of your current hardware (and thus it can run many many times faster as well).

I currently use the Bochs emulator because it seems to be stable and works well. I have also successfully run my kernel in Microsoft Virtual PC. I tried to use QEMU but it just crashed for me…

IDE
Now it would be really nice to suggest a great development IDE here, but the fact is that I haven’t found any good one yet. I tried out Dev-C++ but it is too tied up to MinGW for my taste. And besides it was kinda quirky when editing code because it kept screwing up the indentation. I could probably work around the MinGW dependency in Dev-C++ but I rather try to do that with Visual Studio instead, i.e. have VS run a custom make tool.

In other words, right now I’m just using Notepad++ for editing and compile all source using a makefile. Although Notepad++ is a great editor, it’s not a development IDE. I feel I definitely need to improve this work environment as the project grows bigger…

Where’s the kernel, colonel?

This rainy Sunday I’ve been spending some time taking the first baby steps for my brand new project: PontOS!

I now have an extremely simple kernel working and running in the Bochs emulator.

Here are a couple of screenshots taken from Bochs:

As you can see I’ve used GRUB as the boot loader which is common among Linux systems. I decided quite early on that it would be a wise choice since it is very flexible and supports many features. I will maybe write my own boot loader someday just for educational purposes, but for now this works just fine.

GRUB loads and launches my kernel image which performs some initialization and prints out a simple text message. It took quite a while to get this up and running but my goal for today is accomplished! Stay tuned if you want to know more about how it works.

Riding on the waves of inspiration!

A new OS is born

Have you ever dreamed of creating your own operating system? Well, I guess not…Only a true masochist would come up with an idea like that. I mean, it’s 2008 now and we already have plenty of solid operating systems to choose among. Creating a new one would be just plain stupid…

Of course I couldn’t agree more, but for me it’s the ultimate challenge. I’ve been thinking about it for many years now so why not give it a shot? I really like to learn how things work under the hood. I understand it is a huge task and I admit that I certainly never will finish it. It will probably never even reach a state where it is usable to anyone, but at least for me it will be a very interesting learning experience.

So without further ado – welcome to the world PontOS!

Silly name?
Hey, what kind of a name is that? PontOS? Sounds incredibly silly…

Yes, I agree. I took that name because my OS is just silly! It’s a silly hobby project that doesn’t really do anything good (yet). Actually, a friend of mine joked about it and said something like "if you ever make your own OS it should be called PontOS!" In fact, it’s also a name of a Greek god of the sea and it was also better than the other lame suggestion I had…Pontux!

Background
Back in the early 90’s I wrote something similar to an OS for the Amiga. It was pretty simple by today’s standards but it had a simple memory manager, a custom track loader for floppy disk access along with its own file system handler and a simple debugger with disassembling functionality. It was used for taking over the machine completely when loading my demo for maximum performance (quite important on a 7.14MHz computer…).

After my Amiga years I jumped on the PC scene bandwagon and wrote plenty of demos and intros in assembler before I learned C and C++. Although it has been over ten years since I wrote my last line of assembler code, I still remember much of it and believe that it will help me out in this project. After all, writing kernel code is not for the faint of heart…

Foreground
Ok, now focus on the future and imagine the PontOS system. My goal for the first version is a very simple kernel. This will not be a multi purpose OS. I have a vision of a system that resembles the AmigaOS, with instant app switching and solid media performance. I hate when Windows starts swapping and thrashing the hard drive even on a 4GB RAM machine just to switch between two applications. It’s just so stupid. I want to try to write a more memory efficient system. It would also be awesome if I could use it from my living room couch as a media station…Hmm, yes I know, that’s far far in the future but at least I have a vision! For now I will be happy to just get a basic kernel up and running. Let’s see where this journey takes us…

Riding on the waves of inspiration!