Edward Peterlin
May 21, 2002
The events leading up to our release at the WWDC were an exciting time for myself and for the rest of the OS X porting team. After about a year of effort by many different members of the open source community, we reached our first grand achievement, a build of OpenOffice.org that could launch and run! We finished just in time to unleash our build upon the world at Apple's Worldwide Developer's Conference and try our best to recruit more developers to join the cause. I've done many crazy things, but this one seemed like it would be one of the crazier right from the start...here's the story as I remember...
At the end of February 2002, I had been busy working on the source code for OpenOffice.org for at least half a year. The progress we were making seemed mediocre at best; fixing one build failure simply quickly led to uncovering the next. I had been working on trying to get the Mozilla integration working for a month, but decided that things hadn't been building for too long. I decided to just start hacking away at the remaining modules. Although we had just over ninety percent of the source code building by this point, some of the problems seemed like they couldn't be solved. Even worse, we still had nothing to show for all of our effort except a percentage and some error messages in a Terminal, not exactly comforting things for either a developer or an end-user.
Then, right at the end of the month, I got a message from one of our community developers. It contained a patch. It contained instructions too. And, more impressively, it would allow OpenOffice.org to have a successful build. Naturally, I was intrigued. More than just a little. With my TiBook utterly out of hard drive space, I reluctantly blew away the source code I had been working on for the last six months and started anew trying to apply the patch.
After some thought, I decided that it was necessary to try and replicate the build process and make a correct set of instructions to help the other developers out. The most useless thing in the world would be to have a repeat of what had existed before I began updating the website...a set of build instructions full of errors.
Having wasted a month myself when I got onto the project attempting to just fix problems with the build system on 10.1, this was going to be a crucial element in allowing our other porting members to get up to speed. The successful build had come out of the X11 effort and not the Quartz effort, requiring the instructions to specify (and myself to learn!) how to get X11 and all the supporting programs installed on an OS X box. I let people on the dev list know, alebit cryptically, that there were hopefully going to be build changes to come once we can get a reproducible build process and work out the licensing issues necessary to get the changes into our repository. I go underground to try and get this all in place and verify the patch. After about two weeks of work and beer, it happened. I saw OpenOffice.org launch for the first time on OS X.
It was surreal. I was the second person to see the culmination of the efforts of all of these people who had been sadistic enough to spend nights and weekends staring at nothing but text flying by in a terminal. We had finally gotten tangible results. I played with it a bit more, and surprisingly quite a bit of it was functional. That's probably when I realized that we might be able to get something to a state where we could make our first binary installer. And we might even be able to pull it together in a few weeks...in time for WWDC.
The build still had a lot of problems with it. The word processing module refused to start at all. The build instructions still had errors in them. With such a great end-user interest in this port, it seemed premature to claim victory publicly, so I assembled the build instructions and sent them off to the core members of the OS X porting team to make sure they too could reproduce the build and to start our grand push for reaching the first moderately functional binary!
Once I had decided that we could push for WWDC, the other elements of the community started to step up and help us out. Already overloaded with the impending 1.0 release on the first tier platforms, I'm sure this project was the last thing they needed.
I was determined that we needed to at least be able to launch all four major components before we released a first binary build, so I started trying to debug the swriter problem. One problem...my TiBook didn't feel like showing up. The debugger sucked away over a gig of memory and started going to the hard drive to find swap space. I removed everything on my computer except the operating system, development tools, the debugger, OpenOffice.org, and the array of supporting programs. My experience fixing the Quartz VCL in the previous year was helpful, but this was probably the largest program I'd ever attempted to debug. The TiBook took over 30 minutes to load the symbols the first time, about 3 minutes to step over a line of code. It appeared it was doing magical pointer voodoo, so naturally I would step over many lines to try and find that one location that mucked the pointer. So I would drink while waiting. And I was waiting a lot. The local recycling company must have had a field day.
In the meantime, I was helping the other developers get up to speed on the build since most of us were all focusing on the Quartz build. Of course, there are other problems with the build that the other guys found aside from one of the components failing to launch. The program seemed to be spamming us with memory errors, but yet it still ran. Bizarre. Two guys started to track this down. The other guys were working on getting it to build. Of course, they found the errors in the build instructions. By the end of the first two weeks of April the build instructions were clean. And probably twice as complicated as they really needed to be.
I again spent the entire week trying to debug swriter and sucking down booze. It was at the point where I could locate where it crashes, step over the next line of code, and then watch OS X log me out as it runs out of swap space. This happened all the time, and after about 40 minutes of waiting. I just couldn't feasibly debug it. I sent off stack traces to the team in Germany, but they don't seem to see anything blatantly incorrect. All I knew was that a pointer was getting hosed. Not an easy thing to track down in a few million lines of code.
In the meantime, all of the other members were starting to get geared up. We managed to arrange a slot for pushing OpenOffice.org during one of the open source sessions at WWDC. I figured that we could demo what we have and have it be impressive, so we could have done it regardless of whether or not we succeeded in fixing the word processing module.
We had also been having some discussions with Martin Hollmichel about the possibility of getting our own CVS branch. We'd been hacking away at the makefiles and had done bad things to break other ports, including our Quartz build. Towards the end of the week, we got our own branch! Now if only I didn't need my whole hard drive for swap space I may have been able to get the CVS branch functional...
The best news of the week was that the memory problems were in the dynamic loader and apparently not coming from our code. Even cooler was that they seemed to be innocuous, explaining why everything seemed so stable regardless of the scary things OpenOffice.org and the compiler printed in the Terminal.
I've now spent three weekends mostly working on OpenOffice.org and miss the mountains. But I spend another weekend indoors, plugging away and watching hockey. Saturday morning rolls around and I spend two hours debugging swriter again, only to have X log me out over and over when it runs out of swap space. Frustrated, I break out my financials and assess the current state of my credit card debt. The answer, as always: not good. So if it's already bad, what's the problem with making it a bit worse? I drive out to CompUSA to get a hard drive. They have no firewire hard drives for less then $400. So I decided to buy beer instead. On my way out, though, I notice that they have a refurbished G4 Cube sitting in its original packaging underneath a bunch of refurbished iMacs. I leave the store, get the beer, go home, drink a beer, try to debug swriter again, watch my TiBook quit, take another long hard look at my credit card debt over another beer, and then decide to F-it and buy the cube. If nothing else it'll get me closer to seeing if those free gift credit card point programs actually do give you free gifts.
I go back to Comp USA and get the cube. There's actually two there, one with the box still taped, the other with the box opened. Naturally I pick the sealed one first figuring it's been tampered with less. I ask the sales rep to let me open it to examine if it's got all the contents. Yup. And I had forgotten how sexy the cubes look. Mmm...new toy euphoria sets in. I then take it home in my car. In homage, I set it up in the smack center of my desk and chug a beer as my first sacrifice to it. I decide to relegate my TiBook to the portable typewriter heap and get the cube ready for doing OpenOffice.org work on it. I download the firmware updates and OS 9 updates from the web, and then I drop in my OS X CD. Literally. It just drops in. Hmm. It doesn't show up on the desktop. I try and reboot off of it, but the cube doesn't want to know it. I open the system profiler. No DVD-ROM drive. That's funny, I thought. I figure perhaps it would make the cube happier if I had another beer. It didn't seem to notice. After an hour I flip the cube over and take the core out. Funny enough, the software was right and no amount of beer would have helped resurrect my newest geek idol. There's an empty space where the DVD-ROM drive should be and an OS X CD rolling around in it. Great.
Of course, I've finished half a case of PBR by now and am in no state to be driving. So I package the cube back up and have my friend drive me back to the Comp USA. The Apple rep there looks at the machine, astonished. After 15 minutes of some interesting vibes, I get them to agree to exchange it for the other cube they've got sitting around. Wiser for the experience, I ask to take the computer core itself out to examine the guts of the computer. This second one has a DVD-ROM drive. But it's got more case flaws. So I start trying to sweet talk the Apple rep there and let him know what I'm doing for WWDC. Mostly blank looks go across his face, with a slight recognition of the WWDC acronym. But he does agree to let me take the new core with the better case. I get it home and set it up while consuming the rest of the cube of PBR, and voila, I've got a development cube of a different sort!
I find out that a cube with 64 megs of memory isn't happy running OS X. So I place an order for some memory and start setting up our CVS branch. That shouldn't be too processor intensive. Poor Martin seemed to have been going schitzo with the sheer number of branches he had to manage between the 1.0 release and all the internal StarOffice stuff as well. So I take on the task of getting all of the required sources onto our new branch. The final hurrah for my build on my TiBook...it's the only one I know works. After a day of having fun with cvs tag and the usual updates, everything is in place...and in my first fell swoop, all of the patches developed by our team go into the repository! I then start building. It's slow as sin, but on par with my TiBook. I figure I'll have something to debug in a week or so.
512 MB of RAM arrives. I stop the build and plunk it into the cube. I restart the build. It's done in eight hours by the time I get home from work. gdb loves having memory and can step over a line of code in less than a minute. It's a miracle.
swriter launches :) It turns out the compiler had been trying to do some sex magic to upcasted pointers, but it's different if you include the full structure definition or only a forward declaration. I commit the change straight to the repository. A week and a half before WWDC, every component is launching! We've just got our first binary build :)
It's now one week to WWDC, and things start getting crazy from here. I've got a feeling that this thing won't be relegated to the news scrap heap, so I decide to spend time trying to make a relatively idiot-proof installer that will make sure the rather specific other pieces of software are installed on someone's machine to run. There's no way we can get an OS X Quartz native installer ready in time, so I decide to try using an installer builder I'm familiar with. They even had a shareware and freeware license! Unfortunately, they want the full installer to be assembled for their review before granting one. Seems a bit bizarre, but what the hell.
I spend the whole weekend developing the installer. Two whole days. I hate making installers.
1.0 still isn't released, so the non-OS X specific guys trying to help out are basically tied up getting everything ready for that release. The OS X guys are on their own. I hear nothing back from the company from whom I'm trying to get a license. But in the meantime, Dan downloads the installer and starts verifying that it works and helps me iron out places that still have my username or build path incorporated into them.
And he finds out it's working pretty well. But it doesn't save. Bleh.
Zaheda lets us know that we'll even probably have a booth at WWDC so we can evangelize there. How cool would it be to have CDs to give out at the show instead of just demonstrating it on a machine? Hmm....I wonder...
Still nothing back from the installer company. So I start sending out frantic e-mails to the list of people trying to help out to get them to try and contact the company. We need some sort of testing time for something that'll be this big. Scott calls them up, and I call up the sales rep from my real company on my lunch break. And...no dice. They believe that the three of us developers are funded by Sun and aren't freeware developers. Tell me another.
I wig and frantically ask anyone who might be able to help if they know of any other companies who would give us a license without having to evaluate the full installer again...I don't have two days to spend on a build.
The big 1.0 launch of OpenOffice.org on Win32, Solaris, and Linux x86. All of the community marketing people and other developers are busy arranging all of the stuff for that release. But a friend manages to route me along to ZeroG. They agree to let us use their installer builder for our first binary distribution! After work I go home and learn InstallAnywhere. I build a first attempt at an installer over a case of Sammy Adams. In 8 hours everything but the final postprocessing script is working fine, including the billboards and splashscreen. I'm so impressed with the product. I hate making installers less. I immediately think about switching over to doing all of the installers in my real job with it :)
Dan starts looking into the save bug. Wouldn't it be great if we could fix that before WWDC? But it's rough going...it seems to make no sense.
1.0 is released, and the marketing volunteers are probably burned out. But they step up to the plate and help us out. This day they rested, but the task ahead was a daunting one, even for a fully commercial team: organize a release in four days.
During my lunch break from work, I finally manage to make the one magical change that gets the installer working. But I don't know it yet...it took eight hours for someone else to tell me :) After I found out at home I spent another hour cleaning it up a bit more and packaged it up to send off into the abyss! Dan downloads it and tests it again.
Still no luck on the save bug, but Dan manages to send us stack traces of where it's happening. Apparently it can only save into /tmp, but even then locks occasionally. Martin tells us that the code doesn't exist in 1.0 where it's locking, so perhaps we can patch up the folder stuff to work and call it a final binary. But I've got to get on my walking shoes...I drive up to San Jose since on Friday I have to set up the demo machine. I get to my friend's place, drink some port, and then stay up for three hours trying to assemble a reasonable developer CD with the new installer.
Everyone involved with the organizational aspects of the release is now perhaps a little rested from the 1.0 release, so we decide to have a conference call. At 9 AM. With people in Germany, the US, and who knows where else. In an hour, though, it's all ironed out...we've apportioned the tasks we need to do like the logistics of updating the websites, working on CDs, trying to come up with a FAQ, writing a press release, etc. To pull it off everyone will be pushing for the next two days, and for some continually. We even shift work across time zones so we can have enough time for mirrors to update that are being saturated by the 1.0 release. Amazing. And a few hours later Zaheda summarizes our tasks into a great list. Ah, how there are times when it's good to have the experience of someone who's obviously managed in the past :)
Then I go to Apple's campus to go over the demo and the slides for our presentation and meet with the person organizing the session I'll be speaking in. In my 12 years as a Mac developer and three living in California, I'd never seen Apple's campus. Upon driving in, I felt like I had finally completed a pilgrimage I should've undertaken years earlier when there would've still been multicolored logos and dogcow statues sitting on the lawn. I run around getting everything ready, but get a good taste of things to come...people seem to be curious about the port and looking at seeing it run! I just wish the other guys could've been there to witness the first excitement of showing it to someone who hadn't programmed it...and also being kinda bummed when it first locked up during a demo too.
But I also got to meet Zaheda for the first time too! How cool :) We also decide that we can't get any kind of funds for short run CD duplication, and no one we know can make enough. So...the only thing left is to try and burn them ourselves. Why not?
I then drive back to Santa Barbara and crash hard.
Everyone's off busy doing their own thing, and I'm busy trying to check all of the tasks off of my list. The biggest one...I've got to finalize the binary. So I work with Dan to try and track it down. I then see why he was having so much trouble...it's a bear to debug and work on printing out unicode strings and such. After a few hours, and a few PBRs, I decide "what the he**...let's just comment it out and see what happens." Lo and behold, it can save into places other than /tmp. It still locks, but the saves seem to complete :) I spend the next couple of hours finalizing the installer, recompiling without debug symbols, and assembling the final installer to upload to Germany.
I then take a nap deemed necessary by the combination of exhaustion and 8 beers.
Kevin, another one of my friends who lives up in the Bay Area, goes to a flea market and buys 200 blank high quality 24x CDs for dirt cheap. Those are the ones with the black faces :)
I assemble a final CD image and get Isaiah to come in the back. We had three CD burners between us, some beer, generic blank CDs from CompUSA (the ones with silver faces and no markings whatsoever), and a bad chorizo pizza that gave us the runs. But we had fun burning CDs. Continually. After I fell asleep, he stayed up burning more, and I continued the moment I woke up again. By the time I left Santa Barbara the next day, we had over 70 CDs...not too shabby. And I was exhausted from waking up every 45 minutes to run to the bathroom. Word to the wise: while it tastes real good, expect side effects from the jalapeno and chorizo pizza toppings.
I wake up and begin burning CDs and packing for a fun filled week of OpenOffice.org stuff and obtaining general training and ideas for my real life work. After talking the night before, it was rather clear that I probably should have a cell phone to really allow this week to work out. Not only may I be called by press type people, but I was going to be meeting up with people I've never seen before. At a conference with over 2000 people at it. Yup, I need a cell phone. I'll spare the litany of the cell phone, but let's just say it involved waiting in a store for three hours listening to muzak play through a speakerphone while trying to get my phone activated. Bleh.
I get everything packed hastily and drive on up to San Jose. I tried stopping off at Kevin's to pick up the 200 CDs, but no luck. He was asleep by 12:30AM. So I went to the hotel and got lost in downtown San Jose at night. I got to the hotel at 1:30AM. By now I'm fairly wired...I'll be going to my first WWDC! So I have two beers and stay up for another three hours burning CDs in the hotel room (still ones with the blank faces).
I get to go to my first keynote and hear Steve Jobs speak! While waiting around in the lobby to get in, by chance I meet Louis for the first time. After the keynote, it's time to meet up with everyone else and try to get the booth set up, while installing 10.2 on that venerable TiBook of mine :) I finally got a chance to meet Scott too! It was so nifty to finally be able to associate faces with e-mail addresses.
Our booth looked so ghetto. Everyone else was there with all kinds of nice marketing signs and everything. We had a sign above it indicating who we were, but aside from that we had the demo machine, a nice 24 inch LCD monitor on loan, and some stacks of blank CDs. I stay there in the afternoon to watch over the booth while everyone else does their things, and burn CDs. The show's supposed to start at 6. But at 5PM, they decide to drop the curtain early and let people walk around! Thankfully Scott was there to help out.
Things started out pretty slowly. I was able to take some time out and help a nice college student get a window maker installed on his portable. After that one person stopped by and kind of looked up. "Does it run?" he asked. "Take a look." It was cool to see that first person get kinda excited about the program. So I give him one of our completely unlabeled CDs. We didn't even have a pen to write what they were. How low rent. But the guy was still excited nonetheless. Scott and I then have a short break to continue burning CDs on the demo machine and the burner he brought. Shortly thereafter two people stopped by, then it grew to four, then six. By the end of the evening there must've been a crowd of people about 3 to 4 people deep surrounding our booth while Scott and I were talking about OpenOffice.org and handing out CDs. With some time left, Zaheda and the others wind their way around and brought gifts with them...labels! We had black and white labels to put on our CDs. Could it be that we were beginning to become a credible operation and case aside that look, aside from some bit**ing demo hardware, like we were running out of a basement?
Towards the end of the show I also met Martin for the first time. He had flown in from Germany to help me out with all the technical questions. This project is so large. Even though I'd been working on it for just 3/4 of a year, I still had nowhere near enough familiarity to answer real tech questions that I hadn't come across in the OS X port. We all went out for a nice dinner...the first beer I got as a reward for OpenOffice.org work :) Hopefully it's not the last ;)
After we split our separate ways, I drove up to Kevin's place and picked up those black blank CDs. Got back to the hotel and continued burning until about 3AM when I went to sleep.
I woke up on Tuesday at 7 AM...I was too excited I couldn't sleep. It was the day I'd be presenting! I hadn't experienced that raw adrenaline rush since my first day of college, I think. I burned some CDs and then headed on over to the booth where the faster burner lived. Joe, Scott, and a couple of other guys arrived in the morning as our booth was open all day. We set up the machines, burned CDs, and got some nice demonstration files to open. Everything was going well; CDs were continuing to be handed out, and one or two people looked like they may fall victim to my recruitment tactics. Everything was going swimmingly until...at about 11AM, the booth demo machine died. Ugh. And I was supposed to meet people at noon since I was speaking in the afternoon. OOo was dying right after launching, while spewing messages about not knowing its X locale. Thus started the frantic latte-fueled harrowing fun of trying to rebuild a machine as fast as possible. We couldn't remember if we had been changing languages on the machine, so we removed everything but English. Nada. Then reinstalled X. Nada. I knew it would work from scratch on a fresh installation, so tried to install off of an X CD from a friend at the conference. Unfortunately, the CD didn't boot...10.1 came before our machine's motherboard was officially in existence. So it was time to try out the magical software restore CDs. 5 of them. And an hour had already flown by. This was just way too slow. So, we just swapped out the hard drive for a clean one. Set up the basic user accounts, reinstalled OOo and its supporting cast yet again, and fired it up...nada. So we started playing with the network configuration...obviously it may need to have been reset. Then, bingo. We found it. The dynamic servers were giving the machine a bogus IP address. Apparently X11 didn't like that. So, after an hour and a half of frustration, we found that all we had to do was get a valid IP. Mark one bug up that we'll have to make sure to fix later on. Hopefully as I wigged I didn't upset too many people...or at least provided some interesting entertainment...
Tuesday afternoon was the presentation. Just a little five minute bit and a chance to make another plug to get some developers psyched up to help us out, or at least try to build the code. Martin and I got there thirty minutes early to check out the demo machine and make sure everything was still functional. I met some of the other guys who were speaking and went backstage to interact with the tech crew. The session was in the Civic Auditorium, basically a small theatre. We had a backstage behind the curtain with all kinds of equipment there, and a staff to help us out. We went over basics of transitions, how to click the projector to get to new slides, and the like. Soon people started coming in, so we all got ready to present. I was towards the end, so there was a decent amount of time I had to wait, pace around, and get psyched up. I hadn't been speaking on-stage since my failed high-school acting career. Right in the middle of the first speaker's presentation, one of the tech guys had come over. Apparently the machine which was going to be used for our demos had gone to sleep. That wasn't particularly nice of it, especially for the tech crews who had to try and synchronize the projectors with the video output of the machine. Between the first and second speakers, me and a tech guy nonchalantly walked out to the demo machine and woke it up and told it to never go to sleep again. After 30 seconds, we walked off and went backstage again. Five minutes later, though, the machine went to sleep again. So after the next speaker got on stage, we went back out again. Of course, we had forgotten to close the system preferences. We did, and with a machine that would sleep no more, went backstage again. Everything was fine until about 15 minutes later when its screensaver went on. Bleh. One of the tech crew went out between the next speaker transition and set things up so the screensaver wouldn't go on again. This was definitely not one of my days of better luck with computers. Soon after that I was hooked up to the wireless mike and strutted out for my five minutes in the limelight. I thought everything went well. I didn't fall down, and the very brief demonstrations of the software worked except for a crazy spreadsheet formula I entered incorrectly at the end. I guess I must've made some impression since after the session about 20 people walked over to our booth to get some CDs :)
For me, though, it was a relief. All the major stuff had finished just fine, and we had completed our major task of getting the word out without a hitch. The sheer amount of caffiene, adrenaline, CD burning, and beer definitely took its toll as I started to slowly burn out for the rest of the week. Not to mention when I got back. But hopefully it'll get a few other people to come join our little insanity and begin experiencing the haphazard craziness that can make open source so much fun. With luck, perhaps we'll have something else to announce next year and maybe I can be famous again for a day. Thanks to the entire OpenOffice.org team and our hardcore OS X porters for making the whole thing possible...congratulations to you all!