Crazy claim, eh? I figure there’s no better way to get this claim tested that by posting it as a truth!

We’ll here’s the story: Once upon a time, lloyd (that’s me) wanted a parser that didn’t suck that he could embed in a particular product, primarily because he was sick of patching other peoples code: solving problems like:

1. crashes on certain malformed inputs
2. the niggling tendency to scan from the begging of the input text every time a number was encountered
3. the requirement that you had to hold the full json text in memory at one time (usally in addition to an in-memory representation).

To be fair, the project that I was previously using got a lot better, but it still lacked a couple key features that were extremely important to me: stream parsing, and representation independence. At the time (and actually still), we were using JSON as the data representation over our IPC channel (in that same particular product), and this meant that a lot of data was flowing. We had one of two options:

1. throw away json and move to a binary format
2. make a json parser that caused the IPC serialization overhead to fall off our our profiles…

Cause’ I love transparency over the wire (binary still ain’t that scrutable), and because JSON is the representation language of the web (hush now XML people) I decided to try to build this parser.

Here’s the thing, I’m rather stupid and short sighted, and didn’t realize that it’s actually extremely well suited to high level language bindings for other languages, say, ruby or python. Luckily, Brian Lopez is not stupid, and he has brought yajl to ruby.

Initially Brian is reporting that it’s about 4 times as fast as YAML parsing via syck, about 20% faster than the other native JSON parser for ruby](http://flori.github.com/json/) out there, and in general uses up to ½ the memory (now this claim especially is difficult given the exponential nature of the heap mechanism in ruby, so perhaps we need to retry this with some of them gc patches I wrote, or with 1.9).

I’m posting to re-iterate how awesome Brian and I are? No, not really. I’m posting to challenge skeptics to validate these findings, just cause’ I was extremely pleased and surprised to hear these numbers from brian, and a little bit of that rigor that comes from public scrutiny couldn’t hurt…

till the next, lloyd

Later this week I’ll be moderating a chat entitled “Implementing the open web” at gluecon in my home town, Denver.

So in preparation for this panel, the logical first step seems to be to establish a clear and concrete definition for the “open web”…

Stealing Brad’s work

Aha, this is easy, I’ll flip back to one of my favorite responses to this question, written by Brad Neuberg last April. I feel that Brad has nicely covered the bases, and tend to agree with most of his points. Some things that I think are good enough to repeat include “Freedom of Social Forms”:

The Open Web should support extreme gift economies, such as open source and Wikis, all the way to traditional free market entities, such as Amazon.com and Google. I call this Freedom of Social Forms; the tent is big enough to support many forms of social and economic organization, including ones we haven’t imagined yet.

We must leave our philosophies around making a buck at the door. People are free to experiment with different business models all they like, from advertising to free tools with commercial consulting available, to whatever you can think up… People are free to build commercial and proprietary “derivative” work on top of the open web, and the platform is in no way geared to one model or another. The key here is that the “open web” attempts to serve all without bias, and in order to properly do so cannot in itself be owned nor governed by any small set of companies nor individuals, nor can it involve any technologies that infringe on the freedom of applications built on top of it.

Next on the list of restatables includes what Brad calls Openness (ok, a criticism, Openness is vauge, perhaps we should call it the Specification or Documentation requirement):

Openness – Whether the protocols used are de facto or de-jure, they should either be documented with open specifications or open code. Any entity should be able to implement these standards or use this code to hook into the system, without penalty of patents, copyright of standards, etc.

Specifications OR code. Hear that? Quite forward thinking given the present state of SQL in the browser. At present, the code is the specification. I think this is extremely messy — Having “standards” where there is no specification — however who ever claimed that evolution was pretty? Look at a very nice bit of work that I’ve come across lately, JSONQuery – A means of performing sophisticated queries against JSON documents. But here’s the odd part, The best “specification” I can find is that blog post. No real specifications, but a couple different implementations. So we could say Brad’s forcing a mess on us here by allowing the code to be the specification – But my claim is that as technologies gain popularity, the BNF will get written – And you can’t really write a specification until you’ve kicked the tires and actually seen it in action. And a specification needs a champion anyway, I think JSONRequest is a fine piece of work, but it’s still just sitting there.

Open Questions

Transparency. Brad argues the should here, that we should be able to inspect the web at all levels from protocols, to documents, to layout, to code. SSL is inscrutable over the wire, and minified javascript is also. Further, it could be argued that this goes against our previous claim, that the open web is a platform which allows for Freedom of Social Forms, now we turn around and say that whoever you are writing a web application for profit, you should take pains to lay out your site so that it’s scrutable for others. Perhaps Brad was taking a jab at Flash, don’t get me wrong, I dislike the clumsy way that flash fits into the web as much as the next guy (there’s the discoverability problems, the search problem, absence of good free tools, yeah yeah.) But, I think that this argument applies more to the sites written on top of the technology stack that is the open web, rather than the open web itself. So as much as I like it, I don’t know how much this belongs in the core definition.

Missing Bullets

Here’s where I get into some difficult terrain where my judgement may be clouded by my dayjob, but I shall forge ahead despite that. This point comes when you cross several of Brad’s ideas about what the open web is:

1. It enables “Third Party Innovation – without asking the powers that be for permission”
2. “Decentralization – …the web, an open system that anyone can plug into and create information at the end-points…”
3. “Third-Party Integration – At all layers of the system third-parties should be able to hook into the system, whether creating web browsers, web servers, web services, etc.”
4. “Hackability – It should be easy to lash together and script the different portions of this web.”

All of these points somewhat redundantly point to a basic idea. Consider web applications (aka sites): Anyone should be able to write a site using well documented and accessible technologies and tools (today: HTTP, javascript, CSS, php or rails). This person should be able to consume aggregate and perhaps enhance information from multiple sources (today: leveraging APIs serving structured data over HTTP). They should be free to make money off this site in any way that they like (today sell ads from google or yahoo, charge your users, offer advanced paid features, whatever).

This is fine and makes sense the web platform itself must be application agnostic and free for all to apply, but let’s dig into the “Third Party Integration” point. Anyone should be able to write a web browser. Really? Think about that. Writing a web browser involves a lot of work, and if you regularly read ajaxian it feels like more and more every day. Javascript evaluation? Document rendering? CSS parsing? Caching? Option management? Localization? A web browser is not a trivial piece of software, and it feels like with innovations in the open web we’re looking to make them exponentially more complex… We have already have started putting more and more in the browser, richer video rendering, 3d vector graphics, a full blown database… Yikes.

So no, with the way this is going, not anyone can go out and build a web browser. And in fact the people building them right now can’t really keep up. But stay with me, the fix is simple. I think we can just take a look at what the bright folks at Mozilla have already done:

First let’s look at Geode – a GeoLocation extension for firefox. A user installing this ext. would be able to easily share their location with javascript at a level of granularity that they specify. Now check out the weave extension , also from mozilla labs. This one most recently attacks the problem of identity management in the browser.

Both of these projects are clearly working to forward the open web. Both satisfy all of the criteria set forward above by Brad, and both are innovations that could push our web forward. But this isn’t the point. The point is take a look at how Mozilla initially realized these features? Not inside but outside of the browser, as extensions.

Here’s the problem, viscerally. The latest Firefox 3.5 beta has GeoLocation built-in:

However, we were still left with a chicken-or-egg problem: unless lots and lots of users installed an add-on, websites wouldn’t have a significant audience for which to develop location-enabled services; and without lots of useful web content using the feature, users had no reason to install an add-on.

Herein lies the problem folks. This is the WRONG DIRECTION. We shouldn’t be moving more shit into the browser, we should be pulling it out! Why? For the good of innovation. It goes like this:

A select few companies have the resources to build a browser, while many folks in the community have vested interest in getting distinct new features into the browser (aka, the open web). However the current situation requires that browser vendors are the ones doing the innovating (because if it’s not built into the browser, we “wouldn’t have a significant audience for which to develop” using it). Further, the moment something gets into more than one browser it gains a momentum, a dangerous momentum. We’re not free to change our minds. Have a look at SQL in the browser, and search around for alternatives to SQL client side storage. There’s a uncertainty here around which way is the right way, but it may be too late: we already have SQLlite implementations in several major modern browsers.

The Core Problem

The core problem is simple. There is no real way to get people excited about a new javascript API other than building it into a browser. And even then, you’re going to have to wait for other browser vendors to come along that implement the feature in the same way. This nebulous thing that we now call the open web is dangerously slow in comparison to other runtimes that don’t require inter-company participation.

The core problem is that we’re experimenting inside the browser because we don’t have any really viable ways to experiement outside of the “core” browser (yes, client side SQL is still an experiment as far as I’m concerned). This problem is fueled by the following facts:

1. There is no reasonable extension API that runs across all browsers.
2. This extension API that doesn’t exist doesn’t support extremely simple component installation, update, and revocation.
3. This extension API that doesn’t exist isn’t rich enough to allow something like SQLLite, or a JSON database, or identity mangement to be wholly implemented inside of it.

Thus finally we have enough to correctly patch the definition of the open web: The open web should allow decentralized extensions to it’s own fabric. That is any rendering engine that is capable of rendering the open web must be augmentable with additional versioned APIs in a way that is seemless to the end user.

What did you say the open web was?

Oh, right. That’s where we started, we need a clear and concrete definition of the open web. Well shucks, let’s try it, borrowing heavily from Brad:

The open web is a set of technologies that enable the production of network accessible applications that are highly accessible to users everywhere. These specific technologies may change over time but are governed by the following properties:

• Documented – Every technology that is part of the open web is documented by either a rigorous standard or a reference implementation. This documentation is freely available.
• “Free” – Every technology comprising the open web may be freely used (no need to pay royalties, worry about infringing on patents, etc)
• Application Agnostic – The technologies included in the open web enable equally enable free or proprietary extensions or applications.
• Simple to Use – The open web is a platform designed for applications targeted at an average person without significant technical knowledge – an overall principle is that none of the technologies that are part of the open web should preclude a presentation that achieves this goal of accessibility.
• Extensible – The core technologies of the open web should be extensible at many different levels. That is it should be possible in a decentralized fashion and without any approval to write web applications, web services that serve information to applications, new web browsers that can render applications, or extensions that add new features to existing browsers.

In short, the Open Web to me is an idea. It is a set of technologies that are well documented, cost nothing, and allow for the creation of applications that are accessible to a wide audience. The open web is an extensible platform that allows equally for the creation of free or commercial applications. Finally the Open Web is designed to evolve, embracing at every level, the notions of extension and evolution.

Expiration of my account at hub.org, and my discovery of mosso & slicehost has prompted me to move all my personal shit around… Along with that move I figured I might as well disband my efforts at a ground up implementation of every piece of technology required to run a site, and just throw apache, php, and a little wordpress at the problem (sorry erlang & yaws, I still love you. don’t hate me ruby & lighttpd, you guys are really cute!). This is a common theme in my life, get stuck in the interesting problems that pop up while trying to solve a problem…

anyhow, welcome! we will shortly return to your previously scheduled programming (woteva that woz)…

lloyd

Seeing all this action in ruby trunk, combined with what I’ve read ’round the net had piqued my interest in 1.9 performance differences.

Given the set of contributed benchmarks that I used when developing the inital patch to improve the reclamation and decrease memory usage of ruby, I did some comparisons of ruby 1.9 vs ruby 1.8.6, and of ruby 1.9 vs a patched ruby 1.8.6.

In short, looking at this data leads me to some preliminary conclusions:

• 1.9 is decidedly “faster” than 1.8.6. Especially when runtimes are longer, or yaml is involved.
• 1.9 uses slightly less memory overall. *There is considerable room for improvement in 1.9’s memory reclamation.

So why is memory usage important? Well, here I’m personally biased. My primary day to day application of ruby is embedding. In this scenario, I really want a small and tight ruby that I can use to move a buncha code out of c++ and into ruby. I’m also interested in portability of ruby to less capable devices. In both of these situations, memory usage is an important factor. Additionally, I think more generally, low memory usage minimizes the copy on write issues prevalent in ruby on rails environments. If the process has a small, tight, compact heap, it doesn’t matter so much that we have to copy the whole thing on each fork.

My educated guess here is that we could focus energy on minimizing memory usage rather than a external bitset, and we’d get the desired COW friendly ruby at a similar (minimal) performance cost, but the solution would yield more across-the-board benefits. To be clear, moving mark bits into a external data structure (and out of the heap) is really an optimization focused at ruby in web environments that comes with a cost. I don’t want to pay this performance penalty for something that doesn’t really benefit me.

So I guess the task is to continue to follow 1.9 and to attempt to figure out which 2 lines are inverted that could really make the difference… It can’t be too hard, right?

till the next,

lloyd

w00t. An email from matz, and a little spelunking in the ruby subversion repository shows that there’s some tinkering going on in ruby garbage collection land. Here are the interesting change logs:

r15674 | matz | 2008-03-03 01:27:43 -0700 (Mon, 03 Mar 2008) | 5 lines

* gc.c (add_heap): sort heaps array in ascending order to use
  binary search.

* gc.c (is_pointer_to_heap): use binary search to identify object
  in heaps.  works better when number of heap segments grow big.

and

r16194 | matz | 2008-04-25 03:03:32 -0600 (Fri, 25 Apr 2008) | 7 lines

* gc.c (free_unused_heaps): preserve last used heap segment to
  reduce malloc() call.

* gc.c (HEAP_SIZE): use smaller heap segment (2K) for more chance
  to be freed.  based on patch from authorNari .

* gc.c (rb_newobj_from_heap): eventually allocate heap segments.

So now in ruby 1.9 trunk we’re keeping heaps in sorted order by memory address, and using binary search to answer the is_pointer_to_heap() question quickly. This optimizes things to the point where we can really crank down heap size. Smaller heaps means more OS reclaimation, means reduced resource usage, and should even mean a ruby with reduced COW badness. All this at a minimal performance impact for normal execution (maybe none, matz knows).

So applause to open source, and matz specifically for sifting through all the ideas/hacks/and patches to realize this thing. It will be interesting to include 1.9 in the

overview

This page details some changes to the ruby garbage collector which seem to afford a 25% reduction in maximum heap memory usage, and nearly double the amount of heap space ruby’s is able to reclaim. This comes at the cost of a 2% performance hit. More to come, stay tuned.

patch

The latest patch to ruby 1.8.6 is available here.

summary of changes

• Fixed 2k heaps
• Heap implementation broken into separate c/h file
• heaps stored sorted by address for efficient is_pointer_to_heap()
• “adaptive GC frequency” – perform GC more frequently when free percentage is high.

testing strategy

To test changes, I’m comparing the performance of a handful of tests (most contributed by Hugh Sasse, many thanks!). Ruby 1.8.6 unpatched is run over these tests, then compared against the performance of ruby 1.8.6 with my patch. In order to get “maximum heap size”, “current heap size”, and “nmber of gc passes”, a GC.heap_info function has been hacked in to both.

NOTE: When referring to heap size, we’re really talking about the amount of memory dedicated to ruby heaps, not the total heap usage of the ruby process.

finally, all tests are performed on a macbook dual core with 2 gigs of RAM. We use wallclock time, but I’ve ensured an idle machine and rerun tests to verify that system load doesn’t significantly affect results.

data

test                   before                           after                            difference
                       time  gc  max mem   final mem    time  gc  max mem   final mem    time  gc   max / final mem

weak_hashes_read.rb    0.459  8  4473980     4473980    0.448  9  2570240     2570240     97% 112%  57%  57%
create_hashes_yaml.rb  25.91 75 27300060     8253120    26.43 75 18012160      614400    102% 100%  65%   7%
create_hashes.rb       0.134  4  1208000      560000    0.142  6  1077248       75776    105% 150%  89%  13%
arrays_read_yaml.rb    0.760 14  4473980     4473980    0.753 13  3278848     3278848     99%  92%  73%  73%
ostruct_read.rb        2.643 19 15055440    15055440    2.866 22  8781824     8781824    108% 115%  58%  58%
create_ostructs_yaml.rb13.67 55 27299780    15055440    14.47 57 15616000     9125888    105% 103%  57%  60%
classes_read_yaml.rb   0.825 13  4473940     4473940    0.850 13  4151296     4149248    102% 100%  92%  92%
create_weak_hashes2.rb 0.497  7  4473940     4473940    0.512  9  2643968     2490368    102% 128%  59%  55%
create_arrays_yaml.rb  9.003 43 15055660     8253140    9.169 44  9859072     2252800    101% 102%  65%  27%
create_arrays.rb       0.171  4  2374420      560000    0.179  6  1912832       75776    104% 150%  80%  13%
growarray.rb           1.933  9   200000      200000    2.116  9   102400       75776    109% 100%  51%  37%
arrays_read.rb         0.089  4  2374460      560000    0.0922 5  1679360       73728    103% 125%  70%  13%
hashes_read_yaml.rb    1.982 23  4473920     4473920    1.908 20  4532224     4532224     96%  86% 101% 101%
plist.rb               4.926 23 15055760    15055760    4.773 18 12734464    10463232     96%  78%  84%  69%
create_ostructs.rb     3.077 22 15055660    11276480    3.284 24  8800256       92160    106% 109%  58%   0%
create_weak_hashes.rb  1.206 17  4473960     4473960    1.132 14  5253120     4933632     93%  82% 117% 110%
classes_read.rb        0.057  3  1208000     1208000    0.063  5   919552      919552    110% 166%  76%  76%
shrinkarray.rb         1.932  9   200000      200000    2.1230 9   100352       77824    109% 100%  50%  38%
ostruct_read_yaml.rb   1.143 20  2374400     2374400    1.114 17  3178496     3178496     97%  85% 133% 133%
hashes_read.rb         0.052  3  1208000     1208000    0.052  4   876544      876544    100% 133%  72%  72%

                                                                              overall:   102% 110%  75%  55%

feedback

Send feedback to lloyd 4t trickyco.de

Here’s the Problem:

http://lists.apple.com/archives/x11-users/2007/Oct/msg00065.html

Here’s a proposed “fix”:

http://aaroniba.net/articles/x11-leopard.html

If they didn’t add RPATH support, DTrace, and pretty much avoid judicious changes, I’d be throwing stones. As it stands, this is extremely annoying, but tolerable. Looking forward to the fix…

-lloyd

overview

Ruby’s GC & heap implementation uses a lot of memory. The thing is based around the idea of “heaps”. Heaps are chunks of memory where ruby objects are stored. Each heap consists of a number of slots. Slots are between 20 and 40 bytes, depending on sizeof(long). When ruby runs out of heap space, it first does a GC run to try to free something up, and then allocates a new heap. the new heap is 1.8 times larger than the last. Every time a GC run happens, the entire heap is written to turn off mark bits, these are stored in the heap. Then we run through top level objects, and mark them, and all their descendents. Then we throw away anything that’s not marked (sweep). Because of the way ruby works, objects may never be moved around in heaps. That means from the time they’re allocated to the time they’re freed they may not be moved to a new memory address.

So this is a very terse summary, more is available in the ruby hackers guide. But it’s enough. There are a couple bad things here:

1. in order for a heap to be reclaimed, all entries on the heap need to be freed. The bigger a heap is, the more likely that it will contain at least one long lived object. The 1.8 growth factor makes it bloody unlikely that you’ll ever get to reclaim heap space.
2. A big heap makes GC slower. You have to scan the whole thing
3. Because we do scanning copy on write semantics are blown. you do a fork, and as soon as GC runs, your whole heap is resident and private memory.
4. We cannot do compaction at GC time, we must either change the way ruby works in a very fundamental way (bad), or think of a creative lightweight way to just keep the heap compacted.

Plan of attack

1. develop a method of quantifying performance of ruby GC & heap
2. prove some of these ideas can have positive effects in terms of memory usage and performance
3. produce patches for the ruby community and anyone who wants em
4. drink beer

Quantifying

First thing we need is a way to get a look at statistics of the gc stuff. So we hack in a GC.heap_info function that returns

• num_heaps – the number of allocated heaps
• heap_slots_free – the total number of free slots
• heap_memory – the amount of memory allocated to ruby heaps
• heap_slots_allocated – the total number of available slots
• heap_slots_used – the total number of used slots
• num_gc_passes – the number of times GC has been run

Great. Next we need test cases. I start with three:

• grow array build a buncha arrays bigger and bigger
• shrink array start with a big array, build a buncha ones smaller.
• PLIST parsing Parse a huge plist file.

Yeah, they’re artificial, but we’ll add more cases as we go. At least we have some constant tests to start with.

Hypothesis

By getting rid of the 1.8 growth factor, and making heaps smaller, we can increase the amount of memory that’s reclaimed. And make ruby faster, by reducing the amount of scanning unused memory that occurs.

data

Vanilla ruby:

running cases/growarray.rb

heap before (mem/used slots/% free/heaps/gc passes) 560080/6501/0.767838011570602/2/9
heap after (mem/used slots/% free/heaps/gc passes) 560080/6367/0.772623384043997/2/11
time 1.339915

running cases/plist.rb

heap before (mem/used slots/% free/heaps/gc passes) 15056520/559048/0.257393875553088/7/21
heap after (mem/used slots/% free/heaps/gc passes) 15056520/109351/0.854744633172117/7/23
time 3.773354

running cases/shrinkarray.rb

heap before (mem/used slots/% free/heaps/gc passes) 560080/7537/0.730840654238983/2/9
heap after (mem/used slots/% free/heaps/gc passes) 560080/6373/0.77240911363474/2/11
time 1.338151

Killing 1.8 growth factor:

running cases/growarray.rb

heap before (mem/used slots/% free/heaps/gc passes) 400080/6501/0.674982501749825/2/9
heap after (mem/used slots/% free/heaps/gc passes) 400080/6367/0.681681831816818/2/11
time 1.337504

running cases/plist.rb

heap before (mem/used slots/% free/heaps/gc passes) 9406200/380442/0.191001631002226/47/51
heap after (mem/used slots/% free/heaps/gc passes) 9406200/109351/0.767468416609429/47/53
time 4.278476

running cases/shrinkarray.rb

heap before (mem/used slots/% free/heaps/gc passes) 400080/7525/0.623787621237876/2/9
heap after (mem/used slots/% free/heaps/gc passes) 400080/6373/0.681381861813819/2/11
time 1.343519

Killing 1.8 growth factor and reducing heap size to 1/10th

running cases/growarray.rb

heap before (mem/used slots/% free/heaps/gc passes) 221540/6491/0.413428519790349/11/18
heap after (mem/used slots/% free/heaps/gc passes) 221540/6368/0.424543647207663/11/20
time 1.351549

running cases/plist.rb

heap before (mem/used slots/% free/heaps/gc passes) 7741680/382475/0.0109718192584778/366/365
heap after (mem/used slots/% free/heaps/gc passes) 3795580/109350/0.423259493670886/179/367
time 7.971707

running cases/shrinkarray.rb

heap before (mem/used slots/% free/heaps/gc passes) 221540/7717/0.302638713175493/11/17
heap after (mem/used slots/% free/heaps/gc passes) 221540/6374/0.424001445870233/11/19
time 1.343365

analysis

note “heap before” means “heap before final GC run”. We fork a process which runs the test case, then we check out the heap using GC.heap_info, then we run a GC pass, then we check it out again.

w00t! we made ruby twice as slow! Well hold on. First inspect the run times of plist.rb (the most realistic test case). Also inspect the number of gc passes. Pretty tight correlation, right? reducing heap size, and removing the 1.8 growth factor both increase the number of gc passes that we make. So we see a significant performance degradation proportional to the number of passes that are run.

Inspect memory usage (still looking at only plist.rb). Vanilla ruby is using 15mb. At the end of everything, and that heap is 85% unused. Kill the 1.8 and we’re using 9mb of heap space, 76% unused. Decrease the heap size, and we actually see memory being reclaimed. After the run and final GC we’ve only got ~4mb in use at 42% free. Immediately after the run we were around 8mb.

parting shot/conclusion

By changing two constants we can make ruby a lot more memory efficient, and at the same time a lot slower. The slow down appears to be largely from increased frequency of GC. Maybe we can look at not running GC every heap allocation, but every N heap allocations… Goal here being to restore ruby to it’s original, or better performance characteristics, but reduce the memory usage.

Essence here is that everyone knows you can grow a buffer by a factor and make things faster. But other aspects of ruby make that choice perhaps not optimal here. Stay tuned, we’ll dig further.

one more thing, ideas on “automatic heap compaction” a global freelist? bad. Why not have per-heap freelists. Why not sort the heaps by usage percentage at the end of a sweep? Allocate the heaviest used ones first… There’s some complexity here around when GC is run… Cause it’s only run when everything is full… But perhaps some room for exploration…

I cannot live without X11 emacs! It doesn’t build from macports right now. As far as I can tell, the emacs that apple ships with leopard is broken, at least for me after upgrade I get:

[lth@tumno ~] $ /usr/bin/emacs.broken
Fatal malloc_jumpstart() error

This blog entry documents the steps to get emacs built, installed and in the macports registry. So at the time of writing, this is the package I’m installing:

[lth@tumno ~] $ port search emacs | grep 22.1
emacs                          editors/emacs  22.1         The GNU Emacs text editor (command line only)

Start by trying to install

[lth@tumno ~] $ sudo port install emacs@gtk--->  Fetching emacs
--->  Verifying checksum(s) for emacs
--->  Extracting emacs
--->  Applying patches to emacs
--->  Configuring emacs
--->  Building emacs with target all
Error: Target org.macports.build returned: shell command
     " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_editors_emacs/work/emacs-22.1" && \
     make all " returned error 2
Command output: Highest address of load commands in input file: 0x2b7000
Lowest offset of all sections in __TEXT segment:   0x2380
--- List of Load Commands in Input File ---
# cmd              cmdsize name                address     size
0 LC_SEGMENT             56 __PAGEZERO                0   0x1000
1 LC_SEGMENT            396 __TEXT               0x1000 0x135000
                           __text               0x3380 0x122eb8
                           __cstring          0x126238   0xfc23
                           __literal8         0x135e60     0x78
                           __const            0x135ee0    0x100
                           __literal4         0x135fe0      0x4
2 LC_SEGMENT            328 __DATA             0x136000 0x156000
                           __data             0x136000 0x119ac2
                           __dyld             0x24fac4     0x1c
                           __bss              0x24fae0  0x31aa5
                           __common           0x2815a0   0xaa51
3 LC_SEGMENT            192 __IMPORT           0x28c000   0x2000
                           __pointers         0x28c000   0x1584
                           __jump_table       0x28d5c0    0x3fc
4 LC_SEGMENT             56 __LINKEDIT         0x28e000  0x29000
5 LC_SYMTAB              24
6 LC_DYSYMTAB            80
7 LC_LOAD_DYLINKER       28
8 LC_UNIXTHREAD          80
9 LC_LOAD_DYLIB          60
10 LC_LOAD_DYLIB          52
11 LC_LOAD_DYLIB          52
make[1]: *** [emacs] Abort trap
make[1]: *** Deleting file `emacs'
make: *** [src] Error 2

Error: Status 1 encountered during processing.

So the apple folks have changed the structure of a dylib (to add, among other things, RPATH support. that’s a huge w00t)

Let’s patch this source. Now for some reason, if we apply a patch directly to the current state of the working directory, things go boom. I’m guessing the fix doesn’t play nice with other current patches applied by macports. So I use a sledgehammer. Download emacs-22.1, configure as you like, apply this patch (credit to YAMAMOTO Mitsuharu for producing this patch) and build. Then move the built emacs-22.1 into your working directory, and port install.

The result for me is a emacs which runs in console mode. once I get X11 set up and running sawfish we’ll see if it actually works in X11, in the past the emacs-devel package was required for X11 support, and the brief port description suggests this is still the case..

lloyd