against the components utopia

Hey, Jamie, I will probably just have to say that I find the strong
form of components utopia to be an unobtainable grail.  Arguing about
it seems about as effective as arguing about mandantory static typing.
I don't understand them, and they don't understand me.  But let me
take one shot at it.


The basic argument is that no components system completely prevents
components fouling each other up even when they ought to work
together.  Therefore, any practical components system must have a way
to cope with this situation, some way for humans to get into the loop
and modify one component or the other.  If you leave this out, your
system is hopeless.  If you put it in first, then a lot of problems
that used to be show stoppers instead turn into opportunities for
improvement.

Component utopians seem to believe that a sufficently advanced
component system can allow all components to work together when they
are intended to work together.  Some start down the path of isolating
components from each other.  That's a good path to start down, I agree
-- having some kind of skin between components help prevent a variety
of problems.  The extreme stance is incorrect, though: components are
not supposed to be *completely* isolated.  They are *supposed* to
interact with each other.  If I install a new version of a library,
it's because I want some new behavior on my system.  That system is a
result of components interacting with each other.

You have to leave up some channels of interaction, and every such
channel is a possibility for components to foul each other up.  Even
speaking through a perfectly type-checked API leaves open questions
like, is 0 a valid argument for this or that method?  Even if you use
a theorem prover on all of your components, you merely defer the
problems to problems of specifying what properties, exactly, have been
proven.  Components that should work together sometimes simply are not
going to.  You have to have some mechanism for dealing with this
situation and tweaking one package or the other so that they work
together.


If you leave out the override mechanism, every possible form of
conflict becomes a show stopper.  Different components were compiled
against different versions of each other?  Then you *have* to allow
different versions to coexist.  Different components put stuff in
different parts of the namespace?  Then you *can't* have any global
namespace at all.  Etc.  You end up giving yourself ultimatum after
ultimatum.  As I argued above, the ultimatums never run out.
Eventually one component or the other has to be tweaked.  To make a
human analogy, if two people want to cooperate, then one or the other
has to be willing to compromise a little; it is nonsensical for
people to cooperate while being completely unwilling to reach out a
little for each others' needs.



Thus, include the override mechanism from the beginning!  If you do
so, you immediately get a working system.  Maybe overrides are needed
frequently, but at least they are always possible.  From here, you can
take all of the cool ideas that were ultimatums in the above
paragraph, and consider them as possible ways to improve the already
working system.  For example, maybe it helps to have multiple versions
of the same jar coexisting.  Or, maybe the hassle caused by the extra
complexity is worse than the hassle caused by not having it.  It's a
rich design space, now that the design space is non-empty.


I don't think that source recompilation at install time changes any of
this.  So, I don't expect that Gentoo has managed to avoid the above
fundamental problems.  But this email is too long already, so I'll
leave that for another post.

One thing I'll mention is that the multitude of universes you mention
is only a problem if you overlook repackaging.  Debian/unstable has
10,000 packages in it.  That's a finite universe, but an awefully
large one.  The reason it has so much stuff in it is that anyone can
contribute a new package.  Fundamentally, someone has to do the work
to check that a package works nicely with others, and that work might
require a number of trivial little changes to the package.  In
Debian-like systems (and package universes in general), that work is
done by packagers.



> Going back to Java, we don't have to be anywhere near as strict as
> Debian is, because we have much better ABI compatibility than for
> native code. For the most part we /can/ just use the latest binary
> version of software and there's a reasonable chance it will work. The
> Log4j example I gave is certainly far from unique, but I also don't
> think it's so common that it rules out even attempting to have
> independent packages coexist. It only means that you can't just stick
> everything on the classpath and expect it to work.

Just by the way, you can perfectly well statically link in sbaz if you
want.  In fact, the sbaz package itself does that.  In that case,
though, you should probably not put the libraries in /lib.


> 
> > Individual package universes have rules, but they tend to be more
> > informal than what a package tool would insist on.  Importantly, the
> > rules are different for different groups.  The needs of a 6-person
> > collaboration with 10 packages are different than those of a
> > Debian-sized collaboration with 10,000 packages.
> 
> But is the 10-package universe useful? Won't it be forever expanding,
> duplicating the work done in some larger universe? 


One concrete example would be a 3-person team developing the software
infrastructure for a small business.  They work with 10-ish packages
that are specific to the business.  Scala bazaars supports that, just
as Debian's APT does (though, I think, more clumsily).

Now, yes, those 10 packages do not do everything they need.  They
would not include data structure libraries, GUI frameworks, CORBA
integration, etc.  Thus, the 3-person team would work within the
*union* of the 10-package universe plus a larger one that includes all
the reusable stuff.  The rules for the 10-package universe could be
essentially nil, because it's just 3 people who know each other well.



> Incidentally, is there more than one sbaz universe at present?

There are 3 that I know of.  One on lexspoon.org (scala-dev), one is
on a server at EPFL that is planned to be the new scala-dev, and one
on my laptop (sbaztest) used for testing new versions.

I'm not being flippant.  Anyone can create their own bazaar, just like
any group can make their own wiki or mailing list, and just like in
meatspace where anyone can create their own club.  I don't think that
making a mailing list should require registering with a central
authority.  Nor should having a Wednesday-night bridge game with the
Joneses require registering your meeting somewhere.

Likewise for package sharing.  Every sbaz directory includes a file
meta/universe that specifies the universe of packages visible to that
particular sbaz directory.


Less philosophically, we host scala-dev right now, but I would imagine
after 0.5-2 years or so that someone freezes off something like an
Ubuntu release or a Debian/stable release, thus providing a stable set
of standard libraries for Scala hackers to use.  Exactly who does that
and when and what the rules would be and where the result is posted
are all completely up in the air right now.  Enthusiasts are welcome
to create their own distributions if they like....


-Lex