Repository Layout

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Repository Layout

Reid Spencer-2
All,

I'm finding the current layout a bit frustrating, not to mention that it
breaks all the documentation. We currently have modules laid out like
this:

   llvm/trunk/...
   llvm/branches/...
   llvm/tags/...

   stacker/trunk/...
   stacker/branches/...
   stacker/tags/...

and similarly for all the other modules (test-suite, java, poolalloc,
etc.). Unfortunately, this "standard" layout interjects a directory
(trunk/branches/tags) below each module name. I would rather see it laid
out like this:

   trunk/llvm/...
   trunk/stacker/...
   trunk/java/...

   branches/llvm/...
   branches/stacker/...
   branches/java/...

   tags/llvm/...
   tags/stacker/...
   tags/java/..

Fortunately, with SVN, making such a change isn't hard and its tracked
in the commits. Anyone object or have other ideas in this realm?

Reid.


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Repository Layout

Anton Korobeynikov
Hello, Reid.

> Fortunately, with SVN, making such a change isn't hard and its tracked
> in the commits. Anyone object or have other ideas in this realm?
I'm also confused. At one hand, proposed layout (with BTT in the top)
seems to be "unnatural", when we will create new tags/branches.

In theory, each module can have its own version, thus if the want to
"tag" it, we have either copy the whole repository to new tag, wasting
space and checkout times, or "tag" just a subset of the repository,
which seems to be pretty bad, because different tags can contain
different "subsets". The same for branches, but not too "bad", since
branches are more or less temporary. Keeping BTT in the modules will
surely make such problems thrown away.

At the other hand, keeping BTT inside modules can lead to headache, when
some branch, for example, has dependencies on other modules (for
example, some experimental version of CFE can be stick to some version
of LLVM and so on)...

--
With best regards, Anton Korobeynikov.

Faculty of Mathematics & Mechanics, Saint Petersburg State University.


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Repository Layout

Reid Spencer-2
On Sun, 2007-07-01 at 03:14 +0400, Anton Korobeynikov wrote:

> Hello, Reid.
>
> > Fortunately, with SVN, making such a change isn't hard and its tracked
> > in the commits. Anyone object or have other ideas in this realm?
> I'm also confused. At one hand, proposed layout (with BTT in the top)
> seems to be "unnatural", when we will create new tags/branches.
>
> In theory, each module can have its own version, thus if the want to
> "tag" it, we have either copy the whole repository to new tag, wasting
> space and checkout times, or "tag" just a subset of the repository,
> which seems to be pretty bad, because different tags can contain
> different "subsets". The same for branches, but not too "bad", since
> branches are more or less temporary. Keeping BTT in the modules will
> surely make such problems thrown away.

Well, I think its actually less confusing and more amenable to doing
both module-specific tags as well as full-project tags. For example, we
could have:

/tags/llvm/release-21

to mark the 2.1 release of llvm. There really isn't any diference in
creating this tag as opposed to /llvm/tag/release_21. Same mechanisms
apply; the only difference is the initial path. We could also have:

/tags/llvm-project-release-10

which could be a tag for the entire project (all modules), something
we've never done before. Note that in svn, copies are cheap.

>
> At the other hand, keeping BTT inside modules can lead to headache, when
> some branch, for example, has dependencies on other modules (for
> example, some experimental version of CFE can be stick to some version
> of LLVM and so on)...

Right, my second example above addresses that.

Reid.

>

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Repository Layout

Gordon Henriksen-3
In reply to this post by Reid Spencer-2
On 2007-06-30, at 18:36, Reid Spencer wrote:

I'm finding the current layout a bit frustrating, not to mention that it breaks all the documentation. [...] I would rather see it laid out like this:

   trunk/llvm/...
   trunk/stacker/...
   trunk/java/...

   branches/llvm/...
   branches/stacker/...
   branches/java/...

   tags/llvm/...
   tags/stacker/...
   tags/java/..

Reid,

This is a common problem with Subversion. I wouldn't recommend using the layout you suggested here, though. Although it's "just" a convention, please keep the standard layout at the root of each project. (You're proposing interceding a directory layer between trunk, branches, and tags, etc. I don't think you'll find that it solves many management problems.)

If the projects are too granular, merge them together. It might, in fact, be appropriate to use a single project root for the totality of llvm. Getting this correct is by far the most important decision to be made, IMO. It's very easy to check out a subdirectory of a project, but much harder to check out a union of several projects.

However, if a union of several projects is necessary, Subversion does have an svn:external property[1] which can be used to create precisely that. It's much more complicated to tag or branch such a beast, however, so it is strongly recommended that such a thing not be used as a unit of release.

Anton, as Reid mentioned, there's absolutely no need to worry about wasting repository space with copies. However, since copies are so incredibly cheap on the server, there is a need to worry about wasting developer hard disk space. At the client, copies are much more expensive with Subversion than with CVS, so your intuition is still very useful.


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Repository Layout

Reid Spencer-2
Hi Gordon,

On Sun, 2007-07-01 at 08:22 -0400, Gordon Henriksen wrote:

> On 2007-06-30, at 18:36, Reid Spencer wrote:
>
>
> > I'm finding the current layout a bit frustrating, not to mention
> > that it breaks all the documentation. [...] I would rather see it
> > laid out like this:
> >
> >
> > >
> >    trunk/llvm/...
> >    trunk/stacker/...
> >    trunk/java/...
> >
> >
> >    branches/llvm/...
> >    branches/stacker/...
> >    branches/java/...
> >
> >
> >    tags/llvm/...
> >    tags/stacker/...
> >    tags/java/..
>
>
> Reid,
>
>
> This is a common problem with Subversion. I wouldn't recommend using
> the layout you suggested here, though. Although it's "just" a
> convention, please keep the standard layout at the root of each
> project. (You're proposing interceding a directory layer between
> trunk, branches, and tags, etc. I don't think you'll find that it
> solves many management problems.)

I wasn't trying to solve any management problems. I see both schemes
(project at top vs. trunk/tags/branches at top) as having basically the
same management issues. What I was trying to achieve was:

a) a single URL for checking out all llvm-project software. Putting
trunk/tags/branches at the top would give us:
http://llvm.org/svn/llvm-project/trunk as the root of all software.
Without this, you would have one root for each sub-projects (llvm,
stacker, java, ...)

b) not having to fix all the documentation that references paths to
insert a "/trunk" into the path.

> If the projects are too granular, merge them together.

Perhaps, but that's not related to the issue at hand.

> It might, in fact, be appropriate to use a single project root for the
> totality of llvm.

We have that: http://llvm.org/svn/llvm-project.

>  Getting this correct is by far the most important decision to be
> made, IMO. It's very easy to check out a subdirectory of a project,
> but much harder to check out a union of several projects.

if tags/trunk/branches was at the top level then getting the "union of
several projects" is achieved by just checking out
"trunk" (http://llvm.org/svn/llvm-project/trunk). Again, I'm not sure
how this is relevant to the original issue. Am I missing something?

> However, if a union of several projects is necessary, Subversion does
> have an svn:external property[1] which can be used to create precisely
> that. It's much more complicated to tag or branch such a beast,
> however, so it is strongly recommended that such a thing not be used
> as a unit of release.

I'm aware of svn:external and I don't think we should be using it for
internal repositories. We *might* want to use it to integrate another
repository, but I know of no such need currently and can't see it
arising in the future.

> Anton, as Reid mentioned, there's absolutely no need to worry about
> wasting repository space with copies. However, since copies are so
> incredibly cheap on the server, there is a need to worry about wasting
> developer hard disk space. At the client, copies are much more
> expensive with Subversion than with CVS, so your intuition is still
> very useful.

Yes, on the client side, copies are definitely more expensive. This just
underscores the need to set up a way to get "trunk" easily and avoid all
the copies in tags and branches. For example, consider doing something
like this:

svn co http://llvm.org/svn/llvm-project

Right now, what that will get is you is the trunk of all projects and
all the tags and branches for all those projects. Clearly this isn't
desirable. For just the llvm sub-project, this would involve checking
out 28 copies of the llvm source code.

If we had trunk/tags/branches at the top then it is much easier to get
the single copy of the latest software with just:

svn co http://llvm.org/svn/llvm-project/trunk 

So, in summary, I'm not quire sure I understand your original caution
about using this approach. What are the management headaches that we
would incur? Are they any higher than putting trunk/tags/branches one
level lower (by having the module name first)?  A little more detail
would be helpful.

Thanks,

Reid.

>
>
> — Gordon
>
>
> [1] http://svnbook.red-bean.com/en/1.0/ch07s03.html
>
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Repository Layout

Gordon Henriksen-3
On Jul 1, 2007, at 4:47 PM, Reid Spencer <[hidden email]>  
wrote:

> On Sun, 2007-07-01 at 08:22 -0400, Gordon Henriksen wrote:
>
>> On 2007-06-30, at 18:36, Reid Spencer wrote:
>>
>>> I'm finding the current layout a bit frustrating, not to mention
>>> that it breaks all the documentation. [...] I would rather see it
>>> laid out like this:
>>>
>>>
>>>>
>>>   trunk/llvm/...
>>>   trunk/stacker/...
>>>   trunk/java/...
>>>
>>>
>>>   branches/llvm/...
>>>   branches/stacker/...
>>>   branches/java/...
>>>
>>>
>>>   tags/llvm/...
>>>   tags/stacker/...
>>>   tags/java/..
>>
>>
>> Reid,
>>
>>
>> This is a common problem with Subversion. I wouldn't recommend using
>> the layout you suggested here, though. Although it's "just" a
>> convention, please keep the standard layout at the root of each
>> project. (You're proposing interceding a directory layer between
>> trunk, branches, and tags, etc. I don't think you'll find that it
>> solves many management problems.)
>
> I wasn't trying to solve any management problems. I see both schemes
> (project at top vs. trunk/tags/branches at top) as having basically  
> the
> same management issues. What I was trying to achieve was:
>
> a) a single URL for checking out all llvm-project software. Putting
> trunk/tags/branches at the top would give us:
> http://llvm.org/svn/llvm-project/trunk as the root of all software.
> Without this, you would have one root for each sub-projects (llvm,
> stacker, java, ...)

But, unless I misunderstand your suggested layout, that benefit would  
not extend to branches or tags, because you placed the project as the  
parent directory of the tag. This makes branches and tags much less  
useful, because they cannot be used symmetrically with the trunk(s) or  
created with a simple svn copy command, should more than one subfolder  
need to be tagged.

>> If the projects are too granular, merge them together.
>
> Perhaps, but that's not related to the issue at hand.

By "project", I was referring solely to the layout in source control,  
not any source code integration. Sorry for the imprecise language. I  
shall use "SVNROOT" instead.

>> It might, in fact, be appropriate to use a single project root for  
>> the
>> totality of llvm.
>
> We have that: http://llvm.org/svn/llvm-project.

As you note, that directory cannot be (practically) checked out  
because it will contain all branches and tags. I'm referring to a  
folder which transparently references other projects as a shortcut for  
checking out all of the trunks manually, precisely the scenario  
described in the Red Book.

>> Getting this correct is by far the most important decision to be
>> made, IMO. It's very easy to check out a subdirectory of a project,
>> but much harder to check out a union of several projects.
>
> if tags/trunk/branches was at the top level then getting the "union of
> several projects" is achieved by just checking out
> "trunk" (http://llvm.org/svn/llvm-project/trunk).

Yes, exactly. But in your proposed layout, that only works for the  
trunk. That's because you flipped the path from SVNROOT/tags/tagname/
folder to SVNROOT/tags/folder/tagname.

> Again, I'm not sure
> how this is relevant to the original issue. Am I missing something?
>
>> However, if a union of several projects is necessary, Subversion does
>> have an svn:external property[1] which can be used to create  
>> precisely
>> that. It's much more complicated to tag or branch such a beast,
>> however, so it is strongly recommended that such a thing not be used
>> as a unit of release.
>
> I'm aware of svn:external and I don't think we should be using it for
> internal repositories. We *might* want to use it to integrate another
> repository, but I know of no such need currently and can't see it
> arising in the future.
>
>> Anton, as Reid mentioned, there's absolutely no need to worry about
>> wasting repository space with copies. However, since copies are so
>> incredibly cheap on the server, there is a need to worry about  
>> wasting
>> developer hard disk space. At the client, copies are much more
>> expensive with Subversion than with CVS, so your intuition is still
>> very useful.
>
> Yes, on the client side, copies are definitely more expensive. This  
> just
> underscores the need to set up a way to get "trunk" easily and avoid  
> all
> the copies in tags and branches. For example, consider doing something
> like this:
>
> svn co http://llvm.org/svn/llvm-project
>
> Right now, what that will get is you is the trunk of all projects and
> all the tags and branches for all those projects. Clearly this isn't
> desirable. For just the llvm sub-project, this would involve checking
> out 28 copies of the llvm source code.
>
> If we had trunk/tags/branches at the top then it is much easier to get
> the single copy of the latest software with just:
>
> svn co http://llvm.org/svn/llvm-project/trunk
>
> So, in summary, I'm not quire sure I understand your original caution
> about using this approach. What are the management headaches that we
> would incur. Are they any higher than putting trunk/tags/branches one
> level lower (by having the module name first)?  A little more detail
> would be helpful.
>>
>>
>>
>>
>>
>>

All I'm saying is this:

1. You'll cause yourself headaches unless the contents of $SVNROOT/
trunk is symmetrical with $SVNROOT/{tags,branches}/$name.

2. It sounds like you would be happier with fewer SVNROOTs.

But you should also consider whether this is simply a growing pain;  
cvs and svn are fundamentally different in that Subversion makes the  
management structure visible in the URL.

- Gordon
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Repository Layout

Chris Lattner
In reply to this post by Reid Spencer-2
On Sat, 30 Jun 2007, Reid Spencer wrote:

> I'm finding the current layout a bit frustrating, not to mention that it
> breaks all the documentation. We currently have modules laid out like
> this:
>   llvm/trunk/...
>   llvm/branches/...
>   llvm/tags/...
>   stacker/trunk/...
>   stacker/branches/...
>   stacker/tags/...
>
> and similarly for all the other modules (test-suite, java, poolalloc,
> etc.). Unfortunately, this "standard" layout interjects a directory
> (trunk/branches/tags) below each module name. I would rather see it laid
> out like this:
>
>   trunk/llvm/...
>   trunk/stacker/...
>   trunk/java/...
> Fortunately, with SVN, making such a change isn't hard and its tracked
> in the commits. Anyone object or have other ideas in this realm?

I strongly dislike this approach.  This is particularly bad because we
have many subprojects.  Lots of these subprojects are only loosly coupled
to LLVM: things like llvm-tv, llvm-java, llvm-poolalloc, etc.

I think that it is natural for subprojects to go through various
development phases and to sink or swim based on their own merits and the
contributor base around them.  Our goal should be to make it as easy as
possible to check out the desired subset of LLVM and get going quickly.

Instead of trying to make a mega project, I suggest something else.  What
I'd really like to see is this:

1. I check out the "llvm-project" into a directory.  This provides some
    very very basic makefiles and configure script stuff.  This does not
    try to solve all the worlds problems, it is just some simple glue.
2. Based on my interests, I check out a subset of the subprojects into
    directories under llvm-project.
3. Each subproject knows the projects it depends on - either as a hard
    dependency (requirement) or a soft dependency (optional component).
4. Once the stuff is checked out, you type "configure".  Based on the glue
    in #1, this checks to ensure that the various dependencies required are
    checked out.  If not, it errors out, telling you what else to check out
    (or offering to do it for you?)
5. make, make test, make install, etc are provided by the glue.  These
    simply recurse into the subdirectories in the appropriate order and
    invoke the respective targets.

With this design, I'd suggest refactoring the LLVM project a little bit.
I'd like to see:

1. a subproject for libsupport and libsystem.
2. a subproject for the rest of the "llvm" project.  sometime later we can
    separate out vmcore perhaps.
3. a subproject for llvm-gcc, which depends on #1 and #2.
4. a subproject for the new c front-end, which has a hard requirement on
    #1 and a optional requirement for #2.
5. HLVM, llvm-tv, etc (the ones other people care about :)

One of the big in-between things is llvm-config.  I've grown to *love*
this tool, but it only works for the main llvm project.  I'd like to see
it move up and works as part of the glue, so that it is easy to
link/include anything that is part of the "llvm-project".

The overall goal is to make the system more modular (NOT more monolithic)
and to make it easier to get stuff started: it should be enough to check
out the llvm-project glue (very very quick checkout) and run "make
get-me-llvm-gcc-and-build-it" :)

As an aside, I've been working on a revamp of the website, to make *it*
more modular, which is also required for the docs.  I think it's
amazing(ly bad) that we don't have a documentation subsection for llvm-gcc
anywhere!

Thoughts?

-Chris

--
http://nondot.org/sabre/
http://llvm.org/
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Repository Layout

Keith Bauer
I think the trunk/subproject layout is inappropriate, since it also
implies branches/name/subprojects, where a branch is likely to apply
only to a single subproject.

how about a top-level llvm-and-kitchen-sink module which simply
contains svn:externals to the trunks of all the subprojects, allowing
a monolithic checkout for those who want it, and separate modules for
better branching and more isolated development?

This would also provide the "easy (but bandwidth consuming) to get
started" module
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Repository Layout

Devang Patel
In reply to this post by Reid Spencer-2
Reid,

On Jun 30, 2007, at 3:36 PM, Reid Spencer wrote:

> All,
>
> I'm finding the current layout a bit frustrating, not to mention  
> that it
> breaks all the documentation. We currently have modules laid out like
> this:
>
>   llvm/trunk/...
>   llvm/branches/...
>   llvm/tags/...
>
>   stacker/trunk/...
>   stacker/branches/...
>   stacker/tags/...
>
> and similarly for all the other modules (test-suite, java, poolalloc,
> etc.). Unfortunately, this "standard" layout interjects a directory
> (trunk/branches/tags) below each module name. I would rather see it  
> laid
> out like this:
>
>   trunk/llvm/...
>   trunk/stacker/...
>   trunk/java/...
>
>   branches/llvm/...
>   branches/stacker/...
>   branches/java/...
>
>   tags/llvm/...
>   tags/stacker/...
>   tags/java/..
>
> Fortunately, with SVN, making such a change isn't hard and its tracked
> in the commits. Anyone object or have other ideas in this realm?

Your suggestion makes life difficult for someone who wants to mirror
one subproject. Once, llvm-gcc is using svn at llvm.org, most likely
I will create a local mirror to follow llvm-gcc development. In that  
case
it is easier to mirror

        .../llvm-gcc

then
       
        ../trunk/llvm-gcc/
        ../branches/llvm-gcc/
        ../tags/llvm-gcc

-
Devang

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Repository Layout

Scott Michel
Devang Patel wrote:
 > Your suggestion makes life difficult for someone who wants to mirror

> one subproject. Once, llvm-gcc is using svn at llvm.org, most likely
> I will create a local mirror to follow llvm-gcc development. In that  
> case
> it is easier to mirror
>
> .../llvm-gcc
>
> then
>
> ../trunk/llvm-gcc/
> ../branches/llvm-gcc/
> ../tags/llvm-gcc

+1 strong agreement from another local mirror-er.


-scooter
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Repository Layout

Reid Spencer-2
Hi Scott,

On Fri, 2007-07-06 at 17:54 -0700, Scott Michel wrote:

> Devang Patel wrote:
>  > Your suggestion makes life difficult for someone who wants to mirror
> > one subproject. Once, llvm-gcc is using svn at llvm.org, most likely
> > I will create a local mirror to follow llvm-gcc development. In that  
> > case
> > it is easier to mirror
> >
> > .../llvm-gcc
> >
> > then
> >
> > ../trunk/llvm-gcc/
> > ../branches/llvm-gcc/
> > ../tags/llvm-gcc

We backed off this plan. Please see http://llvm.org/SVNLayout.html 

Reid.

>
> +1 strong agreement from another local mirror-er.
>
>
> -scooter
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev