make dist?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

make dist?

Tanya Lattner-2

Reid,

Could you explain in detail what make dist does? :) I'd like to see how it
can be integrated into the release process.

Thanks,
Tanya

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: make dist?

Reid Spencer
Hi Tanya,

The first thing you need to understand is that there are multiple make targets to support this feature. I'll briefly describe each here so you have an overview and then delve into the details later.
  • distdir - builds the distribution directory from which the distribution will be packaged
  • dist - builds each of the distribution tarballs (tar.gz, tar.bzip2, .zip). These can be built individually as well, with separate targets.
  • dist-check - this is identical to "dist" but includes a check on the distribution that ensures the tarball can: be unpacked, compiles correctly, passes 'make check', passes 'make clean'.
  • dist-clean - this just does a normal clean but also cleans up the stuff generated by the other three "dist" targets (above).
Okay, that's the basic functionality. When making a release, we want to ensure that the tree you build the distribution from passes "dist-check".  Beyond fixing the usual bugs, there is generally one impediment to making the release in this fashion: missing files.  The "dist-check" process guards against that possibility. It will either fail and that failure will indicate what's missing, or it will succeed meaning that it has proved that the tarballs can actually succeed in building LLVM correctly and that it passes "make check". 


distdir

This target builds the distribution directory which is the directory from which the tarballs are generated. The distribution directory has the same name as the release, e.g. LLVM-1.7).  This target goes through he following process:
  1. First, if there was an old distribution directory (for the current release), it is removed in its entirety and you see "Removing old LLVM-1.7"
  2. Second, it issues a "make all ENABLE_OPTIMIZED=1" to ensure that the everything in your tree can be built in release mode. Often times there are discrepancies in building between debug and release modes so it enforces release mode first.  If that fails, the distdir target fails too. This is preceded by the message "Making 'all' to verify build".
  3. Next, it traverses your source tree and copies it to a new directory that has the name of the release (LLVM-1.7 in our current case). This is the directory that will get tar'd. It contains all the software that needs to be in the distribution.  During the copying process, it omits generated files, CVS directories, and any other "cruft" that's in your build tree. This is done to eliminate the possibility of huge distribution tarballs that include useless or irrelevant stuff in them.  This is the trickiest part of making the distribution. Done manually you will either include stuff that shouldn't be in the distribution or exclude stuff that should. This step is preceded by the message "Building Distribution Directory LLVM-1.7"
  4. The distribution directory is then traversed and all CVS or .svn directories are removed. You see: "Eliminating CVS/.svn directories from distribution"
  5. The recursive dist-hook target is executed. This gives each directory a chance to modify the distribution in some way (more on this below).
  6. The distribution directory is traversed and the correct file permissions and modes are set based on the type of file.
To control the process of making the distribution directory correctly, each Makefile can utilize two features:
  1. EXTRA_DIST - this make variable specifies which files it should distribute. By default, all source files are automatically included for distribution as well as certain "well known" files (see DistAlways variable in Makefile.rules for details).  Each Makefile specifies, via the EXTRA_DIST variable, which additional files need to be distributed.  Only those files that are needed to build LLVM should be added to EXTRA_DISTEXTRA_DIST contains a list of file or directory names that should be distributed. For example, the top level Makefile contains "EXTRA_DIST := test llvm.spec include". This means that in addition to regular things that are distributed at the top level (CREDITS.txt, LICENSE.txt, etc.) the distribution should contain the entire test and include directories as well as the llvm.spec file.
  2. dist-hook - this make target can be used to alter the content of the distribution directory.  For example, in the top level Makefile there is some logic to eliminate files in the include subtree that are generated by the configure script. These should not be distributed. Similarly, any dist-hook target found in any directory can add or remove or modify things just before it gets packaged. Any transformation is permitted. Generally, not much is needed.
You will see various messages if things go wrong:
  1. During the copying process, any files that are missing will be flagged with: "===== WARNING: Distribution Source 'dir/file' Not Found!" These must be corrected by either adding the file or removing it from EXTRA_DIST.
  2. If you build the distribution with VERBOSE=1, then you might also see: "Skipping non-existent 'dir/file'" in certain cases where its okay to skip the file.
  3. The target can fail if any of the things it does fail .. error messages should indicate what went wrong.

dist

This target does exactly what distdir target does, but also includes assembling the tarballs. There are actually four related targets here:
  • dist-gzip: package the gzipped distribution tar file. The distribution directory is packaged into a single file ending in .tar.gz which is gzip compressed.
  • dist-bzip2: package the bzip2 distribution tar file. The distribution directory is packaged into a single file ending in .tar.bzip2 which is bzip2 compressed.
  • dist-zip: package the zip distribution file. The distribution directory is packaged into a single file ending in .zip which is zip compressed.
  • dist: does all three, dist-gzip, dist-bzip2, dist-zip

dist-check

This target checks the distribution. The basic idea is that it unpacks the distribution tarball and ensures that it can build. It takes the following actions:
  1. It depends on the dist-gzip target which, if it hasn't already been built, builds the gzip tar bundle (see dist and distdir above)
  2. removes any pre-existing _distcheckdir at the top level.
  3. creates a new _distcheckdir directory at the top level
  4. creates a build subdirectory and an install subdirectory under _distcheckdir
  5. unzips and untars the release tarball into _distcheckdir , creating LLVM-1.7 directory (from the tarball)
  6. in the build subdirectory, it configures with appropriate options to build from the unpacked source tarball into the build directory with installation in the install directory
  7. runs make all
  8. runs make check
  9. runs make install
  10. runs make uninstall
  11. runs make dist
  12. runs make clean
  13. runs make dist-clean
If it can pass all that, the distribution will be deemed distribution worthy and you will see:
===== LLVM-1.7.tar.gz Ready For Distribution =====

This means the tarball should then be tested on other platforms and have the nightly test run against it. If those all pass, THEN its ready for distribution.

A note about disk space: using "dist-check" will easily triple the amount of disk space your build tree is using. Might want to check available space before you being :)

dist-clean

In addition to doing a normal "clean", this target will clean up the files and directories created by the distribution targets. In particular the distribution directory (LLVM-X.X), check directory (_distcheckdir), and the various tarballs will be removed.  You do this after the release has shipped and you no longer need this stuff in your build tree.


I hope this brief tutorial has helped.  Feel free to ask questions.  Further details can be found in Makefile.rules.

Reid.

On Tue, 2006-04-11 at 13:07 -0500, Tanya Lattner wrote:
Reid,

Could you explain in detail what make dist does? :) I'd like to see how it 
can be integrated into the release process.

Thanks,
Tanya

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: make dist?

Ralph Corderoy

Hi,

> I hope this brief tutorial has helped.  Feel free to ask questions.
> Further details can be found in Makefile.rules.

Although I don't think it uses automake, chapter 13 of
http://sources.redhat.com/autobook/autobook/autobook_toc.html may still
be useful since it gives a similar overview.

Cheers,


Ralph.


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: make dist?

Reid Spencer
The RedHat system uses automake and it is no coincidence that the
facility built for LLVM and the one provided by automake are similar. I
used automake for many years and the idea for the various "dist" targets
comes directly from automake.  We've toyed with using automake in LLVM,
but it has not yet matched up with our diverse needs.  

Thanks for the pointer to redhat's documentation.

Reid.

In Tue, 2006-04-11 at 23:07 +0100, Ralph Corderoy wrote:

> Hi,
>
> > I hope this brief tutorial has helped.  Feel free to ask questions.
> > Further details can be found in Makefile.rules.
>
> Although I don't think it uses automake, chapter 13 of
> http://sources.redhat.com/autobook/autobook/autobook_toc.html may still
> be useful since it gives a similar overview.
>
> Cheers,
>
>
> Ralph.
>
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: make dist?

Tanya Lattner-2
In reply to this post by Reid Spencer

Reid,

First, thanks for the excellent overview. This seems like it will be very
useful.

I have a question about dist-check.

You mention that it will triple the amount of disk space I am using. I am
guessing its because it does the following:
>    11. runs make dist
>    12. runs make clean
>    13. runs make dist-clean

It it necessary for dist-check to run make dist? Why would we want to make
sure that our distribution can create a distribution? If this is not
really necessarily, that would cut down the amount of disk space used..
correct?

-Tanya

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: make dist?

Reid Spencer
On Tue, 2006-04-11 at 21:03 -0500, Tanya Lattner wrote:

> Reid,
>
> First, thanks for the excellent overview. This seems like it will be very
> useful.
>
> I have a question about dist-check.
>
> You mention that it will triple the amount of disk space I am using. I am
> guessing its because it does the following:
> >    11. runs make dist
> >    12. runs make clean
> >    13. runs make dist-clean
>
> It it necessary for dist-check to run make dist?
Y'know, when I typed that, I knew you were going to ask that. It doesn't
make sense to me either. Seems a bit overkillish to me.  

> Why would we want to make
> sure that our distribution can create a distribution?

I think when I put it in Makefile.rules originally I was wanting to
ensure that all the targets would be buildable, just to make sure.

> If this is not
> really necessarily, that would cut down the amount of disk space used..
> correct?

Yes, it would .. by one third.  I'll take it out.

Reid.

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: make dist?

Ralph Corderoy

Hi Reid,

> On Tue, 2006-04-11 at 21:03 -0500, Tanya Lattner wrote:
> > It it necessary for dist-check to run make dist?
>
> Y'know, when I typed that, I knew you were going to ask that. It
> doesn't make sense to me either. Seems a bit overkillish to me.  

I think a `make distcheck' should do a `make dist' because it checks
that a tarball can reproduce itself.  It doesn't ensure that a file
checked-out of CVS makes it into the tarball, but if `make dist' depends
on ./foo being present and it's there from CVS the first time, but
doesn't make it into the tarball then it won't be there for distcheck's
build of the dist target.

I too have rolled by own automake-style targets in the past, and having
distcheck do a dist has caught problems.  That's probably why you
included it  ;-)

> > If this is not really necessarily, that would cut down the amount of
> > disk space used..  correct?
>
> Yes, it would .. by one third.  I'll take it out.

Would the saving be that great?  The `make dist' is just copying source
files and tarring them up whereas the bulk of LLVM seems to be in the
object files produced when building.

Cheers,


Ralph.


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: make dist?

Reid Spencer
Good point. Anyone have a contrary opinion?

Reid

On Fri, 2006-04-14 at 18:23 +0100, Ralph Corderoy wrote:

> Hi Reid,
>
> > On Tue, 2006-04-11 at 21:03 -0500, Tanya Lattner wrote:
> > > It it necessary for dist-check to run make dist?
> >
> > Y'know, when I typed that, I knew you were going to ask that. It
> > doesn't make sense to me either. Seems a bit overkillish to me.  
>
> I think a `make distcheck' should do a `make dist' because it checks
> that a tarball can reproduce itself.  It doesn't ensure that a file
> checked-out of CVS makes it into the tarball, but if `make dist' depends
> on ./foo being present and it's there from CVS the first time, but
> doesn't make it into the tarball then it won't be there for distcheck's
> build of the dist target.
>
> I too have rolled by own automake-style targets in the past, and having
> distcheck do a dist has caught problems.  That's probably why you
> included it  ;-)
>
> > > If this is not really necessarily, that would cut down the amount of
> > > disk space used..  correct?
> >
> > Yes, it would .. by one third.  I'll take it out.
>
> Would the saving be that great?  The `make dist' is just copying source
> files and tarring them up whereas the bulk of LLVM seems to be in the
> object files produced when building.
>
> Cheers,
>
>
> Ralph.
>
>

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

signature.asc (196 bytes) Download Attachment