[llvm-dev] GN build roundtable summary; adding GN build files to the repo

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
48 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
Hi,

first things first: If you're happy with cmake, you can stop reading now. Nobody is proposing that LLVM moves off cmake, and nobody is proposing anything that's causing people using cmake more work.

At the LLVM conference, I gave a lightning talk [1] about using GN [2] to build LLVM and clang. cmake is great for many use cases, but in my opinion local day-to-day development isn't one of them. So I wrote GN build files for LLVM and clang, enough to make `ninja check-llvm check-clang check-lld` build everything needed for these three test suites and that all tests pass. This works on Linux, Mac, Win hosts targeting X86, ARM, AArch64. You can see them at [3].

I had been worried that it would be a lot of work to keep the build files up to date, but I've been using this for all my LLVM/clang/lld development the last 8 months, and it turned out to not be a big problem -- LLVM's build files don't change very often, and GN build files are a pleasure to work with in my opinion.

I gave the lightning talk just to talk about my personal workflow, but there was a lot of interest. We had a roundtable on the next day, and about 20 people said they'd be interested in using this for their development too. The main request was that the .gn files are checked in upstream, so that we can collaborate on keeping them working. Two to three orgs even said they'd be interested in moving their buildbots to GN.

As mentioned at the top, the intention here is not to replace cmake, only to offer an opt-in alternative for people who are interested in it. Keeping the GN build going would be the responsibility of people using it, not of the general LLVM community. If this fails to find use and bitrots, we can easily remove it again.

Are there any concerns with checking in GN files? I've put some initial docs for the GN build at https://reviews.llvm.org/D53944 ; it describes what the GN build is and is not, what its advantages are (speed and easier configurability), and some points about the philosophy behind the LLVM GN build.

If folks are generally ok with GN files living in-tree, I'll start sending patches for gradually adding gn files through the regular review process.

If having a BUILD.gn file in every directory being confusing is a concern, GN has the concept of a "secondary tree" so that all GN files could be below llvm/gn/tree/{llvm,clang,lld,...}.

Cheers,
Nico


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [cfe-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev


On Wed, Oct 31, 2018, 1:18 PM Nico Weber via cfe-dev <[hidden email]> wrote:
Hi,

first things first: If you're happy with cmake, you can stop reading now. Nobody is proposing that LLVM moves off cmake, and nobody is proposing anything that's causing people using cmake more work.

At the LLVM conference, I gave a lightning talk [1] about using GN [2] to build LLVM and clang. cmake is great for many use cases, but in my opinion local day-to-day development isn't one of them. So I wrote GN build files for LLVM and clang, enough to make `ninja check-llvm check-clang check-lld` build everything needed for these three test suites and that all tests pass. This works on Linux, Mac, Win hosts targeting X86, ARM, AArch64. You can see them at [3].


I'm in favor of including the gn build files. 

Aside: build of gn itself failed for me on Ubuntu 14, link errors related to unsupported relocations, AFAICT this is just an incompatibility between the Debian filesystem downloaded and Ubuntu 14.  Could you provide a release tarball of gn for a couple of key platforms?

-Brian

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [cfe-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
On Wed, Oct 31, 2018 at 11:30 AM Brian Cain <[hidden email]> wrote:


On Wed, Oct 31, 2018, 1:18 PM Nico Weber via cfe-dev <[hidden email]> wrote:
Hi,

first things first: If you're happy with cmake, you can stop reading now. Nobody is proposing that LLVM moves off cmake, and nobody is proposing anything that's causing people using cmake more work.

At the LLVM conference, I gave a lightning talk [1] about using GN [2] to build LLVM and clang. cmake is great for many use cases, but in my opinion local day-to-day development isn't one of them. So I wrote GN build files for LLVM and clang, enough to make `ninja check-llvm check-clang check-lld` build everything needed for these three test suites and that all tests pass. This works on Linux, Mac, Win hosts targeting X86, ARM, AArch64. You can see them at [3].


I'm in favor of including the gn build files. 

Aside: build of gn itself failed for me on Ubuntu 14, link errors related to unsupported relocations, AFAICT this is just an incompatibility between the Debian filesystem downloaded and Ubuntu 14.  Could you provide a release tarball of gn for a couple of key platforms?

We provide prebuilts for GN now, see https://gn.googlesource.com/gn/#getting-started 

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [cfe-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev


On Wed, Oct 31, 2018, 1:32 PM Petr Hosek <[hidden email]> wrote:
On Wed, Oct 31, 2018 at 11:30 AM Brian Cain <[hidden email]> wrote:


On Wed, Oct 31, 2018, 1:18 PM Nico Weber via cfe-dev <[hidden email]> wrote:
Hi,

first things first: If you're happy with cmake, you can stop reading now. Nobody is proposing that LLVM moves off cmake, and nobody is proposing anything that's causing people using cmake more work.

At the LLVM conference, I gave a lightning talk [1] about using GN [2] to build LLVM and clang. cmake is great for many use cases, but in my opinion local day-to-day development isn't one of them. So I wrote GN build files for LLVM and clang, enough to make `ninja check-llvm check-clang check-lld` build everything needed for these three test suites and that all tests pass. This works on Linux, Mac, Win hosts targeting X86, ARM, AArch64. You can see them at [3].


I'm in favor of including the gn build files. 

Aside: build of gn itself failed for me on Ubuntu 14, link errors related to unsupported relocations, AFAICT this is just an incompatibility between the Debian filesystem downloaded and Ubuntu 14.  Could you provide a release tarball of gn for a couple of key platforms?

We provide prebuilts for GN now, see https://gn.googlesource.com/gn/#getting-started 

Oh, I see, that works well. Thanks!

-Brian

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [cfe-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev


> -----Original Message-----
> From: cfe-dev [mailto:[hidden email]] On Behalf Of Stephen
> Kelly via cfe-dev
> Sent: Wednesday, October 31, 2018 4:20 PM
> To: [hidden email]
> Cc: [hidden email]
> Subject: Re: [cfe-dev] GN build roundtable summary; adding GN build files
> to the repo
>
> On 31/10/2018 18:18, Nico Weber via cfe-dev wrote:
> > Keeping the GN build going would be the responsibility of people using
> > it, not of the general LLVM community.
>
> If these files exist in the repo, I could imagine that reviewers will
> reject patches that might/likely break the gn files.
>
> I could also imagine that contributors will be pressured to fix breakage
> of gn files that they introduce, or commits being reverted etc.
>
> So, I'm not sure that a policy of 'it's ok to break this stuff' will
> actually work.

If there are public bots using GN, which I think Nico said some people
wanted to do, that really won't work.  (Private bots that don't complain
to the general user community are a different story.)
--paulr

>
> Do you think there will be some point in time that these files will be
> out of 'experimental' status?
>
> Thanks,
>
> Stephen.
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
On 31.10.18 19:18, Nico Weber via llvm-dev wrote:

> first things first: If you're happy with cmake, you can stop reading
> now. Nobody is proposing that LLVM moves off cmake, and nobody is
> proposing anything that's causing people using cmake more work.
>
> At the LLVM conference, I gave a lightning talk [1] about using GN [2]
> to build LLVM and clang. cmake is great for many use cases, but in my
> opinion local day-to-day development isn't one of them. So I wrote GN
> build files for LLVM and clang, enough to make `ninja check-llvm
> check-clang check-lld` build everything needed for these three test
> suites and that all tests pass. This works on Linux, Mac, Win hosts
> targeting X86, ARM, AArch64. You can see them at [3].
>
> I had been worried that it would be a lot of work to keep the build
> files up to date, but I've been using this for all my LLVM/clang/lld
> development the last 8 months, and it turned out to not be a big problem
> -- LLVM's build files don't change very often, and GN build files are a
> pleasure to work with in my opinion.
>
> I gave the lightning talk just to talk about my personal workflow, but
> there was a lot of interest. We had a roundtable on the next day, and
> about 20 people said they'd be interested in using this for their
> development too. The main request was that the .gn files are checked in
> upstream, so that we can collaborate on keeping them working. Two to
> three orgs even said they'd be interested in moving their buildbots to GN.
>
> As mentioned at the top, the intention here is not to replace cmake,
> only to offer an opt-in alternative for people who are interested in it.
> Keeping the GN build going would be the responsibility of people using
> it, not of the general LLVM community. If this fails to find use and
> bitrots, we can easily remove it again.
>
> Are there any concerns with checking in GN files? I've put some initial
> docs for the GN build at https://reviews.llvm.org/D53944 ; it describes
> what the GN build is and is not, what its advantages are (speed and
> easier configurability), and some points about the philosophy behind the
> LLVM GN build.
>
> If folks are generally ok with GN files living in-tree, I'll start
> sending patches for gradually adding gn files through the regular review
> process.
>
> If having a BUILD.gn file in every directory being confusing is a
> concern, GN has the concept of a "secondary tree" so that all GN files
> could be below llvm/gn/tree/{llvm,clang,lld,...}.

So maintain it in a separate repository.

Seriously, I am also working on another project (Mesa) which has a
proliferation of build systems for historical and cultural (Android...)
reasons, and it sucks.

Besides, how on earth are you using LLVM that cmake times are an issue?
Given that cmake times seem to be the only argument you have in favor of
GN...

I just measured the typical case (for LLVM only, debug build, all
non-experimental targets enabled) by touching all files. cmake takes 5
seconds to run, while the build as a whole takes about 5 minutes. (This
is on Ubuntu 18.10, in case that matters.)

Cheers,
Nicolai



>
> Cheers,
> Nico
>
> 1: https://llvm.org/devmtg/2018-10/talk-abstracts.html#lt2
> 2: https://gn.googlesource.com/gn , https://is.gd/gn_intro
> 3:
> https://github.com/llvm-project/llvm-project-20170507/compare/master...nico:gn 
> , click "Files Changed" to see the GN files.
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

--
Lerne, wie die Welt wirklich ist,
Aber vergiss niemals, wie sie sein sollte.
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev

On Wed, Oct 31, 2018 at 3:11 PM Nicolai Hähnle via llvm-dev <[hidden email]> wrote:


Besides, how on earth are you using LLVM that cmake times are an issue?
Given that cmake times seem to be the only argument you have in favor of
GN...

I just measured the typical case (for LLVM only, debug build, all
non-experimental targets enabled) by touching all files. cmake takes 5
seconds to run, while the build as a whole takes about 5 minutes. (This
is on Ubuntu 18.10, in case that matters.)

I am using CMake with LLVM with all targets OFF except for x86.  Only clang, lldb, and lld enabled.

$ timecmd cmake ..\..\..\..\llvm-mono\llvm
<output snipped>
command took 0:0:34.05 (34.05s total)



_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
And by the way, that is with my cache already populated.  Here it is on a fresh build directory where it has to generate everything.  (this time release build)

$ timecmd cmake -G Ninja -DLLVM_ENABLE_PROJECTS=clang;lld;lldb -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=Release ..\..\..\..\llvm-mono\llvm
<output snipped>
command took 0:0:57.42 (57.42s total)

This is on a 48-core machine btw, with an SSD.

On Wed, Oct 31, 2018 at 3:29 PM Zachary Turner <[hidden email]> wrote:
On Wed, Oct 31, 2018 at 3:11 PM Nicolai Hähnle via llvm-dev <[hidden email]> wrote:


Besides, how on earth are you using LLVM that cmake times are an issue?
Given that cmake times seem to be the only argument you have in favor of
GN...

I just measured the typical case (for LLVM only, debug build, all
non-experimental targets enabled) by touching all files. cmake takes 5
seconds to run, while the build as a whole takes about 5 minutes. (This
is on Ubuntu 18.10, in case that matters.)

I am using CMake with LLVM with all targets OFF except for x86.  Only clang, lldb, and lld enabled.

$ timecmd cmake ..\..\..\..\llvm-mono\llvm
<output snipped>
command took 0:0:34.05 (34.05s total)



_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
FWIW, this is consistent with my experience as well.

I also wrote a presubmit script that I run locally to build/test the cross-product of:

COMPILER={gcc+gold, clang+lld}
MODE={debug, release-with-assertions}
TARGETS={llvm+clang+lld+compiler-rt, compiler-rt(standalone)}

While the build/test times are helped by using ninja, the CMake config times are less than optimal.

It’s bad enough that I think I’ll give gn a whirl if the files were in the project(s) too.

> On 1 Nov 2018, at 09:33, Zachary Turner via llvm-dev <[hidden email]> wrote:
>
> And by the way, that is with my cache already populated.  Here it is on a fresh build directory where it has to generate everything.  (this time release build)
>
> $ timecmd cmake -G Ninja -DLLVM_ENABLE_PROJECTS=clang;lld;lldb -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=Release ..\..\..\..\llvm-mono\llvm
> <output snipped>
> command took 0:0:57.42 (57.42s total)
>
> This is on a 48-core machine btw, with an SSD.
>
> On Wed, Oct 31, 2018 at 3:29 PM Zachary Turner <[hidden email]> wrote:
> On Wed, Oct 31, 2018 at 3:11 PM Nicolai Hähnle via llvm-dev <[hidden email]> wrote:
>
>
> Besides, how on earth are you using LLVM that cmake times are an issue?
> Given that cmake times seem to be the only argument you have in favor of
> GN...
>
> I just measured the typical case (for LLVM only, debug build, all
> non-experimental targets enabled) by touching all files. cmake takes 5
> seconds to run, while the build as a whole takes about 5 minutes. (This
> is on Ubuntu 18.10, in case that matters.)
>
> I am using CMake with LLVM with all targets OFF except for x86.  Only clang, lldb, and lld enabled.
>
> $ timecmd cmake ..\..\..\..\llvm-mono\llvm
> <output snipped>
> command took 0:0:34.05 (34.05s total)
>
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- Dean

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
On 31/10/2018 22:44, Dean Michael Berris via llvm-dev wrote:
> FWIW, this is consistent with my experience as well.
>
> I also wrote a presubmit script that I run locally to build/test the cross-product of:
>
> COMPILER={gcc+gold, clang+lld}
> MODE={debug, release-with-assertions}
> TARGETS={llvm+clang+lld+compiler-rt, compiler-rt(standalone)}
>
> While the build/test times are helped by using ninja, the CMake config times are less than optimal.


It is possible that we have features/complexity in our CMakeLists.txt
which is not needed. The CMake files are not using modern cmake
features. Maybe it's time to change that. It might simplify the
buildsystem a bit.

Thanks,

Stephen.

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
I'll volunteer to maintain the GN files if the community is in favor
of having those in the tree. I mentioned that I'd consider using GN
build on bots (and Nico said the same for Chromium) during the
roundtable discussion, but what I meant was for our downstream bots.
If there are upstream bots using GN, they should be FYI only bots (I
don't know if there are any such bots in LLVM already) that don't send
out any notifications on breakages.

There are other advantages to GN but those are not immediately
obvious. GN has a first-class concept of a toolchain and it allows
"applying" these toolchains on dependencies, so you can express the
fact that a particular dependency should be built with a different
toolchain, which could be a completely different compiler or just a
different set of flags. This is an extremely powerful concept that
would be useful for things like runtimes build. Right now, those
require re-running CMake multiple times for each target you want to
build runtimes for, so take the number from above responses and
multiply that by the number of targets you're supporting in your
toolchain and suddenly it may become significant (it definitely is in
our case).

Regarding the experimental status, I think we need to get the GN build
to the point where it provides parity with the CMake build and handles
all common use cases and workflows before we can have that discussion.
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
That is certainly possible, however if we're going to re-write the entire build system anyway.......

One thing to think about is that generated IDE projects from CMake are less than ideal.  In fact I actually made a post about this a few weeks ago.  In MSVC I consider them barely usable.  I can edit files and get code completion with them, but that's about it.  Navigation is painful at best (can take several up to 10 seconds just to open a file), and everything is extremely slow because MSVC is churning away trying to process its build dependency graph.  Occasionally things just stop working for 10-20 seconds at a time.

I've heard that Xcode is also barely usable but can't speak for this myself.

Is this any better with gn-generated IDE projects?  I actually don't know!  Because I've never used them.  But it's at least something to think about.  

On Wed, Oct 31, 2018 at 4:21 PM Stephen Kelly via llvm-dev <[hidden email]> wrote:
On 31/10/2018 22:44, Dean Michael Berris via llvm-dev wrote:
> FWIW, this is consistent with my experience as well.
>
> I also wrote a presubmit script that I run locally to build/test the cross-product of:
>
> COMPILER={gcc+gold, clang+lld}
> MODE={debug, release-with-assertions}
> TARGETS={llvm+clang+lld+compiler-rt, compiler-rt(standalone)}
>
> While the build/test times are helped by using ninja, the CMake config times are less than optimal.


It is possible that we have features/complexity in our CMakeLists.txt
which is not needed. The CMake files are not using modern cmake
features. Maybe it's time to change that. It might simplify the
buildsystem a bit.

Thanks,

Stephen.

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev


> On 1 Nov 2018, at 10:24, Petr Hosek via llvm-dev <[hidden email]> wrote:
>
> I'll volunteer to maintain the GN files if the community is in favor
> of having those in the tree. I mentioned that I'd consider using GN
> build on bots (and Nico said the same for Chromium) during the
> roundtable discussion, but what I meant was for our downstream bots.
> If there are upstream bots using GN, they should be FYI only bots (I
> don't know if there are any such bots in LLVM already) that don't send
> out any notifications on breakages.
>
> There are other advantages to GN but those are not immediately
> obvious. GN has a first-class concept of a toolchain and it allows
> "applying" these toolchains on dependencies, so you can express the
> fact that a particular dependency should be built with a different
> toolchain, which could be a completely different compiler or just a
> different set of flags. This is an extremely powerful concept that
> would be useful for things like runtimes build. Right now, those
> require re-running CMake multiple times for each target you want to
> build runtimes for, so take the number from above responses and
> multiply that by the number of targets you're supporting in your
> toolchain and suddenly it may become significant (it definitely is in
> our case).

This.

Although I don’t use GN myself, it would be great if we can somehow port the compiler-rt unit-test generation process to use the “just-recently-built toolchain”, it would break up the non-parallelism of those actions.

>
> Regarding the experimental status, I think we need to get the GN build
> to the point where it provides parity with the CMake build and handles
> all common use cases and workflows before we can have that discussion.

+1 from me.

-- Dean

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev


On Wed, 31 Oct 2018, 23:28 Zachary Turner <[hidden email] wrote:
That is certainly possible, however if we're going to re-write the entire build system anyway.......

One thing to think about is that generated IDE projects from CMake are less than ideal.  In fact I actually made a post about this a few weeks ago.  In MSVC I consider them barely usable.  I can edit files and get code completion with them, but that's about it.  Navigation is painful at best (can take several up to 10 seconds just to open a file), and everything is extremely slow because MSVC is churning away trying to process its build dependency graph.  Occasionally things just stop working for 10-20 seconds at a time.


One of the advantages of clang being in a separate repo which can be built independent of llvm is that both are then in smaller solutions. The clang build would treat llvm as an external dependency and not have it's cpp files in the solution. 

I know we're moving to a monorepo and VS solution size is not part of that consideration, so that size reduction won't be possible anymore.

I don't know if there are other ways to reduce solution size. I also haven't hit performance problems related to VS, though I only build llvm/clang/cte on windows, I don't do development on it.

I can see if I can reproduce the kinds of problems you see tomorrow. Can you give me a got URL and cake command line to repro exactly your experience?

Thanks,

Stephen. 


I've heard that Xcode is also barely usable but can't speak for this myself.

Is this any better with gn-generated IDE projects?  I actually don't know!  Because I've never used them.  But it's at least something to think about.  

On Wed, Oct 31, 2018 at 4:21 PM Stephen Kelly via llvm-dev <[hidden email]> wrote:
On 31/10/2018 22:44, Dean Michael Berris via llvm-dev wrote:
> FWIW, this is consistent with my experience as well.
>
> I also wrote a presubmit script that I run locally to build/test the cross-product of:
>
> COMPILER={gcc+gold, clang+lld}
> MODE={debug, release-with-assertions}
> TARGETS={llvm+clang+lld+compiler-rt, compiler-rt(standalone)}
>
> While the build/test times are helped by using ninja, the CMake config times are less than optimal.


It is possible that we have features/complexity in our CMakeLists.txt
which is not needed. The CMake files are not using modern cmake
features. Maybe it's time to change that. It might simplify the
buildsystem a bit.

Thanks,

Stephen.

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
On Wed, Oct 31, 2018 at 4:24 PM Petr Hosek via llvm-dev <[hidden email]> wrote:
I'll volunteer to maintain the GN files if the community is in favor
of having those in the tree.
... 
GN has a first-class concept of a toolchain and it allows
"applying" these toolchains on dependencies, so you can express the
fact that a particular dependency should be built with a different
toolchain, which could be a completely different compiler or just a
different set of flags. This is an extremely powerful concept that
would be useful for things like runtimes build.

I think your use case and Nico's are a bit different. In the fullness of time, I expect that you will show that gn's toolchain support works really well, and more people will want to use it for similar reasons. Eventually, consensus will grow that it should be an officially supported build system. I don't doubt that you are volunteering to maintain it yourself for now, but I want to acknowledge that there is a very real possibility of scope creep. If runtimes are in scope for this, then that really means the gn files are going to be a full second build system with support for making a real toolchain package. At some point, we may wind up with a second system just as complicated as cmake, although it'll be much faster.

After writing the paragraph above, I realized that one of the big costs of cmake is iteration time. If you want to fix a build system problem today, you have to run cmake as part of your inner development loop. This incentivizes people to avoid touching the build system if at all possible, and it fills up with little quick fixes that nobody wants to refactor. If we were using gn, I would be much happier iterating on changes to the build system, since I would be able to test small incremental changes quickly.

I guess my opinion is, LLVM is big enough to have a second build system in tree, with all the costs that that implies. gn is really the only new build system that I'm actually excited about, so if we're going to have a second one, it should be this one. 

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev


> On 1 Nov 2018, at 10:21, Stephen Kelly via llvm-dev <[hidden email]> wrote:
>
> On 31/10/2018 22:44, Dean Michael Berris via llvm-dev wrote:
>> FWIW, this is consistent with my experience as well.
>> I also wrote a presubmit script that I run locally to build/test the cross-product of:
>> COMPILER={gcc+gold, clang+lld}
>> MODE={debug, release-with-assertions}
>> TARGETS={llvm+clang+lld+compiler-rt, compiler-rt(standalone)}
>> While the build/test times are helped by using ninja, the CMake config times are less than optimal.
>
>
> It is possible that we have features/complexity in our CMakeLists.txt which is not needed. The CMake files are not using modern cmake features. Maybe it's time to change that. It might simplify the buildsystem a bit.
>

I think this might be easier when we finish the move to the monorepo, and can better coordinate/simplify the build across projects. I certainly think it’s worth doing, even if we end up with GN and CMake files in the project.

I’m personally not aware of the “modern” CMake patterns. Do you have a reference online that interested individuals might be able to casually approach?

Cheers

-- Dean

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev


On Wed, 31 Oct 2018, 23:43 Dean Michael Berris <[hidden email] wrote:


> On 1 Nov 2018, at 10:21, Stephen Kelly via llvm-dev <[hidden email]> wrote:
>
> On 31/10/2018 22:44, Dean Michael Berris via llvm-dev wrote:
>> FWIW, this is consistent with my experience as well.
>> I also wrote a presubmit script that I run locally to build/test the cross-product of:
>> COMPILER={gcc+gold, clang+lld}
>> MODE={debug, release-with-assertions}
>> TARGETS={llvm+clang+lld+compiler-rt, compiler-rt(standalone)}
>> While the build/test times are helped by using ninja, the CMake config times are less than optimal.
>
>
> It is possible that we have features/complexity in our CMakeLists.txt which is not needed. The CMake files are not using modern cmake features. Maybe it's time to change that. It might simplify the buildsystem a bit.
>

I think this might be easier when we finish the move to the monorepo, and can better coordinate/simplify the build across projects. I certainly think it’s worth doing, even if we end up with GN and CMake files in the project.

I’m personally not aware of the “modern” CMake patterns. Do you have a reference online that interested individuals might be able to casually approach?

Here is a talk with unfortunate audio quality:



And documentation:


The big topic that llvm does not take advantage of is usage requirements on targets as documented there. 

Thanks,

Stephen. 


Cheers

-- Dean


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev


> On 1 Nov 2018, at 10:47, Stephen Kelly <[hidden email]> wrote:
>
>
>
> On Wed, 31 Oct 2018, 23:43 Dean Michael Berris <[hidden email] wrote:
>
>
> > On 1 Nov 2018, at 10:21, Stephen Kelly via llvm-dev <[hidden email]> wrote:
> >
> > On 31/10/2018 22:44, Dean Michael Berris via llvm-dev wrote:
> >> FWIW, this is consistent with my experience as well.
> >> I also wrote a presubmit script that I run locally to build/test the cross-product of:
> >> COMPILER={gcc+gold, clang+lld}
> >> MODE={debug, release-with-assertions}
> >> TARGETS={llvm+clang+lld+compiler-rt, compiler-rt(standalone)}
> >> While the build/test times are helped by using ninja, the CMake config times are less than optimal.
> >
> >
> > It is possible that we have features/complexity in our CMakeLists.txt which is not needed. The CMake files are not using modern cmake features. Maybe it's time to change that. It might simplify the buildsystem a bit.
> >
>
> I think this might be easier when we finish the move to the monorepo, and can better coordinate/simplify the build across projects. I certainly think it’s worth doing, even if we end up with GN and CMake files in the project.
>
> I’m personally not aware of the “modern” CMake patterns. Do you have a reference online that interested individuals might be able to casually approach?
>
> Here is a talk with unfortunate audio quality:
>
> https://steveire.wordpress.com/2017/11/05/embracing-modern-cmake
>
>

Thanks — I’ll try and listen/watch through this in the background.

> And documentation:
>
> https://cmake.org/cmake/help/latest/manual/cmake-buildsystem.7.html
>
> The big topic that llvm does not take advantage of is usage requirements on targets as documented there.
>

Interesting.

I don’t see anything here that’s popping out to indicate that there’s something new/modern compared to the CMake files I’ve read/written before.

I may need to read more about this.

Thanks for the links!

Cheers

-- Dean

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
Hello!

On 10/31/18 7:18 PM, Nico Weber via llvm-dev wrote:
> At the LLVM conference, I gave a lightning talk [1] about using GN [2] to build LLVM and clang. cmake is great for many use cases, but in my opinion local day-to-day development isn't one of them. So I wrote GN build files for LLVM and clang, enough to make `ninja check-llvm check-clang check-lld` build everything needed for these three test suites and that all tests pass. This works on Linux, Mac, Win hosts targeting X86, ARM, AArch64. You can see them at [3].
> (...)
> Are there any concerns with checking in GN files? I've put some initial docs for the GN build at https://reviews.llvm.org/D53944 ; it describes what the GN build is and is not, what its advantages are (speed and easier configurability), and some points about the philosophy behind the LLVM GN build.

My personal experience with GN - last time I tested it - was that it was impossible to
build from source as the build procedure was anything but straight-forward and pretty
much architecture-locked in.

And I just tried it again on a POWER machine:

glaubitz@redpanda:/srv/glaubitz/gn$ python build/gen.py
Installing Debian root image from https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/1015a998c2adf188813cca60b558b0ea1a0b6ced/debian_sid_amd64_sysroot.tar.xz
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/1015a998c2adf188813cca60b558b0ea1a0b6ced/debian_sid_amd64_sysroot.tar.xz
glaubitz@redpanda:/srv/glaubitz/gn$

Jepp, hasn't changed. If building your build system involves downloading a Debian
chroot with a hard-coded architecture, you're doing something wrong. And from my personal
experience with Google upstream, you can outright forget that they would accept patches
for any target that they are not using themselves.

I really wouldn't recommend using GN. It might be fast, but the whole design seems
to be tailored to Google's own needs.

Adrian

--
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - [hidden email]
`. `'   Freie Universitaet Berlin - [hidden email]
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] GN build roundtable summary; adding GN build files to the repo

David Jones via llvm-dev
On 11/1/18 1:10 AM, John Paul Adrian Glaubitz via llvm-dev wrote:
> Jepp, hasn't changed. If building your build system involves downloading a Debian
> chroot with a hard-coded architecture, you're doing something wrong. And from my personal
> experience with Google upstream, you can outright forget that they would accept patches
> for any target that they are not using themselves.

Oh, and btw, any serious Linux distribution would outright reject to package
something that involves downloading another chroot to be able to build it,
build daemons are - by definition - offline.

Adrian

--
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - [hidden email]
`. `'   Freie Universitaet Berlin - [hidden email]
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
123