[RFC] Road map for CMake

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[RFC] Road map for CMake

Chris Bieneman
Hi LLVMDev,

I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).

The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.

(1) Reconcile out-of-tree functionality

Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.

D11083 [CMake] Add experimental support for building compiler-rt for iOS
D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
D11073 Architectures for darwin need to be conditionalized based on the operating system.

One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.

(2) Build the new Darwin build behavior

One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.

The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.

There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.

(3) Make compiler-rt an external project when built in-tree with Clang

Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.

This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.

Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.

(4) Celebrate!

That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.

So... what's next?

With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.

Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.

Questions, comments, concerns, condolences?

Thanks,
-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Eric Christopher


On Wed, Jul 29, 2015 at 1:39 PM Chris Bieneman <[hidden email]> wrote:
Hi LLVMDev,

I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).


Excellent email...
 
Questions, comments, concerns, condolences?


Condolences for sure. I appreciate all the work you've been doing here and thanks!

-eric
 

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Tom Stellard-3
In reply to this post by Chris Bieneman
On Wed, Jul 29, 2015 at 01:36:43PM -0700, Chris Bieneman wrote:

> Hi LLVMDev,
>
> I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).
>
> The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.
>
> (1) Reconcile out-of-tree functionality
>
> Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.
>
> D11083 [CMake] Add experimental support for building compiler-rt for iOS
> D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
> D11073 Architectures for darwin need to be conditionalized based on the operating system.
>
> One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.
>
> (2) Build the new Darwin build behavior
>
> One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.
>
> The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.
>
> There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.
>
> (3) Make compiler-rt an external project when built in-tree with Clang
>
> Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.
>
> This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.
>
> Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.
>
> (4) Celebrate!
>
> That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.
>
> So... what's next?
>
> With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.
>
> Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.
>
> Questions, comments, concerns, condolences?
>

Are we close enough to fixing all the CMake bugs that we can mark autoconf
as deprecated now and plan to remove it after the 3.8 release?

-Tom

> Thanks,
> -Chris
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Bob Wilson-3

On Jul 29, 2015, at 7:03 PM, Tom Stellard <[hidden email]> wrote:

On Wed, Jul 29, 2015 at 01:36:43PM -0700, Chris Bieneman wrote:
Hi LLVMDev,

I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).

The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.

(1) Reconcile out-of-tree functionality

Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.

D11083 [CMake] Add experimental support for building compiler-rt for iOS
D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
D11073 Architectures for darwin need to be conditionalized based on the operating system.

One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.

(2) Build the new Darwin build behavior

One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.

The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.

There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.

(3) Make compiler-rt an external project when built in-tree with Clang

Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.

This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.

Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.

(4) Celebrate!

That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.

So... what's next?

With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.

Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.

Questions, comments, concerns, condolences?


Are we close enough to fixing all the CMake bugs that we can mark autoconf
as deprecated now and plan to remove it after the 3.8 release?

-Tom

That is premature. The compiler-rt build is not even close to parity with the makefiles (at least for Apple platforms). Chris is working on a road map to fix that, but we’re still at the beginning of that road, not the end.


Thanks,
-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Martell Malone
In reply to this post by Chris Bieneman
Questions, comments, concerns, condolences?

While in the process of moving away from autotools and fully supporting cmake be should look at the underlying generator to build with.
I'm sure many of the llvm clang devs already use cmake with the ninja build tool provided here.

We have an interesting issue however when we want to install a single components of llvm with this tool.

This seems like something that would have to be done in the CMakeLists with new targets like install-clang  similarly to how we already have check-clang.

Of course I man this to be done for every component not just clang

Thoughts?

On Wed, Jul 29, 2015 at 9:36 PM, Chris Bieneman <[hidden email]> wrote:
Hi LLVMDev,

I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).

The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.

(1) Reconcile out-of-tree functionality

Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.

D11083 [CMake] Add experimental support for building compiler-rt for iOS
D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
D11073 Architectures for darwin need to be conditionalized based on the operating system.

One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.

(2) Build the new Darwin build behavior

One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.

The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.

There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.

(3) Make compiler-rt an external project when built in-tree with Clang

Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.

This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.

Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.

(4) Celebrate!

That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.

So... what's next?

With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.

Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.

Questions, comments, concerns, condolences?

Thanks,
-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Tom Stellard-3
In reply to this post by Bob Wilson-3
On Wed, Jul 29, 2015 at 07:17:13PM -0700, Bob Wilson wrote:

>
> > On Jul 29, 2015, at 7:03 PM, Tom Stellard <[hidden email]> wrote:
> >
> > On Wed, Jul 29, 2015 at 01:36:43PM -0700, Chris Bieneman wrote:
> >> Hi LLVMDev,
> >>
> >> I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).
> >>
> >> The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.
> >>
> >> (1) Reconcile out-of-tree functionality
> >>
> >> Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.
> >>
> >> D11083 [CMake] Add experimental support for building compiler-rt for iOS
> >> D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
> >> D11073 Architectures for darwin need to be conditionalized based on the operating system.
> >>
> >> One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.
> >>
> >> (2) Build the new Darwin build behavior
> >>
> >> One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.
> >>
> >> The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.
> >>
> >> There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.
> >>
> >> (3) Make compiler-rt an external project when built in-tree with Clang
> >>
> >> Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.
> >>
> >> This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.
> >>
> >> Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.
> >>
> >> (4) Celebrate!
> >>
> >> That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.
> >>
> >> So... what's next?
> >>
> >> With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.
> >>
> >> Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.
> >>
> >> Questions, comments, concerns, condolences?
> >>
> >
> > Are we close enough to fixing all the CMake bugs that we can mark autoconf
> > as deprecated now and plan to remove it after the 3.8 release?
> >
> > -Tom
>
> That is premature. The compiler-rt build is not even close to parity with the makefiles (at least for Apple platforms). Chris is working on a road map to fix that, but we’re still at the beginning of that road, not the end.
>

Is it possible to move to CMake earlier on projects besides comiler-rt,
or does compiler-rt require building llvm with autoconf?

-Tom

> >
> >> Thanks,
> >> -Chris
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> [hidden email] <mailto:[hidden email]>         http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
> > _______________________________________________
> > LLVM Developers mailing list
> > [hidden email] <mailto:[hidden email]>         http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Chris Bieneman

On Jul 30, 2015, at 7:35 AM, Tom Stellard <[hidden email]> wrote:

On Wed, Jul 29, 2015 at 07:17:13PM -0700, Bob Wilson wrote:

On Jul 29, 2015, at 7:03 PM, Tom Stellard <[hidden email]> wrote:

On Wed, Jul 29, 2015 at 01:36:43PM -0700, Chris Bieneman wrote:
Hi LLVMDev,

I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).

The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.

(1) Reconcile out-of-tree functionality

Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.

D11083 [CMake] Add experimental support for building compiler-rt for iOS
D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
D11073 Architectures for darwin need to be conditionalized based on the operating system.

One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.

(2) Build the new Darwin build behavior

One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.

The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.

There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.

(3) Make compiler-rt an external project when built in-tree with Clang

Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.

This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.

Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.

(4) Celebrate!

That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.

So... what's next?

With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.

Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.

Questions, comments, concerns, condolences?


Are we close enough to fixing all the CMake bugs that we can mark autoconf
as deprecated now and plan to remove it after the 3.8 release?

Tom,

I think it is a bit too soon to mark it as deprecated, but I hope to be in a position to mark it as deprecated sometime this fall. So hopefully, yes it will be deprecated before the 3.8 branch and hopefully we can remove it after the 3.8 release. That said, I think there is still a healthy bit of work before we get to a point where I am comfortable marking it as deprecated.

My hope at this point is that the community will at least mostly agree with the plan I laid out at the start of this thread, and that we can make some rapid progress over the next few months.



-Tom

That is premature. The compiler-rt build is not even close to parity with the makefiles (at least for Apple platforms). Chris is working on a road map to fix that, but we’re still at the beginning of that road, not the end.


Is it possible to move to CMake earlier on projects besides comiler-rt,
or does compiler-rt require building llvm with autoconf?

I’m sure it is possible to make CMake call into compiler-rt’s autoconf build, but compiler-rt’s autoconf build system is not feature compatible against its CMake build system, and I think making this work reliably would be more work than it is worth.

-Chris



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Chris Bieneman
In reply to this post by Martell Malone

On Jul 30, 2015, at 7:20 AM, Martell Malone <[hidden email]> wrote:

Questions, comments, concerns, condolences?

While in the process of moving away from autotools and fully supporting cmake be should look at the underlying generator to build with.
I'm sure many of the llvm clang devs already use cmake with the ninja build tool provided here.

We have an interesting issue however when we want to install a single components of llvm with this tool.

This seems like something that would have to be done in the CMakeLists with new targets like install-clang  similarly to how we already have check-clang.

Of course I man this to be done for every component not just clang

Thoughts?

Funny enough, I landed a patch back in February that added install-${name} targets to all the LLVM tools (r229727). Unfortunately that patch doesn’t apply to the clang tools because they don’t use the AddLLVM infrastructure. I’ve been thinking about adding a similar hook for clang too. I will probably do that at some point.

-Chris


On Wed, Jul 29, 2015 at 9:36 PM, Chris Bieneman <[hidden email]> wrote:
Hi LLVMDev,

I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).

The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.

(1) Reconcile out-of-tree functionality

Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.

D11083 [CMake] Add experimental support for building compiler-rt for iOS
D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
D11073 Architectures for darwin need to be conditionalized based on the operating system.

One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.

(2) Build the new Darwin build behavior

One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.

The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.

There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.

(3) Make compiler-rt an external project when built in-tree with Clang

Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.

This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.

Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.

(4) Celebrate!

That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.

So... what's next?

With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.

Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.

Questions, comments, concerns, condolences?

Thanks,
-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Reid Kleckner-2
In reply to this post by Chris Bieneman
On Wed, Jul 29, 2015 at 1:36 PM, Chris Bieneman <[hidden email]> wrote:
I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).

Mono black devotion ftw! Thanks for working on this. :)

The main thing I'm trying to understand is, at each point along the way, will CMake be able to generate a combined build.ninja for all compilation jobs?

You're doing the work, so it's fine if that's a nongoal. However, I'd like it if the simple case where compiler-rt is being compiled for the host, the generated build files don't shell out to another build.

The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.

(1) Reconcile out-of-tree functionality

Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.

D11083 [CMake] Add experimental support for building compiler-rt for iOS
D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
D11073 Architectures for darwin need to be conditionalized based on the operating system.

One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.

(2) Build the new Darwin build behavior

One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.

The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.

There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.

(3) Make compiler-rt an external project when built in-tree with Clang

Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.

This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.

Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.

(4) Celebrate!

That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.

So... what's next?

With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.

Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.

Questions, comments, concerns, condolences?

Thanks,
-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Chris Bieneman

On Jul 30, 2015, at 10:30 AM, Reid Kleckner <[hidden email]> wrote:

On Wed, Jul 29, 2015 at 1:36 PM, Chris Bieneman <[hidden email]> wrote:
I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).

Mono black devotion ftw! Thanks for working on this. :)

The main thing I'm trying to understand is, at each point along the way, will CMake be able to generate a combined build.ninja for all compilation jobs?

In order to generate a single combined build.ninja file we’ll actually need some changes to CMake. My intention is to have a high-level build file (generator agnostic) that will invoke the build tool for sub-projects for compiler-rt. For the Ninja generator we can use a custom job pool for the sub-projects to prevent over saturation of the build machine, but there will be sub-projects.



You're doing the work, so it's fine if that's a nongoal. However, I'd like it if the simple case where compiler-rt is being compiled for the host, the generated build files don't shell out to another build.

I agree, and that is how I would prefer this to work. I had a discussion with some of the guys from Sony a few weeks back about what it would take to be able to generate a single combined build.ninja file. There were three features we would need from CMake to make this work properly.

(1) Prefixing rules so that the sub-project rules wouldn’t conflict with each other or the master project’s rules
(2) Importing the sub-projects into the parent ninja
(3) Mark the CMake invocations for the external projects as “generator” rules so that Ninja would re-exec itself if CMake changed the build files

-Chris


The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.

(1) Reconcile out-of-tree functionality

Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.

D11083 [CMake] Add experimental support for building compiler-rt for iOS
D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
D11073 Architectures for darwin need to be conditionalized based on the operating system.

One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.

(2) Build the new Darwin build behavior

One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.

The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.

There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.

(3) Make compiler-rt an external project when built in-tree with Clang

Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.

This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.

Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.

(4) Celebrate!

That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.

So... what's next?

With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.

Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.

Questions, comments, concerns, condolences?

Thanks,
-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Reid Kleckner-2
Cool, seems reasonable. That CMake feature request doesn't sound too onerous for them.

In the meantime, the Makefile generator will have more-or-less the right job scheduling due make's job server design. Most other generators already have bad scheduling, so it isn't as interesting to have one combined build.

On Thu, Jul 30, 2015 at 10:37 AM, Chris Bieneman <[hidden email]> wrote:

On Jul 30, 2015, at 10:30 AM, Reid Kleckner <[hidden email]> wrote:

On Wed, Jul 29, 2015 at 1:36 PM, Chris Bieneman <[hidden email]> wrote:
I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).

Mono black devotion ftw! Thanks for working on this. :)

The main thing I'm trying to understand is, at each point along the way, will CMake be able to generate a combined build.ninja for all compilation jobs?

In order to generate a single combined build.ninja file we’ll actually need some changes to CMake. My intention is to have a high-level build file (generator agnostic) that will invoke the build tool for sub-projects for compiler-rt. For the Ninja generator we can use a custom job pool for the sub-projects to prevent over saturation of the build machine, but there will be sub-projects.



You're doing the work, so it's fine if that's a nongoal. However, I'd like it if the simple case where compiler-rt is being compiled for the host, the generated build files don't shell out to another build.

I agree, and that is how I would prefer this to work. I had a discussion with some of the guys from Sony a few weeks back about what it would take to be able to generate a single combined build.ninja file. There were three features we would need from CMake to make this work properly.

(1) Prefixing rules so that the sub-project rules wouldn’t conflict with each other or the master project’s rules
(2) Importing the sub-projects into the parent ninja
(3) Mark the CMake invocations for the external projects as “generator” rules so that Ninja would re-exec itself if CMake changed the build files

-Chris


The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.

(1) Reconcile out-of-tree functionality

Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.

D11083 [CMake] Add experimental support for building compiler-rt for iOS
D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
D11073 Architectures for darwin need to be conditionalized based on the operating system.

One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.

(2) Build the new Darwin build behavior

One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.

The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.

There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.

(3) Make compiler-rt an external project when built in-tree with Clang

Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.

This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.

Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.

(4) Celebrate!

That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.

So... what's next?

With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.

Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.

Questions, comments, concerns, condolences?

Thanks,
-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Chris Bieneman

On Jul 30, 2015, at 11:11 AM, Reid Kleckner <[hidden email]> wrote:

Cool, seems reasonable. That CMake feature request doesn't sound too onerous for them.

In the meantime, the Makefile generator will have more-or-less the right job scheduling due make's job server design. Most other generators already have bad scheduling, so it isn't as interesting to have one combined build.

Yes, the expectation is that Make’s job scheduling server makes changes like this unnecessary, and I suspect that by putting the Ninja sub-tasks into a limited size job pool we’ll get acceptable (but not optimal) scheduling.

-Chris


On Thu, Jul 30, 2015 at 10:37 AM, Chris Bieneman <[hidden email]> wrote:

On Jul 30, 2015, at 10:30 AM, Reid Kleckner <[hidden email]> wrote:

On Wed, Jul 29, 2015 at 1:36 PM, Chris Bieneman <[hidden email]> wrote:
I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).

Mono black devotion ftw! Thanks for working on this. :)

The main thing I'm trying to understand is, at each point along the way, will CMake be able to generate a combined build.ninja for all compilation jobs?

In order to generate a single combined build.ninja file we’ll actually need some changes to CMake. My intention is to have a high-level build file (generator agnostic) that will invoke the build tool for sub-projects for compiler-rt. For the Ninja generator we can use a custom job pool for the sub-projects to prevent over saturation of the build machine, but there will be sub-projects.



You're doing the work, so it's fine if that's a nongoal. However, I'd like it if the simple case where compiler-rt is being compiled for the host, the generated build files don't shell out to another build.

I agree, and that is how I would prefer this to work. I had a discussion with some of the guys from Sony a few weeks back about what it would take to be able to generate a single combined build.ninja file. There were three features we would need from CMake to make this work properly.

(1) Prefixing rules so that the sub-project rules wouldn’t conflict with each other or the master project’s rules
(2) Importing the sub-projects into the parent ninja
(3) Mark the CMake invocations for the external projects as “generator” rules so that Ninja would re-exec itself if CMake changed the build files

-Chris


The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.

(1) Reconcile out-of-tree functionality

Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.

D11083 [CMake] Add experimental support for building compiler-rt for iOS
D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
D11073 Architectures for darwin need to be conditionalized based on the operating system.

One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.

(2) Build the new Darwin build behavior

One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.

The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.

There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.

(3) Make compiler-rt an external project when built in-tree with Clang

Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.

This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.

Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.

(4) Celebrate!

That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.

So... what's next?

With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.

Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.

Questions, comments, concerns, condolences?

Thanks,
-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev




_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Renato Golin-2
In reply to this post by Chris Bieneman
On 30 July 2015 at 16:59, Chris Bieneman <[hidden email]> wrote:
> So hopefully, yes it will be deprecated before the 3.8 branch and hopefully we can remove it
> after the 3.8 release.

That's a stunt I want to see happening! Clearly worth a nice bottle of
wine in the next EuroLLVM. :D

My question is: today we still use autoconf (config.guess) with CMake
to pick the correct architecture triple. Are we getting rid of that,
too?

cheers,
--renato
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Chris Bieneman

> On Jul 30, 2015, at 1:07 PM, Renato Golin <[hidden email]> wrote:
>
> On 30 July 2015 at 16:59, Chris Bieneman <[hidden email]> wrote:
>> So hopefully, yes it will be deprecated before the 3.8 branch and hopefully we can remove it
>> after the 3.8 release.
>
> That's a stunt I want to see happening! Clearly worth a nice bottle of
> wine in the next EuroLLVM. :D
>
> My question is: today we still use autoconf (config.guess) with CMake
> to pick the correct architecture triple. Are we getting rid of that,
> too?

Honesty, I haven’t entirely thought through what it is going to mean to remove the autoconf system. There are some standalone makefiles that might make sense to keep around.

Since the current CMake build system depends on config.guess I don’t think it is going anywhere, but maybe eventually we’ll do something different.

-Chris

>
> cheers,
> --renato


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Daniel Sanders
In reply to this post by Chris Bieneman

> I think it is a bit too soon to mark it as deprecated, but I hope to be in a position to mark

> it as deprecated sometime this fall. So hopefully, yes it will be deprecated before the 3.8

> branch and hopefully we can remove it after the 3.8 release. That said, I think there is still

> a healthy bit of work before we get to a point where I am comfortable marking it as deprecated.

 

Should we still mention in the release notes that it's something that's happening in the next few releases? I'm just thinking that the more notice users and out-of-tree projects get the better.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Chris Bieneman
Sent: 30 July 2015 17:00
To: Tom Stellard
Cc: LLVM Developers Mailing List
Subject: Re: [LLVMdev] [RFC] Road map for CMake

 

 

On Jul 30, 2015, at 7:35 AM, Tom Stellard <[hidden email]> wrote:

 

On Wed, Jul 29, 2015 at 07:17:13PM -0700, Bob Wilson wrote:



On Jul 29, 2015, at 7:03 PM, Tom Stellard <[hidden email]> wrote:

On Wed, Jul 29, 2015 at 01:36:43PM -0700, Chris Bieneman wrote:

Hi LLVMDev,

I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana).

The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take.

(1) Reconcile out-of-tree functionality

Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that.

D11083 [CMake] Add experimental support for building compiler-rt for iOS
D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module
D11073 Architectures for darwin need to be conditionalized based on the operating system.

One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking.

(2) Build the new Darwin build behavior

One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin.

The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths.

There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers.

(3) Make compiler-rt an external project when built in-tree with Clang

Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users.

This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory.

Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together.

(4) Celebrate!

That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things.

So... what's next?

With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly.

Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds.

Questions, comments, concerns, condolences?


Are we close enough to fixing all the CMake bugs that we can mark autoconf
as deprecated now and plan to remove it after the 3.8 release?

 

Tom,

 

I think it is a bit too soon to mark it as deprecated, but I hope to be in a position to mark it as deprecated sometime this fall. So hopefully, yes it will be deprecated before the 3.8 branch and hopefully we can remove it after the 3.8 release. That said, I think there is still a healthy bit of work before we get to a point where I am comfortable marking it as deprecated.

 

My hope at this point is that the community will at least mostly agree with the plan I laid out at the start of this thread, and that we can make some rapid progress over the next few months.

 




-Tom


That is premature. The compiler-rt build is not even close to parity with the makefiles (at least for Apple platforms). Chris is working on a road map to fix that, but we’re still at the beginning of that road, not the end.


Is it possible to move to CMake earlier on projects besides comiler-rt,
or does compiler-rt require building llvm with autoconf?

 

I’m sure it is possible to make CMake call into compiler-rt’s autoconf build, but compiler-rt’s autoconf build system is not feature compatible against its CMake build system, and I think making this work reliably would be more work than it is worth.

 

-Chris




-Tom




Thanks,
-Chris
_______________________________________________
LLVM Developers mailing list
[hidden email] <[hidden email]>         http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>

_______________________________________________
LLVM Developers mailing list
[hidden email] <[hidden email]>         http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>

 


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Road map for CMake

Renato Golin-2
On 31 July 2015 at 12:10, Daniel Sanders <[hidden email]> wrote:
> Should we still mention in the release notes that it's something that's
> happening in the next few releases? I'm just thinking that the more notice
> users and out-of-tree projects get the better.

If we mention that this is a possibility, not a fact, I think we should.

Right now, most of us really want to, but there are some serious
considerations to take into account. It might not be possible for
post-3.8, or at all.

cheers,
--renato
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev