[llvm-dev] New LLVM git repository conversion prototype

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
75 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

[llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.

Let me know what you think.

I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.

At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.

This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.

I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.

Some things that could be discussed in such a plan:
  * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
    * Any particular steps wanted here?
  * Converting buildbots to use git.
  * Phabricator changes?
  * How do email notifications get sent for commits?
  * Gathering github accounts for all committers, adding them to a github team.
  * Deciding upon and announcing a timeline for switching over.
  * Proposing, implementing, and testing new workflows for direct git usage:
    * Github pull requests instead of (or in addition to?) phabricator?
    * Github Protected Branch configuration options?
      * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
      * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org) to do some testing on pull requests, to reduce avoidable tree breakages?
      * Any other github configuration options that need to be decided upon?
  * ....other things I forgot about at the moment...
  * Timeline for switchover.



Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
  * Places the SVN revision number into the commit message, as "llvm-svn=1234"

  * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).

  * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.

  * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.

  * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).



Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
  * cfe (renamed to clang in the conversion)
  * clang-tools-extra
  * compiler-rt
  * debuginfo-tests
  * dragonegg (also "gcc-plugin", the original name)
  * libclc
  * libcxx
  * libcxxabi
  * libunwind
  * lld
  * lldb
  * llgo
  * llvm
  * openmp
  * parallel-libs
  * polly
  * pstl
  * stacker (deleted after r40406)
(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).

Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
  * lnt
  * test-suite
  * www
  * www-pubs
  * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
  * zorg

A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
  * poolalloc
  * safecode

Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
  * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
  * clang-tests-external # Copy of GDB testsuite
  * llvm-gcc-4.0 # GCC 4.0, modified for llvm
  * llvm-gcc-4.2 # GCC 4.2, modified for llvm
  * llvm-gcc-4-2 # (merge with above)
  * java
  * vmkit
  * nightly-test-server
  * llbrowse # An LLVM bitcode GUI browser
  * television # A different LLVM GUI browser; shows effects of transforms, etc
  * website # 2007-era snapshot of website, not actually maintained here.
  * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.

Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
  * giri # Never actually developed here; actually https://github.com/liuml07/giri
  * klee # Already migrated to github with history; https://github.com/klee/klee


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
It brings a tear to my eye seeing this actually make real progress.

"Let's not get ahead of ourselves" is the correct answer here, but I do wonder if we should consider moving our issue tracker to Github as well, since it integrates nicely with the pull request workflow and you can reference bugs in commit messages for cross-referencing.

On Thu, Oct 11, 2018 at 3:28 PM James Y Knight via llvm-dev <[hidden email]> wrote:
TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.

Let me know what you think.

I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.

At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.

This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.

I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.

Some things that could be discussed in such a plan:
  * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
    * Any particular steps wanted here?
  * Converting buildbots to use git.
  * Phabricator changes?
  * How do email notifications get sent for commits?
  * Gathering github accounts for all committers, adding them to a github team.
  * Deciding upon and announcing a timeline for switching over.
  * Proposing, implementing, and testing new workflows for direct git usage:
    * Github pull requests instead of (or in addition to?) phabricator?
    * Github Protected Branch configuration options?
      * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
      * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org) to do some testing on pull requests, to reduce avoidable tree breakages?
      * Any other github configuration options that need to be decided upon?
  * ....other things I forgot about at the moment...
  * Timeline for switchover.



Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
  * Places the SVN revision number into the commit message, as "llvm-svn=1234"

  * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).

  * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.

  * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.

  * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).



Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
  * cfe (renamed to clang in the conversion)
  * clang-tools-extra
  * compiler-rt
  * debuginfo-tests
  * dragonegg (also "gcc-plugin", the original name)
  * libclc
  * libcxx
  * libcxxabi
  * libunwind
  * lld
  * lldb
  * llgo
  * llvm
  * openmp
  * parallel-libs
  * polly
  * pstl
  * stacker (deleted after r40406)
(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).

Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
  * lnt
  * test-suite
  * www
  * www-pubs
  * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
  * zorg

A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
  * poolalloc
  * safecode

Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
  * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
  * clang-tests-external # Copy of GDB testsuite
  * llvm-gcc-4.0 # GCC 4.0, modified for llvm
  * llvm-gcc-4.2 # GCC 4.2, modified for llvm
  * llvm-gcc-4-2 # (merge with above)
  * java
  * vmkit
  * nightly-test-server
  * llbrowse # An LLVM bitcode GUI browser
  * television # A different LLVM GUI browser; shows effects of transforms, etc
  * website # 2007-era snapshot of website, not actually maintained here.
  * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.

Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
  * giri # Never actually developed here; actually https://github.com/liuml07/giri
  * klee # Already migrated to github with history; https://github.com/klee/klee

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
One thing at a time... :-)

James' list is already very large indeed as it is. I'd even be happy with core move before less active / dead projects. 

On Fri, 12 Oct 2018, 03:27 Zachary Turner via llvm-dev, <[hidden email]> wrote:
It brings a tear to my eye seeing this actually make real progress.

"Let's not get ahead of ourselves" is the correct answer here, but I do wonder if we should consider moving our issue tracker to Github as well, since it integrates nicely with the pull request workflow and you can reference bugs in commit messages for cross-referencing.

On Thu, Oct 11, 2018 at 3:28 PM James Y Knight via llvm-dev <[hidden email]> wrote:
TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.

Let me know what you think.

I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.

At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.

This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.

I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.

Some things that could be discussed in such a plan:
  * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
    * Any particular steps wanted here?
  * Converting buildbots to use git.
  * Phabricator changes?
  * How do email notifications get sent for commits?
  * Gathering github accounts for all committers, adding them to a github team.
  * Deciding upon and announcing a timeline for switching over.
  * Proposing, implementing, and testing new workflows for direct git usage:
    * Github pull requests instead of (or in addition to?) phabricator?
    * Github Protected Branch configuration options?
      * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
      * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org) to do some testing on pull requests, to reduce avoidable tree breakages?
      * Any other github configuration options that need to be decided upon?
  * ....other things I forgot about at the moment...
  * Timeline for switchover.



Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
  * Places the SVN revision number into the commit message, as "llvm-svn=1234"

  * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).

  * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.

  * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.

  * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).



Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
  * cfe (renamed to clang in the conversion)
  * clang-tools-extra
  * compiler-rt
  * debuginfo-tests
  * dragonegg (also "gcc-plugin", the original name)
  * libclc
  * libcxx
  * libcxxabi
  * libunwind
  * lld
  * lldb
  * llgo
  * llvm
  * openmp
  * parallel-libs
  * polly
  * pstl
  * stacker (deleted after r40406)
(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).

Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
  * lnt
  * test-suite
  * www
  * www-pubs
  * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
  * zorg

A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
  * poolalloc
  * safecode

Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
  * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
  * clang-tests-external # Copy of GDB testsuite
  * llvm-gcc-4.0 # GCC 4.0, modified for llvm
  * llvm-gcc-4.2 # GCC 4.2, modified for llvm
  * llvm-gcc-4-2 # (merge with above)
  * java
  * vmkit
  * nightly-test-server
  * llbrowse # An LLVM bitcode GUI browser
  * television # A different LLVM GUI browser; shows effects of transforms, etc
  * website # 2007-era snapshot of website, not actually maintained here.
  * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.

Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
  * giri # Never actually developed here; actually https://github.com/liuml07/giri
  * klee # Already migrated to github with history; https://github.com/klee/klee

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
I notice there’s a README.md in the top level llvm folder. With the current monorepo it seems impossible to put files at that level. It seems that will no longer be the case?

I’d like for our top level CMakeLists.txt to eventually be at this level as it makes more sense. It sounds like this will eventually be possible?
On Thu, Oct 11, 2018 at 9:52 PM Renato Golin <[hidden email]> wrote:
One thing at a time... :-)

James' list is already very large indeed as it is. I'd even be happy with core move before less active / dead projects. 

On Fri, 12 Oct 2018, 03:27 Zachary Turner via llvm-dev, <[hidden email]> wrote:
It brings a tear to my eye seeing this actually make real progress.

"Let's not get ahead of ourselves" is the correct answer here, but I do wonder if we should consider moving our issue tracker to Github as well, since it integrates nicely with the pull request workflow and you can reference bugs in commit messages for cross-referencing.

On Thu, Oct 11, 2018 at 3:28 PM James Y Knight via llvm-dev <[hidden email]> wrote:
TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.

Let me know what you think.

I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.

At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.

This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.

I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.

Some things that could be discussed in such a plan:
  * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
    * Any particular steps wanted here?
  * Converting buildbots to use git.
  * Phabricator changes?
  * How do email notifications get sent for commits?
  * Gathering github accounts for all committers, adding them to a github team.
  * Deciding upon and announcing a timeline for switching over.
  * Proposing, implementing, and testing new workflows for direct git usage:
    * Github pull requests instead of (or in addition to?) phabricator?
    * Github Protected Branch configuration options?
      * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
      * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org) to do some testing on pull requests, to reduce avoidable tree breakages?
      * Any other github configuration options that need to be decided upon?
  * ....other things I forgot about at the moment...
  * Timeline for switchover.



Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
  * Places the SVN revision number into the commit message, as "llvm-svn=1234"

  * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).

  * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.

  * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.

  * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).



Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
  * cfe (renamed to clang in the conversion)
  * clang-tools-extra
  * compiler-rt
  * debuginfo-tests
  * dragonegg (also "gcc-plugin", the original name)
  * libclc
  * libcxx
  * libcxxabi
  * libunwind
  * lld
  * lldb
  * llgo
  * llvm
  * openmp
  * parallel-libs
  * polly
  * pstl
  * stacker (deleted after r40406)
(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).

Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
  * lnt
  * test-suite
  * www
  * www-pubs
  * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
  * zorg

A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
  * poolalloc
  * safecode

Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
  * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
  * clang-tests-external # Copy of GDB testsuite
  * llvm-gcc-4.0 # GCC 4.0, modified for llvm
  * llvm-gcc-4.2 # GCC 4.2, modified for llvm
  * llvm-gcc-4-2 # (merge with above)
  * java
  * vmkit
  * nightly-test-server
  * llbrowse # An LLVM bitcode GUI browser
  * television # A different LLVM GUI browser; shows effects of transforms, etc
  * website # 2007-era snapshot of website, not actually maintained here.
  * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.

Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
  * giri # Never actually developed here; actually https://github.com/liuml07/giri
  * klee # Already migrated to github with history; https://github.com/klee/klee

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
Yes, files in the toplevel are sourced from http://llvm.org/svn/llvm-project/monorepo-root/

(Please don't commit anything in there which has the same name as a project subdir, it'll break migration!)

On Fri, Oct 12, 2018 at 11:04 AM Zachary Turner <[hidden email]> wrote:
I notice there’s a README.md in the top level llvm folder. With the current monorepo it seems impossible to put files at that level. It seems that will no longer be the case?

I’d like for our top level CMakeLists.txt to eventually be at this level as it makes more sense. It sounds like this will eventually be possible?
On Thu, Oct 11, 2018 at 9:52 PM Renato Golin <[hidden email]> wrote:
One thing at a time... :-)

James' list is already very large indeed as it is. I'd even be happy with core move before less active / dead projects. 

On Fri, 12 Oct 2018, 03:27 Zachary Turner via llvm-dev, <[hidden email]> wrote:
It brings a tear to my eye seeing this actually make real progress.

"Let's not get ahead of ourselves" is the correct answer here, but I do wonder if we should consider moving our issue tracker to Github as well, since it integrates nicely with the pull request workflow and you can reference bugs in commit messages for cross-referencing.

On Thu, Oct 11, 2018 at 3:28 PM James Y Knight via llvm-dev <[hidden email]> wrote:
TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.

Let me know what you think.

I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.

At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.

This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.

I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.

Some things that could be discussed in such a plan:
  * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
    * Any particular steps wanted here?
  * Converting buildbots to use git.
  * Phabricator changes?
  * How do email notifications get sent for commits?
  * Gathering github accounts for all committers, adding them to a github team.
  * Deciding upon and announcing a timeline for switching over.
  * Proposing, implementing, and testing new workflows for direct git usage:
    * Github pull requests instead of (or in addition to?) phabricator?
    * Github Protected Branch configuration options?
      * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
      * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org) to do some testing on pull requests, to reduce avoidable tree breakages?
      * Any other github configuration options that need to be decided upon?
  * ....other things I forgot about at the moment...
  * Timeline for switchover.



Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
  * Places the SVN revision number into the commit message, as "llvm-svn=1234"

  * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).

  * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.

  * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.

  * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).



Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
  * cfe (renamed to clang in the conversion)
  * clang-tools-extra
  * compiler-rt
  * debuginfo-tests
  * dragonegg (also "gcc-plugin", the original name)
  * libclc
  * libcxx
  * libcxxabi
  * libunwind
  * lld
  * lldb
  * llgo
  * llvm
  * openmp
  * parallel-libs
  * polly
  * pstl
  * stacker (deleted after r40406)
(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).

Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
  * lnt
  * test-suite
  * www
  * www-pubs
  * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
  * zorg

A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
  * poolalloc
  * safecode

Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
  * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
  * clang-tests-external # Copy of GDB testsuite
  * llvm-gcc-4.0 # GCC 4.0, modified for llvm
  * llvm-gcc-4.2 # GCC 4.2, modified for llvm
  * llvm-gcc-4-2 # (merge with above)
  * java
  * vmkit
  * nightly-test-server
  * llbrowse # An LLVM bitcode GUI browser
  * television # A different LLVM GUI browser; shows effects of transforms, etc
  * website # 2007-era snapshot of website, not actually maintained here.
  * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.

Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
  * giri # Never actually developed here; actually https://github.com/liuml07/giri
  * klee # Already migrated to github with history; https://github.com/klee/klee

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
Congrats on the progress.

On Thu, Oct 11, 2018 at 10:27 PM Zachary Turner via llvm-dev <[hidden email]> wrote:
 I do wonder if we should consider moving our issue tracker to Github as well, since it integrates nicely with the pull request workflow and you can reference bugs in commit messages for cross-referencing.

I ask this earnestly, not as a disagreement -
Would the ~11,600 open bug reports get migrated, with the CC email lists preserved?
 

On Thu, Oct 11, 2018 at 3:28 PM James Y Knight via llvm-dev <[hidden email]> wrote:
TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.

Let me know what you think.

I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.

At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.

This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.

I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.

Some things that could be discussed in such a plan:
  * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
    * Any particular steps wanted here?
  * Converting buildbots to use git.
  * Phabricator changes?
  * How do email notifications get sent for commits?
  * Gathering github accounts for all committers, adding them to a github team.
  * Deciding upon and announcing a timeline for switching over.
  * Proposing, implementing, and testing new workflows for direct git usage:
    * Github pull requests instead of (or in addition to?) phabricator?
    * Github Protected Branch configuration options?
      * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
      * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org) to do some testing on pull requests, to reduce avoidable tree breakages?
      * Any other github configuration options that need to be decided upon?
  * ....other things I forgot about at the moment...
  * Timeline for switchover.



Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
  * Places the SVN revision number into the commit message, as "llvm-svn=1234"

  * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).

  * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.

  * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.

  * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).



Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
  * cfe (renamed to clang in the conversion)
  * clang-tools-extra
  * compiler-rt
  * debuginfo-tests
  * dragonegg (also "gcc-plugin", the original name)
  * libclc
  * libcxx
  * libcxxabi
  * libunwind
  * lld
  * lldb
  * llgo
  * llvm
  * openmp
  * parallel-libs
  * polly
  * pstl
  * stacker (deleted after r40406)
(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).

Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
  * lnt
  * test-suite
  * www
  * www-pubs
  * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
  * zorg

A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
  * poolalloc
  * safecode

Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
  * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
  * clang-tests-external # Copy of GDB testsuite
  * llvm-gcc-4.0 # GCC 4.0, modified for llvm
  * llvm-gcc-4.2 # GCC 4.2, modified for llvm
  * llvm-gcc-4-2 # (merge with above)
  * java
  * vmkit
  * nightly-test-server
  * llbrowse # An LLVM bitcode GUI browser
  * television # A different LLVM GUI browser; shows effects of transforms, etc
  * website # 2007-era snapshot of website, not actually maintained here.
  * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.

Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
  * giri # Never actually developed here; actually https://github.com/liuml07/giri
  * klee # Already migrated to github with history; https://github.com/klee/klee

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
On Fri, 12 Oct 2018 at 17:05, Andrew Kelley via llvm-dev
<[hidden email]> wrote:
> I ask this earnestly, not as a disagreement -
> Would the ~11,600 open bug reports get migrated, with the CC email lists preserved?

(same as you, not against, just opining).

This is the big question, and why originally we decided not to look
into it until the repo migration was well over.

Just moving the comments seem a bit pointless... Especially when not
everyone that used out bugzilla for the past 10+ years has a Github
account, or with the same email.

Furthermore...
 * Do we upload all attachments? Does Github want to have all that
history? Do we?
 * Do we migrate the emails? People have to register on bugzilla, so
we'd need to find all their accounts on github?
 * Do we create tags for all categories, new, old, deprecated? Do we
map old into new?
 * Do we move all SVN commit references to Git hashes from GitHub repositories?

I'm sure there are a number of other questions that, adding to James'
list, will be impossible to disentangle before 2050.

I'll be very sad is 2020 passes and we're still using SVN... :(

--renato
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
There certainly cannot be an issue tracker migration, without migrating all of the existing reports with high enough fidelity that the bugzilla server can be permanently shut down. Beyond that, I wouldn't venture to say what a conversion may or may not look like, and I'd prefer to not even start thinking about it yet. Let's not get distracted by issue-tracker migration discussion.

IMO, in both the short and medium term, the LLVM project will be well served by sticking with bugzilla. One thing that would be very nice, if someone wants to do the work, would be adding "Sign in with GitHub" support, from bugzilla.mozilla.org's patches. (Previous email where I mentioned that: http://lists.llvm.org/pipermail/llvm-dev/2017-October/118328.html).

On Fri, Oct 12, 2018 at 12:04 PM Andrew Kelley <[hidden email]> wrote:
Congrats on the progress.

On Thu, Oct 11, 2018 at 10:27 PM Zachary Turner via llvm-dev <[hidden email]> wrote:
 I do wonder if we should consider moving our issue tracker to Github as well, since it integrates nicely with the pull request workflow and you can reference bugs in commit messages for cross-referencing.

I ask this earnestly, not as a disagreement -
Would the ~11,600 open bug reports get migrated, with the CC email lists preserved?
 

On Thu, Oct 11, 2018 at 3:28 PM James Y Knight via llvm-dev <[hidden email]> wrote:
TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.

Let me know what you think.

I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.

At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.

This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.

I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.

Some things that could be discussed in such a plan:
  * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
    * Any particular steps wanted here?
  * Converting buildbots to use git.
  * Phabricator changes?
  * How do email notifications get sent for commits?
  * Gathering github accounts for all committers, adding them to a github team.
  * Deciding upon and announcing a timeline for switching over.
  * Proposing, implementing, and testing new workflows for direct git usage:
    * Github pull requests instead of (or in addition to?) phabricator?
    * Github Protected Branch configuration options?
      * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
      * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org) to do some testing on pull requests, to reduce avoidable tree breakages?
      * Any other github configuration options that need to be decided upon?
  * ....other things I forgot about at the moment...
  * Timeline for switchover.



Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
  * Places the SVN revision number into the commit message, as "llvm-svn=1234"

  * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).

  * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.

  * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.

  * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).



Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
  * cfe (renamed to clang in the conversion)
  * clang-tools-extra
  * compiler-rt
  * debuginfo-tests
  * dragonegg (also "gcc-plugin", the original name)
  * libclc
  * libcxx
  * libcxxabi
  * libunwind
  * lld
  * lldb
  * llgo
  * llvm
  * openmp
  * parallel-libs
  * polly
  * pstl
  * stacker (deleted after r40406)
(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).

Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
  * lnt
  * test-suite
  * www
  * www-pubs
  * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
  * zorg

A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
  * poolalloc
  * safecode

Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
  * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
  * clang-tests-external # Copy of GDB testsuite
  * llvm-gcc-4.0 # GCC 4.0, modified for llvm
  * llvm-gcc-4.2 # GCC 4.2, modified for llvm
  * llvm-gcc-4-2 # (merge with above)
  * java
  * vmkit
  * nightly-test-server
  * llbrowse # An LLVM bitcode GUI browser
  * television # A different LLVM GUI browser; shows effects of transforms, etc
  * website # 2007-era snapshot of website, not actually maintained here.
  * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.

Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
  * giri # Never actually developed here; actually https://github.com/liuml07/giri
  * klee # Already migrated to github with history; https://github.com/klee/klee

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
If we're reasonably confident that we would like to be using GitHub Issues in the future, I think it would not be so bad to continue using Bugzilla for the old reports, but have new bug reports disabled, with the understanding that it is deprecated. It's messy, sure, but at the end of the day, you're either getting an email from Bugzilla or from GitHub and it's not all that important. In practice, after about one release cycle, most of the activity would take place in the new system.

I take this approach in my code - refactoring and new/better patterns don't have to be introduced all at once; there can be a gradual transition.

On Fri, Oct 12, 2018 at 12:35 PM Renato Golin <[hidden email]> wrote:
On Fri, 12 Oct 2018 at 17:05, Andrew Kelley via llvm-dev
<[hidden email]> wrote:
> I ask this earnestly, not as a disagreement -
> Would the ~11,600 open bug reports get migrated, with the CC email lists preserved?

(same as you, not against, just opining).

This is the big question, and why originally we decided not to look
into it until the repo migration was well over.

Just moving the comments seem a bit pointless... Especially when not
everyone that used out bugzilla for the past 10+ years has a Github
account, or with the same email.

Furthermore...
 * Do we upload all attachments? Does Github want to have all that
history? Do we?
 * Do we migrate the emails? People have to register on bugzilla, so
we'd need to find all their accounts on github?
 * Do we create tags for all categories, new, old, deprecated? Do we
map old into new?
 * Do we move all SVN commit references to Git hashes from GitHub repositories?

I'm sure there are a number of other questions that, adding to James'
list, will be impossible to disentangle before 2050.

I'll be very sad is 2020 passes and we're still using SVN... :(

--renato

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
On Fri, 12 Oct 2018 at 17:48, Andrew Kelley <[hidden email]> wrote:
> In practice, after about one release cycle, most of the activity would take place in the new system.

You'd be surprised how many 4-year-old bugs I had to deal with...

So, I agree that the only meaningful thing would be to not migrate. If
we start using it later or not, that's for a different thread.

Reiterating James' request: Let's not get distracted by issue-tracker
migration discussion. :)

--renato
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
Great progress!  Thanks for your dedication to getting this done, it's
really important work.

James Y Knight via llvm-dev <[hidden email]> writes:

> Some SVN projects are still active, but not part of the LLVM codebase.
> These get migrated to their own separate git repositories:
> * lnt
> * test-suite
> * www
> * www-pubs
> * www-releases ## TODO. Not done yet as it requires the use of
> git-lfs, due to large files.

I wonder whether www and www-pubs (maybe just www) should be part of the
monorepo.  Keep the documentation with the code.  It's a bit painful to
change, say, TableGen and then have to go make a separate change to
update the TableGen docs.

                           -David
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
Renato Golin via llvm-dev <[hidden email]> writes:

> On Fri, 12 Oct 2018 at 17:48, Andrew Kelley <[hidden email]> wrote:
>> In practice, after about one release cycle, most of the activity would take place in the new system.
>
> You'd be surprised how many 4-year-old bugs I had to deal with...

I hear you.  :)

I know we don't want to get distracted with this but a good compromise
solution might be to only migrate active bugs to GitHub and keep
Bugzilla for legacy bug searches.  I certainly have used Bugzilla to
search resolved bugs so we don't want to lose that information.

I absolutely think GitHub's bug/PR integration is worth the cost of
transition.

                         -David
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
The docs go with the code already. Note that the tablegen docs are in llvm/docs/TableGenFundamentals.rst (this is the pre-existing state of things, it is not changing with the migration!).



On Fri, Oct 12, 2018 at 1:48 PM David Greene <[hidden email]> wrote:
Great progress!  Thanks for your dedication to getting this done, it's
really important work.

James Y Knight via llvm-dev <[hidden email]> writes:

> Some SVN projects are still active, but not part of the LLVM codebase.
> These get migrated to their own separate git repositories:
> * lnt
> * test-suite
> * www
> * www-pubs
> * www-releases ## TODO. Not done yet as it requires the use of
> git-lfs, due to large files.

I wonder whether www and www-pubs (maybe just www) should be part of the
monorepo.  Keep the documentation with the code.  It's a bit painful to
change, say, TableGen and then have to go make a separate change to
update the TableGen docs.

                           -David

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
On Fri, Oct 12, 2018 at 1:50 PM David Greene via llvm-dev <[hidden email]> wrote:
Renato Golin via llvm-dev <[hidden email]> writes:

> On Fri, 12 Oct 2018 at 17:48, Andrew Kelley <[hidden email]> wrote:
>> In practice, after about one release cycle, most of the activity would take place in the new system.
>
> You'd be surprised how many 4-year-old bugs I had to deal with...

I hear you.  :)

I know we don't want to get distracted with this but a good compromise
solution might be to only migrate active bugs to GitHub and keep
Bugzilla for legacy bug searches.  I certainly have used Bugzilla to
search resolved bugs so we don't want to lose that information

No. There cannot be a migration which involves switching away from bugzilla but still requires keeping an up-to-date, secure, patched, bugzilla installation running forever. It's hard work to keep services running, they don't just run themselves, and doing so for legacy reference only is not going to work.

Either we keep actively using bugzilla, or we have a plan which culminates in being able to turn it off without unacceptable loss of data.
 
I absolutely think GitHub's bug/PR integration is worth the cost of
transition.

We do not need to migrate issue tracking in order to use PRs. Those are two entirely distinct things.

If anyone wants to propose issue tracker migration, please write up an entirely separate proposal. We're _definitely_ not going to do it as part of the VCS repository migration.


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev

Great to hear!

Are there some migrations strategies for those who are using llvm-mirror repositories?

I’ve noticed that some projects in llvm git prototype not only have different commit SHAs (which is expected, since the commit messages are slightly different due to llvm-svn=N), but also different histories (first lnt commit in llvm-mirror is 2 years older than in prototype).

 

From: llvm-dev [mailto:[hidden email]] On Behalf Of James Y Knight via llvm-dev
Sent: Friday, October 12, 2018 01:28
To: llvm-dev <[hidden email]>
Subject: [llvm-dev] New LLVM git repository conversion prototype

 

TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.


Let me know what you think.

 

I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.

 

At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.

 

This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.

 

I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.

 

Some things that could be discussed in such a plan:

  * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).

    * Any particular steps wanted here?

  * Converting buildbots to use git.

  * Phabricator changes?

  * How do email notifications get sent for commits?

  * Gathering github accounts for all committers, adding them to a github team.

  * Deciding upon and announcing a timeline for switching over.

  * Proposing, implementing, and testing new workflows for direct git usage:

    * Github pull requests instead of (or in addition to?) phabricator?

    * Github Protected Branch configuration options?

      * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?

      * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org) to do some testing on pull requests, to reduce avoidable tree breakages?

      * Any other github configuration options that need to be decided upon?

  * ....other things I forgot about at the moment...

  * Timeline for switchover.

 

 

 

Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:

  * Places the SVN revision number into the commit message, as "llvm-svn=1234"

 

  * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).

 

  * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.

 

  * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.

 

  * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).

 

 

 

Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:

  * cfe (renamed to clang in the conversion)

  * clang-tools-extra

  * compiler-rt

  * debuginfo-tests

  * dragonegg (also "gcc-plugin", the original name)

  * libclc

  * libcxx

  * libcxxabi

  * libunwind

  * lld

  * lldb

  * llgo

  * llvm

  * openmp

  * parallel-libs

  * polly

  * pstl

  * stacker (deleted after r40406)

(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).

 

Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:

  * lnt

  * test-suite

  * www

  * www-pubs

  * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.

  * zorg

 

A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:

  * poolalloc

  * safecode

 

Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":

  * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang

  * clang-tests-external # Copy of GDB testsuite

  * llvm-gcc-4.0 # GCC 4.0, modified for llvm

  * llvm-gcc-4.2 # GCC 4.2, modified for llvm

  * llvm-gcc-4-2 # (merge with above)

  * java

  * vmkit

  * nightly-test-server

  * llbrowse # An LLVM bitcode GUI browser

  * television # A different LLVM GUI browser; shows effects of transforms, etc

  * website # 2007-era snapshot of website, not actually maintained here.

  * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.

 

Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:

  * giri # Never actually developed here; actually https://github.com/liuml07/giri

  * klee # Already migrated to github with history; https://github.com/klee/klee

 


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:

> TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.
>
> Let me know what you think.
>
> I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.
>
> At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.
>
> This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.
>
> I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.
>

Hi James,

Mike Edwards and I were planning to do a round-table during this week's
developer meeting to work on the migration plan.  I have scheduled it
for 4:30PM during the Wednesday session.  It would be great if you or
anyone else who is interested could make it.

The goal of this round-table is to come away with a concrete set of
actions with assigned owners that we can begin implementing.  We
don't want to use up the meeting time with long debates about undecided
issues, so any longer discussions will likely need to be turned into action
items and taken to the mailing list.

Thanks,
Tom

> Some things that could be discussed in such a plan:
>   * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
>     * Any particular steps wanted here?
>   * Converting buildbots to use git.
>   * Phabricator changes?
>   * How do email notifications get sent for commits?
>   * Gathering github accounts for all committers, adding them to a github team.
>   * Deciding upon and announcing a timeline for switching over.
>   * Proposing, implementing, and testing new workflows for direct git usage:
>     * Github pull requests instead of (or in addition to?) phabricator?
>     * Github Protected Branch configuration options?
>       * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
>       * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org <http://travis-ci.org>) to do some testing on pull requests, to reduce avoidable tree breakages?
>       * Any other github configuration options that need to be decided upon?
>   * ....other things I forgot about at the moment...
>   * Timeline for switchover.
>
>
>
> Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
>   * Places the SVN revision number into the commit message, as "llvm-svn=1234"
>
>   * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).
>
>   * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.
>
>   * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.
>
>   * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).
>
>
>
> Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
>   * cfe (renamed to clang in the conversion)
>   * clang-tools-extra
>   * compiler-rt
>   * debuginfo-tests
>   * dragonegg (also "gcc-plugin", the original name)
>   * libclc
>   * libcxx
>   * libcxxabi
>   * libunwind
>   * lld
>   * lldb
>   * llgo
>   * llvm
>   * openmp
>   * parallel-libs
>   * polly
>   * pstl
>   * stacker (deleted after r40406)
> (Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).
>
> Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
>   * lnt
>   * test-suite
>   * www
>   * www-pubs
>   * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
>   * zorg
>
> A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
>   * poolalloc
>   * safecode
>
> Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
>   * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
>   * clang-tests-external # Copy of GDB testsuite
>   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>   * llvm-gcc-4-2 # (merge with above)
>   * java
>   * vmkit
>   * nightly-test-server
>   * llbrowse # An LLVM bitcode GUI browser
>   * television # A different LLVM GUI browser; shows effects of transforms, etc
>   * website # 2007-era snapshot of website, not actually maintained here.
>   * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.
>
> Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
>   * giri # Never actually developed here; actually https://github.com/liuml07/giri
>   * klee # Already migrated to github with history; https://github.com/klee/klee
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
Thanks for setting up the session.

I'll be attending the Dev meeting, and will be at the 4:30 Wednesday round table.

On Oct 15, 2018 10:58 AM, "Tom Stellard" <[hidden email]> wrote:
On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
> TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.
>
> Let me know what you think.
>
> I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.
>
> At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.
>
> This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.
>
> I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.
>

Hi James,

Mike Edwards and I were planning to do a round-table during this week's
developer meeting to work on the migration plan.  I have scheduled it
for 4:30PM during the Wednesday session.  It would be great if you or
anyone else who is interested could make it.

The goal of this round-table is to come away with a concrete set of
actions with assigned owners that we can begin implementing.  We
don't want to use up the meeting time with long debates about undecided
issues, so any longer discussions will likely need to be turned into action
items and taken to the mailing list.

Thanks,
Tom


> Some things that could be discussed in such a plan:
>   * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
>     * Any particular steps wanted here?
>   * Converting buildbots to use git.
>   * Phabricator changes?
>   * How do email notifications get sent for commits?
>   * Gathering github accounts for all committers, adding them to a github team.
>   * Deciding upon and announcing a timeline for switching over.
>   * Proposing, implementing, and testing new workflows for direct git usage:
>     * Github pull requests instead of (or in addition to?) phabricator?
>     * Github Protected Branch configuration options?
>       * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
>       * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org <http://travis-ci.org>) to do some testing on pull requests, to reduce avoidable tree breakages?

>       * Any other github configuration options that need to be decided upon?
>   * ....other things I forgot about at the moment...
>   * Timeline for switchover.
>
>
>
> Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
>   * Places the SVN revision number into the commit message, as "llvm-svn=1234"
>
>   * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).
>
>   * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.
>
>   * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.
>
>   * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).
>
>
>
> Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
>   * cfe (renamed to clang in the conversion)
>   * clang-tools-extra
>   * compiler-rt
>   * debuginfo-tests
>   * dragonegg (also "gcc-plugin", the original name)
>   * libclc
>   * libcxx
>   * libcxxabi
>   * libunwind
>   * lld
>   * lldb
>   * llgo
>   * llvm
>   * openmp
>   * parallel-libs
>   * polly
>   * pstl
>   * stacker (deleted after r40406)
> (Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).
>
> Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
>   * lnt
>   * test-suite
>   * www
>   * www-pubs
>   * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
>   * zorg
>
> A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
>   * poolalloc
>   * safecode
>
> Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
>   * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
>   * clang-tests-external # Copy of GDB testsuite
>   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>   * llvm-gcc-4-2 # (merge with above)
>   * java
>   * vmkit
>   * nightly-test-server
>   * llbrowse # An LLVM bitcode GUI browser
>   * television # A different LLVM GUI browser; shows effects of transforms, etc
>   * website # 2007-era snapshot of website, not actually maintained here.
>   * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.
>
> Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
>   * giri # Never actually developed here; actually https://github.com/liuml07/giri
>   * klee # Already migrated to github with history; https://github.com/klee/klee
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>



_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev

On Oct 15, 2018, at 1:55 PM, James Y Knight via llvm-dev <[hidden email]> wrote:

Thanks for setting up the session.

I'll be attending the Dev meeting, and will be at the 4:30 Wednesday round table.

On Oct 15, 2018 10:58 AM, "Tom Stellard" <[hidden email]> wrote:
On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
> TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.
>
> Let me know what you think.
>
> I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.
>
> At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.
>
> This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.
>
> I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.
>

Hi James,

Mike Edwards and I were planning to do a round-table during this week's
developer meeting to work on the migration plan.  I have scheduled it
for 4:30PM during the Wednesday session.  It would be great if you or
anyone else who is interested could make it.

The goal of this round-table is to come away with a concrete set of
actions with assigned owners that we can begin implementing.  We
don't want to use up the meeting time with long debates about undecided
issues, so any longer discussions will likely need to be turned into action
items and taken to the mailing list.

Thanks,
Tom


> Some things that could be discussed in such a plan:
>   * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
>     * Any particular steps wanted here?
>   * Converting buildbots to use git.
>   * Phabricator changes?
>   * How do email notifications get sent for commits?
>   * Gathering github accounts for all committers, adding them to a github team.
>   * Deciding upon and announcing a timeline for switching over.
>   * Proposing, implementing, and testing new workflows for direct git usage:
>     * Github pull requests instead of (or in addition to?) phabricator?
>     * Github Protected Branch configuration options?
>       * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
>       * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org <http://travis-ci.org>) to do some testing on pull requests, to reduce avoidable tree breakages?

>       * Any other github configuration options that need to be decided upon?
>   * ....other things I forgot about at the moment...
>   * Timeline for switchover.
>
>
>
> Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
>   * Places the SVN revision number into the commit message, as "llvm-svn=1234"
>
>   * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).
>
>   * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.
>
>   * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.
>
>   * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).
>
>
>
> Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
>   * cfe (renamed to clang in the conversion)
>   * clang-tools-extra
>   * compiler-rt
>   * debuginfo-tests
>   * dragonegg (also "gcc-plugin", the original name)
>   * libclc
>   * libcxx
>   * libcxxabi
>   * libunwind
>   * lld
>   * lldb
>   * llgo
>   * llvm
>   * openmp
>   * parallel-libs
>   * polly
>   * pstl
>   * stacker (deleted after r40406)
> (Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).
>
> Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
>   * lnt
>   * test-suite
When we move the test-suite into a git repository we should remember to split up the ABI-Testsuite directory into a separate repository so we don’t end up with the gigabytes in the history.

We discussed this in http://lists.llvm.org/pipermail/llvm-dev/2018-July/124513.html (and ended up with “let’s do this when we move to git”)

- Matthias

>   * www
>   * www-pubs
>   * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
>   * zorg
>
> A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
>   * poolalloc
>   * safecode
>
> Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
>   * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
>   * clang-tests-external # Copy of GDB testsuite
>   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>   * llvm-gcc-4-2 # (merge with above)
>   * java
>   * vmkit
>   * nightly-test-server
>   * llbrowse # An LLVM bitcode GUI browser
>   * television # A different LLVM GUI browser; shows effects of transforms, etc
>   * website # 2007-era snapshot of website, not actually maintained here.
>   * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.
>
> Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
>   * giri # Never actually developed here; actually https://github.com/liuml07/giri
>   * klee # Already migrated to github with history; https://github.com/klee/klee
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
James,

Thank you to disclose your great work!

I had been waiting for your work due to difficulty of handling inconsistent branches in my repo.
As far as I asked guys then, I wasn't able to hear your work.
Then, I began to reconstruct the repo, rewritten from scratch.

I am happy that I can see your progression.

Interestingly, It seems I was following similar tweaks that you did, esp. revisioning the history and tweaks.
They satisfy me. :)

I suggest you a few functionalities that I was doing, to honor "original" authors.
Excuse me if they are bikeshedding.

1) Apply "Signed-off-by:"
Like git-svn's "--use-log-author". We don't force such a feature for years
but some authors are using it.

2) Discover merged commits and cherry-pick them
In branches, we can see many "Merging rXXXXXX:".
I tried picking them up.
I am afraid that this functionality would make building slower.
In my case, I am using git-fast-import. To do it;
  - Flush blobs with "checkpoint" to consolidate HEAD's commit and tree.
  - If simple comparison (with git-merge-tree) failed, I have to use *slow* gitindex,
     for git-read-tree, git-apply, and git-write-tree
     to confirm that cherry-picked commit is identical to original commit.

3) Resurrect "Revert Revert" with cherry-picking.
I don't like one. It's just my preference.

4) Pull the author from phab
(It's just an idea)

I appreciate your great work. Thanks again!
Ask me anything if you are interested.

P.S. I won't attend the devmtg 2018.

Takumi Nakamura


On Fri, Oct 12, 2018 at 7:28 AM James Y Knight via llvm-dev <[hidden email]> wrote:
TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.

Let me know what you think.

I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.

At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.

This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.

I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.

Some things that could be discussed in such a plan:
  * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
    * Any particular steps wanted here?
  * Converting buildbots to use git.
  * Phabricator changes?
  * How do email notifications get sent for commits?
  * Gathering github accounts for all committers, adding them to a github team.
  * Deciding upon and announcing a timeline for switching over.
  * Proposing, implementing, and testing new workflows for direct git usage:
    * Github pull requests instead of (or in addition to?) phabricator?
    * Github Protected Branch configuration options?
      * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
      * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org) to do some testing on pull requests, to reduce avoidable tree breakages?
      * Any other github configuration options that need to be decided upon?
  * ....other things I forgot about at the moment...
  * Timeline for switchover.



Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
  * Places the SVN revision number into the commit message, as "llvm-svn=1234"

  * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).

  * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.

  * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.

  * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).



Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
  * cfe (renamed to clang in the conversion)
  * clang-tools-extra
  * compiler-rt
  * debuginfo-tests
  * dragonegg (also "gcc-plugin", the original name)
  * libclc
  * libcxx
  * libcxxabi
  * libunwind
  * lld
  * lldb
  * llgo
  * llvm
  * openmp
  * parallel-libs
  * polly
  * pstl
  * stacker (deleted after r40406)
(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).

Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
  * lnt
  * test-suite
  * www
  * www-pubs
  * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
  * zorg

A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
  * poolalloc
  * safecode

Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
  * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
  * clang-tests-external # Copy of GDB testsuite
  * llvm-gcc-4.0 # GCC 4.0, modified for llvm
  * llvm-gcc-4.2 # GCC 4.2, modified for llvm
  * llvm-gcc-4-2 # (merge with above)
  * java
  * vmkit
  * nightly-test-server
  * llbrowse # An LLVM bitcode GUI browser
  * television # A different LLVM GUI browser; shows effects of transforms, etc
  * website # 2007-era snapshot of website, not actually maintained here.
  * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.

Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
  * giri # Never actually developed here; actually https://github.com/liuml07/giri
  * klee # Already migrated to github with history; https://github.com/klee/klee

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev
In reply to this post by David Jones via llvm-dev
Summary
========

I'd love if there was a way in the new repo to mark commits as being known to be bad. I don't know whether this should be a separate database, a text file in the repo with a list of hashes, tags with names fitting some pattern, or something else.

I have an existing database of bad commits from 3.6.0 up until the end of October last year.

If there is interest, I can make this database available, and I can also run tests on commits for the last year, and on an ongoing basis (although I'd really prefer that we don't commit things that are seriously broken).

Discussion
=========

There are, alas, a lot of bad commits in the llvm project history. Most of them get backed out fairly quickly, but sometimes not.

This can cause problems with activities such as bisecting to find when a bug or regression was introduced, or incrementally rebasing a long-lived local branch onto upstream llvm.

I was given the task about a year ago to update a private back end for a proprietary CPU that was based on llvm 3.6. As a preliminary step I wrote a script to attempt to build clang for every llvm project revision since 3.6.0, use that clang to build a native HelloWorld for x86_64 Linux, and execute the resulting HelloWorld program and check the output.

Working from the https://github.com/llvm-project/llvm-project-20170507 repository, I tested 84852 commits from ...

-------------
commit c8c6087cf0cc04bbe9291fadd75f6fcb8290854b
Author: Sanjay Patel <[hidden email]>
Date:   Wed Jan 14 16:03:58 2015 +0000

    fix typos
-------------


... to ...

-------------
commit d060b0c7ca67a02d6775febbd0afe47dfb8f1b58
Author: Marek Olsak <[hidden email]>
Date:   Tue Oct 31 21:06:42 2017 +0000

    AMDGPU: Select s_buffer_load_dword with a non-constant SGPR offset
-------------

This took about a week on a shiny new Core i9-7980XE with 64 GB RAM.

I found 649 commits (0.76%) bad enough to not pass my test. The failures were in 309 runs, with the distribution of lengths as follows (number of runs, length of run):

   189 1
     52 2
     21 3
     17 4
     11 5
      7 6
      3 7
      3 8
      1 9
      2 13
      1 14
      1 16
      1 18

The long runs are in general of course caused by some bad commit that was backed out only after a delay.

There may well be other commits that caused less serious problems, or that affected things other than clang or x86.

I expected to find any of the following problems:

- llvm/clang fails to build
- clang crashes while building HelloWorld
- clang errors while building HelloWorld
- HelloWorld crashes
- HellowWorld produces incorrect output

If I recall correctly, I had instances of all of those. But I also encountered one unexpected failure mode:

- clang infinite loops while building HelloWorld

On Thu, Oct 11, 2018 at 3:27 PM, James Y Knight via llvm-dev <[hidden email]> wrote:
TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.

Let me know what you think.

I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.

At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.

This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.

I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.

Some things that could be discussed in such a plan:
  * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).
    * Any particular steps wanted here?
  * Converting buildbots to use git.
  * Phabricator changes?
  * How do email notifications get sent for commits?
  * Gathering github accounts for all committers, adding them to a github team.
  * Deciding upon and announcing a timeline for switching over.
  * Proposing, implementing, and testing new workflows for direct git usage:
    * Github pull requests instead of (or in addition to?) phabricator?
    * Github Protected Branch configuration options?
      * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?
      * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org) to do some testing on pull requests, to reduce avoidable tree breakages?
      * Any other github configuration options that need to be decided upon?
  * ....other things I forgot about at the moment...
  * Timeline for switchover.



Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:
  * Places the SVN revision number into the commit message, as "llvm-svn=1234"

  * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).

  * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.

  * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.

  * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).



Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:
  * cfe (renamed to clang in the conversion)
  * clang-tools-extra
  * compiler-rt
  * debuginfo-tests
  * dragonegg (also "gcc-plugin", the original name)
  * libclc
  * libcxx
  * libcxxabi
  * libunwind
  * lld
  * lldb
  * llgo
  * llvm
  * openmp
  * parallel-libs
  * polly
  * pstl
  * stacker (deleted after r40406)
(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).

Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:
  * lnt
  * test-suite
  * www
  * www-pubs
  * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.
  * zorg

A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:
  * poolalloc
  * safecode

Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":
  * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
  * clang-tests-external # Copy of GDB testsuite
  * llvm-gcc-4.0 # GCC 4.0, modified for llvm
  * llvm-gcc-4.2 # GCC 4.2, modified for llvm
  * llvm-gcc-4-2 # (merge with above)
  * java
  * vmkit
  * nightly-test-server
  * llbrowse # An LLVM bitcode GUI browser
  * television # A different LLVM GUI browser; shows effects of transforms, etc
  * website # 2007-era snapshot of website, not actually maintained here.
  * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.

Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:
  * giri # Never actually developed here; actually https://github.com/liuml07/giri
  * klee # Already migrated to github with history; https://github.com/klee/klee


_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev



_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
1234