Report on Recent Outages

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Report on Recent Outages

John Criswell-5
Dear All,

Some community members have contacted me directly regarding recent
outages of the LLVM services.  While I'm no longer a University of
Illinois employee, old habits die hard, so I asked around. :)

First, a storage array at UIUC developed issues over the weekend. The
UIUC IT people were on it, but the first fix did not appear to work as
expected, causing performance degradation for systems using the storage
array.  I believe the mailing list server is one such system, explaining
why the mailing lists didn't work yesterday.

There will be more work on the storage array between midnight and 8 am
Central time tonight; it is possible that the LLVM/Clang mailing lists
will be down or experiencing degraded performance during that time.

Second, a router in the UIUC CS department failed today, preventing
access to the LLVM server (which hosts SVN, web, and other services).  
This failure was independent of the storage array failure.  To the best
of my knowledge, that is fixed and running again.

On behalf of my alma mater and former employer, I apologize for the
inconvenience.  However, their IT people are aware of the problems and
are working on it.  With luck, everything will be fixed and running
smooth by 8 am Central time tomorrow.

On a final note, if you need to contact me directly, please use my gmail
address.  My illinois.edu address is forwarding mail for now, but
apparently it won't do so forever.

Thanks for your patience everyone.

-- John T.

--
John Criswell
Assistant Professor
Department of Computer Science, University of Rochester
http://www.cs.rochester.edu/u/criswell

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Report on Recent Outages

David Blaikie


On Thu, Dec 18, 2014 at 1:45 PM, John Criswell <[hidden email]> wrote:
Dear All,

Some community members have contacted me directly regarding recent outages of the LLVM services.  While I'm no longer a University of Illinois employee, old habits die hard, so I asked around. :)

Thanks for the update! Who should we be reaching out to in the future when issues like this come up?
 

First, a storage array at UIUC developed issues over the weekend. The UIUC IT people were on it, but the first fix did not appear to work as expected, causing performance degradation for systems using the storage array.  I believe the mailing list server is one such system, explaining why the mailing lists didn't work yesterday.

There will be more work on the storage array between midnight and 8 am Central time tonight; it is possible that the LLVM/Clang mailing lists will be down or experiencing degraded performance during that time.

Second, a router in the UIUC CS department failed today, preventing access to the LLVM server (which hosts SVN, web, and other services).  This failure was independent of the storage array failure.  To the best of my knowledge, that is fixed and running again.

On behalf of my alma mater and former employer, I apologize for the inconvenience.  However, their IT people are aware of the problems and are working on it.  With luck, everything will be fixed and running smooth by 8 am Central time tomorrow.

On a final note, if you need to contact me directly, please use my gmail address.  My illinois.edu address is forwarding mail for now, but apparently it won't do so forever.

Thanks for your patience everyone.

-- John T.

--
John Criswell
Assistant Professor
Department of Computer Science, University of Rochester
http://www.cs.rochester.edu/u/criswell

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: Report on Recent Outages

David Blaikie


On Fri, Dec 19, 2014 at 9:28 AM, David Blaikie <[hidden email]> wrote:


On Thu, Dec 18, 2014 at 1:45 PM, John Criswell <[hidden email]> wrote:
Dear All,

Some community members have contacted me directly regarding recent outages of the LLVM services.  While I'm no longer a University of Illinois employee, old habits die hard, so I asked around. :)

Thanks for the update! Who should we be reaching out to in the future when issues like this come up?

Acgtually, scratch that - whoever it is doesn't need random LLVM community members pestering them. llvm admins no doubt have the right info.
 
 

First, a storage array at UIUC developed issues over the weekend. The UIUC IT people were on it, but the first fix did not appear to work as expected, causing performance degradation for systems using the storage array.  I believe the mailing list server is one such system, explaining why the mailing lists didn't work yesterday.

There will be more work on the storage array between midnight and 8 am Central time tonight; it is possible that the LLVM/Clang mailing lists will be down or experiencing degraded performance during that time.

Second, a router in the UIUC CS department failed today, preventing access to the LLVM server (which hosts SVN, web, and other services).  This failure was independent of the storage array failure.  To the best of my knowledge, that is fixed and running again.

On behalf of my alma mater and former employer, I apologize for the inconvenience.  However, their IT people are aware of the problems and are working on it.  With luck, everything will be fixed and running smooth by 8 am Central time tomorrow.

On a final note, if you need to contact me directly, please use my gmail address.  My illinois.edu address is forwarding mail for now, but apparently it won't do so forever.

Thanks for your patience everyone.

-- John T.

--
John Criswell
Assistant Professor
Department of Computer Science, University of Rochester
http://www.cs.rochester.edu/u/criswell

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] Report on Recent Outages

Greg Bedwell
Thanks for the update! Who should we be reaching out to in the future when issues like this come up?

Acgtually, scratch that - whoever it is doesn't need random LLVM community members pestering them. llvm admins no doubt have the right info.

 I wonder whether it would make sense, if this happens again in future, to make use of alternative channels such as the llvm.org twitter feed (I assume that's official?) or the IRC channel topic line to put out a status update along the lines of "We're aware of the problems with the lists and are working on restoring them" to avoid people being contacted directly about the issue when it's already being taken care of.

--Greg

 

_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Reply | Threaded
Open this post in threaded view
|

Re: [cfe-dev] Report on Recent Outages

Tanya Lattner-2

On Dec 19, 2014, at 11:14 AM, Greg Bedwell <[hidden email]> wrote:

Thanks for the update! Who should we be reaching out to in the future when issues like this come up?

Acgtually, scratch that - whoever it is doesn't need random LLVM community members pestering them. llvm admins no doubt have the right info.

 I wonder whether it would make sense, if this happens again in future, to make use of alternative channels such as the llvm.org twitter feed (I assume that's official?) or the IRC channel topic line to put out a status update along the lines of "We're aware of the problems with the lists and are working on restoring them" to avoid people being contacted directly about the issue when it's already being taken care of.



Please contact me at [hidden email] for any issues related to LLVM infrastructure if the mailing lists are down. If the lists are up, please email llvm-admin mailing list.

I have the contact information for our UIUC contact and would prefer that requests go through me. I’m only human though and there will be sometimes a delay in my response depending upon the time of day.

the llvm.org twitter feed is official and I hope to be utilizing it more often.

Thanks,
Tanya


--Greg

 
_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


_______________________________________________
LLVM Developers mailing list
[hidden email]         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev