Outage Report

Network/Internet/Service Outage March 5, 2012

The problems with the outage yesterday have now been resolved.  If you are experiencing any problems that you think are related, please contact the Service Desk 801-863-8888.  Below is a summary of the issues that surrounded the outage that lasted from around 9:30am to 3:30pm March 5th with some issues starting as early as Saturday around 10:00am and residual issues lasting as long as March 6th around 9:00am.

Details of the outage – the effects*Saturday March 3  around 10:00am the library databases from EBSCO were unavailable from off campus.
*Monday March 5 around 9:30am the Network, Internet and all UVU services on the Internet became unavailable.  This included printing in the student computer labs, access to all Internet sites from on campus, and access to all UVU websites and services both from on and off campus.
*Monday March 5 around 3:30pm services on and off campus became available.
*Residual effects can continue for about 48 hours where some Internet Service Providers (ISPs) may not resolve uvu.edu addresses.  Specifically some of Digis’s customers could not access service until this morning around 9:00am.

Details of the outage – what happened?
Preliminary investigation indicates that the primary InfoBlox appliance that supplies network and name addresses for UVU (DNS & DHCP) failed on Saturday morning.  The secondary appliance continued to work and supply addresses, however it appears that due to the failure that the secondary appliance (and a third appliance at Wasatch campus) were corrupted which resulted in the failure of the secondary appliance on Monday morning.  The backups from the third appliance were corrupted and could not be used to re-build the secondary appliance which had to be built from an old backup and then updated manually with current information.  This was completed around 3:30pm.  A redundant DNS device was also placed into service.  DNS is propagated across the Internet to ISPs and some ISPs may cache the outage for a period of time up to 48 hours causing some residual problems.  At this time we do not know of any that have not been corrected since 9:00am this morning.

Some rumors were spread via Facebook and social media that UVU would not be back up until Wednesday.  These are obviously false.  Please make sure that if you have questions or hear rumors to check with the university’s IT Service Desk for complete and accurate information.  We appreciate your help in dispelling false rumors and information.

Past Outages Reports

Archive

November 2, 2009 Banner outage
Systems Affected: Banner INB and Banner Self Service
Registrations systems were affected during heavy registration period
Start Time: 7:53am outage was first reported
Time: 10:42am Banner working except pre-requisite checking in registration
End Time: 11:20am everything working including registration and prequisite checking

Executive Summary:
Banner was reported down at 7:53am.  It is suspected that the actual time was approximately 2:00am.  Problem was with the database and processes accessing it.  Servers were eventually re-booted, database checked and restarted and system resumed functioning.  Pipes used in prequisite checking for registration died shortly after system started working.  They were discovered and restarted.  Everything returned to normal at 11:20am.