Network/Internet/Service Outage March 5, 2012
The problems with the outage yesterday have now been resolved. If you are experiencing
any problems that you think are related, please contact the HelpDesk 801-863-8888.
Below is a summary of the issues that surrounded the outage that lasted from around
9:30am to 3:30pm March 5th
with some issues starting as early as Saturday around 10:00am and residual issues
lasting as long as March 6th
around 9:00am.Details of the outage – the effects
*Saturday March 3 around 10:00am the library databases from EBSCO were unavailable
from off campus.
*Monday March 5 around 9:30am the Network, Internet and all UVU services on the Internet
became unavailable. This included printing in the student computer labs, access to
all Internet sites from on campus, and access to all UVU websites and services both
from on and off campus.
*Monday March 5 around 3:30pm services on and off campus became available.
*Residual effects can continue for about 48 hours where some Internet Service Providers
(ISPs) may not resolve uvu.edu addresses. Specifically some of Digis’s customers
could not access service until this morning around 9:00am.
Details of the outage – what happened?
Preliminary investigation indicates that the primary InfoBlox appliance that supplies
network and name addresses for UVU (DNS & DHCP) failed on Saturday morning. The secondary
appliance continued to work and supply addresses, however it appears that due to the
failure that the secondary appliance (and a third appliance at Wasatch campus) were
corrupted which resulted in the failure of the secondary appliance on Monday morning.
The backups from the third appliance were corrupted and could not be used to re-build
the secondary appliance which had to be built from an old backup and then updated
manually with current information. This was completed around 3:30pm. A redundant
DNS device was also placed into service. DNS is propagated across the Internet to
ISPs and some ISPs may cache the outage for a period of time up to 48 hours causing
some residual problems. At this time we do not know of any that have not been corrected
since 9:00am this morning.
Some rumors were spread via Facebook and social media that UVU would not be back up
until Wednesday. These are obviously false. Please make sure that if you have questions
or hear rumors to check with the university’s IT HelpDesk for complete and accurate
information. We appreciate your help in dispelling false rumors and information.
Past Outages Reports
ArchiveNovember 2, 2009 Banner outage
Systems Affected: Banner INB and Banner Self Service
Registrations systems were affected during heavy registration period
Start Time: 7:53am outage was first reported
Time: 10:42am Banner working except pre-requisite checking in registration
End Time: 11:20am everything working including registration and prequisite checkingExecutive Summary:
Banner was reported down at 7:53am. It is suspected that the actual time was approximately
2:00am. Problem was with the database and processes accessing it. Servers were eventually
re-booted, database checked and restarted and system resumed functioning. Pipes used
in prequisite checking for registration died shortly after system started working.
They were discovered and restarted. Everything returned to normal at 11:20am.