Basic Service Level Agreement (SLA)
|SLA Contract Number||SLA10202002-PennNet|
|Customer||Standard PennNet Customers|
|Technical Contact(s)||LSP Community|
|Administrative Contact(s)||Departmental Business Administrators|
|Scope||All Standard PennNet Electronics and Wiring.
(Private Networks and Local Wiring not included)
|Status||Under NPTF Review|
|Contract Duration||Ongoing but subject to annual review by NPTF|
|Last Update||August 25,2008 MHW|
ProDesk is a phone and email support service designed to serve the needs of Local Support Providers (LSP) by coordinating troubleshooting efforts with other ISC groups. ProDesk routes networking trouble reports to network technicians in the Network Operations Center.Network Operations Center (NOC)
Network Operations is responsible for the operation of the PennNet routing core, wide area links to remote campuses, and the Internet and Internet2 gateways. Network Operations also installs and maintains all PennNet wiring and closet electronics in most wiring closets and buildings across campus. The NOC monitors and maintains PennNet by:
- Troubleshooting network problems proactively using monitoring equipment
- Responding to trouble reports called in by LSPs
|Regular Business Hours||M-F 9:00am to 5:00pm|
|Off Hours Voicemail/Paging System
For Priority 1 (P1) Critical Infrastructure Problems Only.
|M-F 5:00pm to 9:00am|
|Network Operations Center||Hours|
|First Shift||M-F 6:00am to 5:00pm
|Second Shift||M-F 3:00pm to 11:00pm
(NOC Monitoring and Dispatch Services)
|On Call Service||M-F 11:00pm to 8:00am
Sat-Mon 8:00am to 8:00am
All University Holidays
Network trouble calls are prioritized based on the critical nature of an outage. Major infrastructure problems receive the highest priority, while smaller outages affecting 10 or fewer customers are more limited in scope and therefore are lower on the priority list.
- Priority 1 (P1) Critical Infrastructure Problem
- Outage to all or a significant part of PennNet
- Considered "Network Down" condition
- Outage involves 96 or more user ports
- Either "Network Down" condition or severe degradation of network performance
- Outage involves at least 10 and not more than 96 user ports
- Either "Network Down" or impaired networking performance in relatively small area of PennNet
- Most of PennNet remains functional
- Outage involves fewer than 10 users
- Little or no impact on PennNet operations, but an outage for a small group of users
- Also included are administrative support issues
Problems affecting Critical Infrastructure (P1s) take priority over other networking problems. Critical Infrastructure will be repaired in the following order:
- PennNet Routing Core
- PennNet Infrastructure Networks (DNS, DHCP, and NMS Systems)
- PennNet Network Aggregation Point (NAP) or Fiber Instrastructure
- Internet/Internet2 Connectivity
- Building Entrance Devices or PennNet Service Loss in major portion of largely populated building
- Any Problem that may adversely affect service with items 1-6 in this list
- Network Equipment covered by Enhanced Service Level Agreements
PennNet Phone is a standard networking service that falls into the Priority Level Four (P4) support category offered by ISC Network Operations.
AirPennNet Expected Level of ServiceAcademic and Administrative Buildings and Open Spaces
AirPennNet is considered a subscribed service or customer-ordered service that offers additional and un-tethered access to PennNet. IT representatives from Schools and Departments work with ISC N&T to design AirPennNet in their respective spaces in order to provide wireless access to what is called intended areas of coverage. In some cases, entire buildings have complete wireless coverage, and in other areas only a select set of offices, classrooms or common spaces have wireless coverage. AirPennNet service in all intended areas of coverage are fully supported and covered within the monthly rate for wireless service and in the Basic PennNet SLA. Omitted or fringe areas or areas outside of intended coverage are not in scope of maintenance support, but are best effort service. Customers who wish to have fringe areas fully covered can order a site survey and new wireless installation by sending and email to email@example.com. This is at an additional cost to the requester.AirPennNet Priority Level Response
All AirPennNet Access Points (AP’s) are monitored by the ISC Network Operations Center. Since most spaces on campus involve the installation of multiple AirPennNet AP’s, loss of service to one AP is considered a Priority Level Three (P3).
In this instance AirPennNet service, though slightly degraded, should still provide acceptable wireless service to most remaining parts of the building or intended coverage areas until the problem gets resolved within the P3 committed timeframe. During this timeframe, end users that require PennNet access can relocate to another area, or connect to PennNet though a wired PennNet port. Other outages to PennNet that affect the building network, and therefore more than one AP will cause the priority level to be increased to Level 2 (P2).AirPennNet-Guest
AirPennNet-Guest is a web-based wireless network with limited bandwidth and offers no encryption. It is set up for visitors to Penn or for those with devices not capable of supporting 802.1x. AirPennNet-Guest is available in all areas where AirPennNet is in operation, except in the residence halls. Since most spaces on campus involve the installation of multiple AirPennNet AP’s, loss of service to one AP is considered a Priority Level Four (P4). In this instance AirPennNet-Guest service, though slightly degraded, should still provide acceptable wireless service to most remaining parts of the building or intended coverage areas until the problem gets resolved within the P4 committed timeframe. During this timeframe, end users that require PennNet access can relocate to another area, or connect to PennNet though a wired PennNet port. Other outages to PennNet that affect the building network, and therefore more than one AP will cause the priority level to be increased to Level 3 (P3).
Reporting and Coordinating Network Outages
Local Support Providers (LSPs) and Information Technology Advisors (ITAs) provide preliminary troubleshooting to confirm that a problem is network-related. Please review the Policy on Troubleshooting Charges for PennNet* which offers "best practice" recommendations to guide an LSP/ITA in preliminary troubleshooting steps. (This policy also defines when labor and material charges will be applied to troubleshooting and remedy of networking problems on PennNet.)
Once an LSP/ITA has determined that a network problem exists, the person must contact ISC ProDesk to log a trouble ticket in order for ISC troubleshooting to begin. Only LSPs and ITAs are authorized to contact ProDesk to report a network problem.
ProDesk will triage network and outage reports and create tickets for those that require the attention of ISC Network Operations. Once a case has been assigned to Network Operations, a NOC technician will communicate with the LSP directly if additional information is needed.
* Located at: http://www.isc-net.upenn.edu/policy/approved/20020827-troubleshooting.htmlProDesk After Hours
Only Critical Infrastructure (P1) and Major Network (P2) outages should be reported outside of ProDesk business hours.
- Call the ProDesk (215 573-4778).
- Follow the voicemail menu to report the critical network problem.
- The On-Call NOC technician will be notified.
LSPs will provide all outage coordination with their local users. They will also continue to work with the Network Operations Center as needed. Specifically, LSPs will:
- Provide ISC access to suites and wiring centers
- Offer initial and ongoing outage information to the NOC
- Communicate advance notification to their users
- Assist in targeting outage times and dates for necessary repair
Repair response time commitments in this agreement cover central wiring closet PennNet PennNet infrastructure equipment and services. Academic and Administrative departments with critical networking services should contact ISC Networking and Telecommunications to develop a departmental SLA for the PennNet electronics and ports that connect their respective network server(s) or services. Departmental SLAs carry an additional monthly charge which is normally added to the PennNet Port Rental and Connection Maintenance Fee.
|Priority Level||ProDesk Business Hours
|ProDesk Off Hours||Resolution Time|
|P1 Critical Infrastructure||Near Immediate||Within 30 Minutes||Work Until Resolved|
|P2 Major||Within 1 Hour||Within 2 Hours||Within 24 Hours|
|P3 Intermediate||Within 2 Hours||Within 4 Hours*||Next Business Day|
|P4 Limited||Within 4 Hours||Within Next Business Day*||Next Business Day|
* Only Critical Infrastructure (P1) and Major Network (P2) outages should be reported outside of ProDesk business hours. Repair response commitments for P3 and P4 level outages refer to network troubles reported by NOC monitoring equipment.Repair Response
Repair Response in this agreement refers to the period of time between the Network Operations Center (NOC) being notified of a network problem and the time NOC members begin work to resolve the reported problem.
The Network Operations Center receives trouble call notifications thru:
- Trouble reports called into ProDesk will be communicated to the NOC within 30 minutes.
- Trouble reports emailed to ProDesk will be communicated to the NOC within 60 minutes.
- ProDesk may choose to escalate an already created trouble ticket.
- NOC monitoring equipment may report network trouble proactively.*
* Communication response times for trouble reported by our network monitoring equipment will meet the same standards (defined in the Communication Response Commitments section below) as response times for tickets reported to ISC by an LSP.On-site Visits
Repair Response does not necessarily imply an on-site visit.
- Site visits may not be required for every network problem.
- Most often, site visits are required only when a network engineer cannot repair a problem remotely.
- The NOC may sometimes work with an on-site LSP or ITA to resolve a problem.
If an on-site visit is deemed necessary by the Network Operations Center in order to troubleshoot or repair a problem, the visit will occur:
- Within 2 hours during normal ProDesk Business Hours
- Within 4 hours during ProDesk off hours
Note: On-site visits after midnight will only be scheduled for Critical Infrastructure (P1) problems.
Resolution is defined in this agreement as the full repair of the problem, or if necessary, a temporary workaround put into place until the underlying problem can be resolved. Most PennNet problems are resolved well within the 24-hour resolution period. However, occasionally a full resolution will take longer due to mitigating factors such as:
- Intermittent network performance problems that require network analysis or packet tracing
- Delay while waiting for a vendor to produce a software repair for equipment operating system problem
- Delay while waiting for a vendor to ship replacement components
- Incomplete or incorrrect information provided by LSP
- Inability to access wiring center or affected area in building
- Minor disasters such as accidental cutting of cables or fiber trunks
- Major disasters such as flooding and fire
|ProDesk Business Hours
|ProDesk Off Hours|
|P1 Critical Infrastructure||Campus-wide Email Announcement or ProAlert Service (if necessary)||First communication within 2 hours. If severity of outage warrants, communications will occur more frequently.||Same as First and Second Shift response.|
|P2 Major||NOC communication to building IT support staff||First communication within 4 hours. If severity of outage warrants, communications will occur more frequently.||Start of Business Day|
|P3 Intermediate||NOC communication
to ticket requestor
|End of Business Day||Start of Business Day|
|P4 Limited||NOC communication
to ticket requestor
|End of Business Day||End of Business Day
ProAlert Service is operated by the ProDesk and provides voicemail notification for campus-wide P1 network outages to service subscribers. To subscribe, use the ProAlert online form located at: (http://www.upenn.edu/computing/provider/proalert/proalert.html).
Network Maintenance services are provided M-F 6:00am-5:00pm.
A link to online request forms for port activations, deactivations, and wallplate installations can be found at the top of the PennNet homepage. For information regarding existing requests, contact our Customer Service Team at firstname.lastname@example.org or 746-6000.
|Service||Time to Complete Request|
|Rapid Activation||90% will be complete within 24 hours|
|Standard Activation||Within 3 business days|
|Standard Wiring Installation||Within 2 weeks|
|Disconnect Requests||Within 2 weeks|
- These timeframes do not apply to wiring installations and MACs related to projects.
- Premium install and MAC services can be performed during overtime at an additional rate.
- Installation and MAC services performed outside of working hours are best effort.
- Occasionally, unforeseen complications arise which prolong an installation or MAC service request response.
ISC Networking and Telecommunications Operations is committed to a 3-year ongoing replacement cycle for wiring closet electronics and a 4-year replacement cycle for building entrance and routing core devices. Each component owned and operated by ISC will be replaced within these cycles.
After building or wiring closet electronics have been upgraded, an ISC NOC technical staff member will work with the area's LSP to verify that networking service has been restored to all users in the affected area. This ISC NOC technical staff member will also be available for direct contact throughout the following business day, should any network trouble arise due to the preventive maintenance work.
ISC N&T Operations' standing outage window for upgrades and maintenance to the network is Tuesday mornings from 5:00am to 7:00am. The actual start and duration of this period may vary due to the complexity of a particular need and whether N&T Operations can be sustained until the next standing outage window.
Other outage windows may be used depending on the scheduling needs presented by an LSP. These outages may not be globally announced if they are small in scale or considered local to a school or department.Notification for Scheduled Outages
Scheduled outages will be announced to the campus community at least 3 business days in advance for all outages that affect an entire building, and at least 5 business days in advance for outages that affect the PennNet routing core.Notification for Emergency Outages
Emergency P1 and P2 outages that require near immediate downtime (i.e. same day) will be scheduled so the outage occurs at or near the end of the University's business day. IT Computing Directors will be notified via email. They may also be notified using the ProAlert voicemail notification service, if necessary.
Emergency Equipment Upgrades and Restarts
Emergency equipment operating system upgrades and restarts may be required due to a vendor's technical support recommendation, particularly after new networking features are deployed. IT Computing Directors will be notified via email in the event an emergency upgrade or restart is scheduled.
Network Trouble Calls are the Network Operations Center's highest priority.
- Trouble Calls and Network Maintenance Calls. Each call is logged and prioritized based on the following criteria:
- Critical Nature of Outage
- Problem Scope
- Enhanced Service Level Requirements
- Order in which Calls are Received
- Dispatch Services (those that require a technician to make a non-emergency on-site visit).
- Network Upgrades
- Moves, Adds, & Changes
- Enhanced Service Level Agreement Requirements
- New Wiring Installations
- Network Administration Services
- Network Monitoring and Troubleshooting Support
- Network Equipment Configurations
- Network Upgrade Support
- Assignments Duties and General Customer Q&A