ISC Networking & Telecomunications
Basic PennNet Service Level Agreement (SLA)

Hours of Operation

ISC Provider Desk (ProDesk) Hours

Network Operations Center (NOC)

Network Operations is responsible for the operation of the PennNet routing core, wide area links to remote campuses, and the Internet and Internet2 gateways. Network Operations also installs and maintains all PennNet wiring and closet electronics in most wiring closets and buildings across campus. The NOC monitors and maintains PennNet by:

• Troubleshooting network problems proactively using monitoring equipment
• Responding to trouble reports called in by LSPs

Network Trouble Call Priority Levels

Network trouble calls are prioritized based on the critical nature of an outage. Major infrastructure problems receive the highest priority, while smaller outages affecting 10 or fewer customers are more limited in scope and therefore are lower on the priority list.

1. Priority 1 (P1) Critical Infrastructure Problem
• Outage to all or a significant part of PennNet
• Considered "Network Down" condition
2. Priority 2 (P2) Major Networking Problem
• Outage involves 96 or more user ports
• Either "Network Down" condition or severe degradation of network performance
3. Priority 3 (P3) Intermediate Networking Problem
• Outage involves at least 10 and not more than 96 user ports
• Either "Network Down" or impaired networking performance in relatively small area of PennNet
• Most of PennNet remains functional
4. Priority 4 (P4) Limited Networking Problem
• Outage involves fewer than 10 users
• Little or no impact on PennNet operations, but an outage for a small group of users
• Also included are administrative support issues

Critical Infrastructure (P1) List By Priority

Problems affecting Critical Infrastructure (P1s) take priority over other networking problems. Critical Infrastructure will be repaired in the following order:

1. PennNet Routing Core
2. PennNet Infrastructure Networks (DNS, DHCP, and NMS Systems)
3. PennNet Network Aggregation Point (NAP) or Fiber Instrastructure
4. Internet/Internet2 Connectivity
5. Building Entrance Devices or PennNet Service Loss in major portion of largely populated building
6. Any Problem that may adversely affect service with items 1-6 in this list
7. Network Equipment covered by Enhanced Service Level Agreements

Reporting and Coordinating Network Outages

Local Support Providers (LSPs) and Information Technology Advisors (ITAs) provide preliminary troubleshooting to confirm that a problem is network-related. Please review the Policy on Troubleshooting Charges for PennNet* which offers "best practice" recommendations to guide an LSP/ITA in preliminary troubleshooting steps. (This policy also defines when labor and material charges will be applied to troubleshooting and remedy of networking problems on PennNet.)

Once an LSP/ITA has determined that a network problem exists, the person must contact ISC ProDesk to log a trouble ticket in order for ISC troubleshooting to begin. Only LSPs and ITAs are authorized to contact ProDesk to report a network problem.

ProDesk will triage network and outage reports and create tickets for those that require the attention of ISC Network Operations. Once a case has been assigned to Network Operations, a NOC technician will communicate with the LSP directly if additional information is needed.

* Located at: http://www.isc-net.upenn.edu/policy/approved/20020827-troubleshooting.html

ProDesk Contact Information

ProDesk After Hours

Only Critical Infrastructure (P1) and Major Network (P2) outages should be reported outside of ProDesk business hours.

• Call the ProDesk (215 573-4778).
• Follow the voicemail menu to report the critical network problem.
• The On-Call NOC technician will be notified.

Outage Coordination

LSPs will provide all outage coordination with their local users. They will also continue to work with the Network Operations Center as needed. Specifically, LSPs will:

• Provide ISC access to suites and wiring centers
• Offer initial and ongoing outage information to the NOC
• Communicate advance notification to their users
• Assist in targeting outage times and dates for necessary repair

Repair Response and Resolution Commitments

Repair response time commitments in this agreement cover central wiring closet PennNet PennNet infrastructure equipment and services. Academic and Administrative departments with critical networking services should contact ISC Networking and Telecommunications to develop a departmental SLA for the PennNet electronics and ports that connect their respective network server(s) or services. Departmental SLAs carry an additional monthly charge which is normally added to the PennNet Port Rental and Connection Maintenance Fee.

* Only Critical Infrastructure (P1) and Major Network (P2) outages should be reported outside of ProDesk business hours. Repair response commitments for P3 and P4 level outages refer to network troubles reported by NOC monitoring equipment.

Repair Response

Repair Response in this agreement refers to the period of time between the Network Operations Center (NOC) being notified of a network problem and the time NOC members begin work to resolve the reported problem.

The Network Operations Center receives trouble call notifications thru:

• ProDesk
• Trouble reports called into ProDesk will be communicated to the NOC within 30 minutes.
• Trouble reports emailed to ProDesk will be communicated to the NOC within 60 minutes.
• ProDesk may choose to escalate an already created trouble ticket.
• NOC monitoring equipment may report network trouble proactively.*

* Communication response times for trouble reported by our network monitoring equipment will meet the same standards (defined in the Communication Response Commitments section below) as response times for tickets reported to ISC by an LSP.

On-site Visits

Repair Response does not necessarily imply an on-site visit.

• Site visits may not be required for every network problem.
• Most often, site visits are required only when a network engineer cannot repair a problem remotely.
• The NOC may sometimes work with an on-site LSP or ITA to resolve a problem.

If an on-site visit is deemed necessary by the Network Operations Center in order to troubleshoot or repair a problem, the visit will occur:

Note: On-site visits after midnight will only be scheduled for Critical Infrastructure (P1) problems.

Resolution Time

Resolution is defined in this agreement as the full repair of the problem, or if necessary, a temporary workaround put into place until the underlying problem can be resolved. Most PennNet problems are resolved well within the 24-hour resolution period. However, occasionally a full resolution will take longer due to mitigating factors such as:

• Intermittent network performance problems that require network analysis or packet tracing
• Delay while waiting for a vendor to produce a software repair for equipment operating system problem
• Delay while waiting for a vendor to ship replacement components
• Incomplete or incorrrect information provided by LSP
• Inability to access wiring center or affected area in building
• Minor disasters such as accidental cutting of cables or fiber trunks
• Major disasters such as flooding and fire

Communication Response Commitments

ProAlert Service is operated by the ProDesk and provides voicemail notification for campus-wide P1 network outages to service subscribers. To subscribe, use the ProAlert online form located at: (http://www.upenn.edu/computing/provider/proalert/proalert.html).

Non-emergency Network Operations Services

Network Moves, Adds, and Changes (MAC) and Wiring Installations

Network Maintenance services are provided M-F 6:00am-5:00pm.

A link to online request forms for port activations, deactivations, and wallplate installations can be found at the top of the PennNet homepage. For information regarding existing requests, contact our Customer Service Team at service-requests@net.isc.upenn.edu or 746-6000.

Please Note:

• These timeframes do not apply to wiring installations and MACs related to projects.
• Premium install and MAC services can be performed during overtime at an additional rate.
• Installation and MAC services performed outside of working hours are best effort.
• Occasionally, unforeseen complications arise which prolong an installation or MAC service request response.

Proactive Network Upgrades (i.e. Preventative Maintenance Commitment)

ISC Networking and Telecommunications Operations is committed to a 3-year ongoing replacement cycle for wiring closet electronics and a 4-year replacement cycle for building entrance and routing core devices. Each component owned and operated by ISC will be replaced within these cycles.

After building or wiring closet electronics have been upgraded, an ISC NOC technical staff member will work with the area's LSP to verify that networking service has been restored to all users in the affected area. This ISC NOC technical staff member will also be available for direct contact throughout the following business day, should any network trouble arise due to the preventive maintenance work.

Outage Periods

Standing Outage Window (Upgrades and Maintenance)

ISC N&T Operations' standing outage window for upgrades and maintenance to the network is Tuesday mornings from 5:00am to 7:00am. The actual start and duration of this period may vary due to the complexity of a particular need and whether N&T Operations can be sustained until the next standing outage window.

Other outage windows may be used depending on the scheduling needs presented by an LSP. These outages may not be globally announced if they are small in scale or considered local to a school or department.

Notification for Scheduled Outages

Notification for Emergency Outages

Emergency P1 and P2 Outages
Emergency P1 outages that require near immediate downtime (i.e. same day) will be scheduled so the outage occurs at or near the end of the University's business day. IT Computing Directors will be notified via email. They may also be notified using the ProAlert voicemail notification service, if necessary.

Emergency Equipment Upgrades and Restarts
Emergency equipment operating system upgrades and restarts may be required due to a vendor's technical support recommendation, particularly after new networking features are deployed. IT Computing Directors will be notified via email in the event an emergency upgrade or restart is scheduled.

Network Operations Services Prioritized

Network Trouble Calls are the Network Operations Center's highest priority.

1. Trouble Calls and Network Maintenance Calls. Each call is logged and prioritized based on the following criteria:
1. Critical Nature of Outage
2. Problem Scope
3. Enhanced Service Level Requirements
4. Order in which Calls are Received
2. Dispatch Services (those that require a technician to make a non-emergency on-site visit).
1. Network Upgrades
2. Moves, Adds, & Changes
3. Enhanced Service Level Agreement Requirements
4. New Wiring Installations
3. Network Administration Services
1. Network Monitoring and Troubleshooting Support
2. Network Equipment Configurations
3. Network Upgrade Support
4. Assignments Duties and General Customer Q&A