Interrupt storm

In operating systems, an interrupt storm is an event during which a processor receives an inordinate number of interrupts that consume the majority of the processor's time. Interrupt storms are typically caused by hardware devices that do not support interrupt rate limiting.

Background

Because interrupt processing is typically a non-preemptible task in time-sharing operating systems, an interrupt storm will cause low perceived system responsiveness, or even appear to be a complete system freeze. This state is commonly known as "live lock". In such a state, the system is spending so much time processing interrupts that it is not completing any other work. Therefore, it does not appear to be processing anything at all, because of a lack of output to the user, the network, or otherwise. An interrupt storm is sometimes mistaken for thrashing, since they both have similar symptoms, but different causes.

An interrupt storm can have many different causes, including misconfigured or faulty hardware devices, faulty device drivers, or flaws in the operating system. Most modern hardware implement methods for reducing or eliminating the possibility of an interrupt storm. For example, many Ethernet controllers implement interrupt "rate limiting", which causes the controller to wait a programmable minimum amount of time between each interrupt it generates.

The most common interrupt storm is a faulty driver under an APIC (Advanced Programmable Interrupt Controller) where a device "behind" another signals an interrupt to the APIC. The OS then asks each driver on that interrupt if it was from its hardware. Faulty drivers may always claim "yes", but then proceed no further as the hardware attached actually did not interrupt. The device which originally interrupted did not get its interrupt serviced, so interrupts again and the cycle begins anew. The system locks dead under an interrupt storm. This was (and remains) a problem on the SoundBlaster Live! series of sound cards on some motherboards; only a kernel debugger can break the storm by unloading the faulty driver.

Many OSes implement a polling mode that disables interrupts for devices which generate too many interrupts. In this mode, the OS periodically queries the hardware for pending tasks. As the number of interrupts increase and the efficiency of an interrupt mode diminishes, an OS may change the interrupting device from an interrupt mode to a polling mode. Likewise, as the polling mode becomes less efficient than the interrupt mode, the OS will switch the device back to the interrupt mode. The implementation of interrupt rate limiting in hardware almost negates the need for such polling modes.

History

Perhaps the first interrupt storm occurred during the Apollo 11's lunar descent in 1969.

Considerations

Interrupt rate limiting must be carefully configured for optimum results. For example, an Ethernet controller with interrupt rate limiting will buffer the packets it receives from the network in between each interrupt. If the rate is set too high, the controller's buffer will overflow, and packets will be dropped. The rate must take into account how fast the buffer may fill between interrupts, and the interrupt latency between the interrupt and the transfer of the buffer to the system.

Interrupt mitigating

There are hardware-based and software-based approaches to the problem. FreeBSD detects interrupt storms and masks problematic interrupt for some time. Other possible scheme is the one used by NAPI
* System (driver) starts in interrupt enabled state
* Interrupt handler disables the interrupt and lets a thread/task handle the event(s) (example of event is an incoming Ethernet packet)
* Task polls the device, processes some number of events and enables the interrupt

Another interesting approach using hardware support — device generates interrupt when event queue state changes from "empty" to "not empty"

Device
* If there is no free DMA descriptors at the RX FIFO tail drop the event
* Add event to the tail and mark the FIFO entry as occupied
* If entry (tail−1) is free (cleared), generate interrupt (level interrupt)
* Increment tail pointer

CPU (interrupt handler)
* Acknowledge the interrupt (if hardware requires acknowledge)
* Handle all (part of) valid DMA descriptors at head
* return from interrupt

See also

* Broadcast radiation
* Inter-processor interrupt
* Interrupt handler
* Non-maskable interrupt
* Programmable Interrupt Controller