Thinking about dropout intuitively

For those of you who are familiar with Leonardo Dicaprio's movie Catch me if you can, you'll recall how Leonardo charmed the bank workers by asking them on dates and buying them treats, only so he could defraud the bank by cashing in his fake airline checks. In fact, due to the frequent fraternization of the employees and DiCaprio's character, the bank workers were paying more attention to irrelevant features such as DiCaprio's charm. What they should have actually been paying attention was the fact that DiCaprio was cashing out his monthly salary checks more than three times each month. Needless to say, businesses don't usually behave so generously. Dropping out some neurons is synonymous to rotating them to ensure that none of them get lazy and let a sleazy Leonardo defraud your network.

When we apply a dropout to a layer, we simply drop some of the outputs it would have otherwise given. Suppose a layer produces the vector [3, 5, 7, 8, 1] as an output for a given input. Adding a dropout rate of (0.4) to this layer would simply convert this output to [0, 5, 7, 0, 1]. All we did was initialize 40% of the scalars in our vector as zero.

Dropout only occurs during training. During testing, layers with dropouts have their outputs scaled down by the factor of the dropout rate that was previously used. This is actually done to adjust for the fact that more neurons are active during testing than training, as a result of the dropout mechanism.