A system that identifies abnormal behavior through an adaptive learning function addresses several business problems currently challenging the video surveillance marketplace. Companies have invested in rules-based video analytics software, which tracks pre-programmed behaviors recognized as aggressive, abnormal or suspicious. A cognitive video analytics system, like BRS Labs’ AISight, learns what is normal activity for any given scene or environment, and then accurately recognizes and alerts whenever abnormal activities or behaviors occur. This system is an adaptive learning video analytics solution for large-scale deployments that provides awareness and real-time responsiveness to security watch centers. The system adaptively learns changes to the scene and environment over time and continually updates.