NAME
ThreatNet::DATN2004 - Proposal: The Decentralised Active Threat Network
This document has been created to describe a concept that may be of use in a variety of fields. It should be considered a general concept only and is subject to change.
This CPAN/POD version of the document, first published in December 2004 at http://ali.as/devel/threatnetwork.html, has been released to independantly timestamp and archive the concept and proposal in case of future patent-related issues by companies and to attempt to keep the core idea available to all.
INTRODUCTION
On the Internet there exists an increasing number of different ways in which hosts are being misused or abused. Likewise there is also an increasing number of ways in which these known-bad hosts are being identified. Most of these occur in the process of a particular task, such as checking an email message for spam status.
As these hosts are identified, their identify is transmitted across to internet to members of threat networks. The most common of these are the various email "black lists", most of which use DNS or some other method to publish lists of known-bad ips or ip ranges. Mail processing services submit requests to a DNS server storing these lists to determine if a particular host contacting them is a known spammer.
This draft specification describes a system which would be used to identify a specific category of these bad ips, hosts that can be considered "Active Threats". An Active Threat is a host that is currently engaged in anti-social, damaging or criminal behaviour, such as actively sending out spam or viruses. This specification is NOT intended to deal with long-term offenders, as they are addressed by a number of current systems.
If applied to long term offenders, hosts would be registered as an Active Threat when they commence their anti-social behaviour, and fall off any list during periods in which they are not conducting this behaviour.
The general intent is to deal with only those hosts that are actively engaged in damaging behaviour, whether or not they are long term offenders or new offenders. And to deal with the hosts as soon as possible, ideally within a few seconds.
GENERAL PRINCIPLES
The following general principles have been establish for this system, and guide the sample implementation described below. They are listed (loosely) in priority order.
Speed of Response
The ThreatNet concept is specifically intended to address active and transient threats. If a Verified Threat is detected at time $t, other members of the same Threat Network should start to recieve notification before time $t + 1 (within 1 second). Full propagation to all members, and any subsequent responses, should be complete within by $t + 60 (within 1 minute).
Within a second of a host being detected conducting some anti-social behaviour, the member should be able to confirm this behaviour and the host ip responsible for it, and classify the host as a Verified Threat, issuing a message into a Threat Network. Notification to all members of the Threat Network and any other linked networks should occur as quickly as the Internet and Threat Network itself is able to spread this information. Any members who wish to respond to the threat, or are able to neutralise it, should be able to act within a few seconds and certainly within a minute.
For the example of spam, this would allow newly compromised hosts ("zombies") acting as spam or virus agents that are not on current block lists to be dealt with and blocked before they are able to send significant amounts of spam. This could also allow for compromised hosts on dynamic IPs to be blocked each time they move IPs without significantly damaging subsequent users of the IPs, or needing to block the entire IP range. This could also help to reduce the impact of new and fast-moving viruses, as any newly infected computers appearing within properly managed ISPs could be disconnected and disabled before they are able to make a significant number of attempts to reproduce.
Flexibility and Integration
The implementation should be as flexible as possible and integrate with many other current systems, or provide a way for developers to do this integration themselves. Although the initial idea for this concept was as an anti-spam system, it should not be considered solely an anti-spam tool, and should be able to be easily reconfigured for use in any other scenario that involves similar sorts of rapid responses to threats.
The entire system should be defined as a set of separate components, any of which can be replaced, improved or set up in a variety of different ways. Each function or role should be identified separately so that new systems and implementations can provide one particular role without needing to implement all of them.
Distributed and Decentralised
Most existing anti-spam or anti-threat networks involve a central authorative data source, and a central authorative distribution network to spread it, typically a managed database of some sort and a DNS/rsync combination. Centralising in this way can lead to the blocking systems themselves becoming the target of both technical and legal attacks.
Extreme measures are often required to protect (technically and legally) the blocking systems.
This proposal involves a decentralised system with no long term state information and no central server or service that can be implemented in an ad-hoc way. Having limited the scope of the network to short term and transient threat this can be done relatively easily.
Short term state information is held locally at each node, and a mandory "quiet period" is enforced when joining a new network to let the new member synchronise with the network.
This quiet period will keep load and communication volume down, and let new members join a network without the need for a synchronisation event.
By waiting for the network's block period before contributing, each node can be sure that it has fully synchronised with the network before beginning to contribute block entries that might otherwise be duplicates.
Ease of Implementation
In order to get this concept off the ground and running quickly, and to help implement it as flexibly as possible, it is proposed that the system makes use of common, time-tested, well-supported and mature systems wherever possible.
Minimisation of Harm
Long term blocking, and the blocking of entire subnets, may not be suitable for dealing with new and short term threats. One bad apple can ruin it for the entire barrel. While the ability to "poison" a Threat Network may still exist as it does in existing blocking systems, this poisoning would need to be active and continuous, "re-poisoning" the Threat Network with new messages each time the short block period nears its end.
For dynamic IP connections, any response taken against a single IP will expire automatically within the block period, once the host stops use of the IP.
Elegant Degradation and Failure
As with any security system, the greater and higher profile its use, the bigger the target it becomes and more likely to be attacked. The system should be designed from the start to be able to stand up to attack when deployed in high-profile scenarios, and to degrade gracefully where possible to retain at least partial functionality.
EXAMPLE IMPLEMENTATION
The following describes the set of elements which would be required to create a single complete system, and a set of optional elements which could be used to enhance or expand the system.
Required Element: Messaging Platform
At the core of the Threat Network is a messaging platform. As an extremely mature and often-attacked network messaging system, IRC would appear an ideal platform to use for this purpose. Libraries to connect to and work with IRC already exists in almost every programming language. In addition, it can be deployed extremely flexibly from either a non-authenticated channel on an existing IRC network to secret and private dedicated servers with encryption, certificate-backed authentication, and agent-based monitoring.
Required Element: Language Specification
Each member of the network should interact in a standard and relatively flexible language. While this specification does not attempt to describe the details of any particular language, for example purposes only we will start with the simplest possible language. The sample language contains only a single message with only the IP address of a validated threat, assumed to have been detected within a few seconds of the threat event occurring.
Required Element: Network Agents
Within the IRC Threat Network, there exists a number of members, each consisting of at least a software agent. Any agent connected to the system will consist of at least the Event Listener element described below, and one or more optional elements. At least one agent is required to implement the Threat Provider element in order to create a working system.
Required Element: Event Listener
A pure Event Listener is a passive agent that listens on a Threat Network for threat messages and optionally acts upon them. Every agent connected to the Threat Network is required to implement an Event Listener to monitor incoming messages, if only to prevent issuing duplicate messages.
Many of the agents that would implement only the Event Listener would act as interfaces and adapters to other systems.
As one example, a ThreatNet to DNSBL adapter might listen for threat messages, submit a DNSBL request against each IP, and for each IP that isn't already listed, add it to the local DNSBL server with a short TTL most likely matching the ThreatNet block period.
A ThreatNet to firewall adapter could listen for specific types of high-impact threat messages and inject blocks into a firewall configuration, allowing particular types of threats to be blocked at a connection level.
Depending on the scale of the network, and the ability of the adapter to handle the volume of responses, the Event Listener may need to institute some form of queuing or flood control. It would be expected that such flood control would become common were the popularity of Threat Networks to grow significantly.
Optional Element: Threat Cache / Threat Filter
While some Threat Network agents do not need to maintain state, a great many agent types do, in particular anything that wishes to submit Verified Threat messages of its own to the network.
A Threat Cache is simply a small local database, probably in-memory, that stores all "current" threats. A Threat Filter handles and passes on only the subset of events that meet a certain set of criteria. For agents joining a network, the purpose of waiting for the full block period until beginning to submit messages is to allow the agent's Threat Cache to fill and thus synchronise with the rest of the members of the network.
In another example, an ISP may deploy a "Zombie Response" agent that applies a filter to limit the event feed to only those Verified Threats that are the inside the ISP's own network. When the agent identifies an infected or trojaned customer by IP, it could terminate the connection, disable the login, and flag the customer's account so that when they try to reconnect, they will be informed they are infected with a virus or that someone may be using their computer without their permission.
Optional Element: Threat Provider
A Threat Provider is an agent that submits Verified Threats messages to a network. Because any message that is added to the network may result in a variety of actions against the host, care should be taken to verify that the host is indeed an Active Threat.
Any Threat Provider should also act as an Event Listener and ensure that Verified Threat events are not submitted if they are already "current" (in the Threat Cache). The default TTL on all threats will be 1 hour.
As an example, a spam "secondary MX trap" might be tied to a local Threat Network agent that would aggregate and filter threats from the host, and submit Veried Threat messages into the network.
By separating the Threat Network from the individual applications, new and more accurate anti-spam scripts can incorporate a connection to the agent and feed more advanced Verified Threat messages into the same network, be it local, regional, or larger.
Optional Element: Network Bridge
To separate a private company-wide network, or a university network from a larger network, you can make use of a Network Bridge. This is an agent that connects to two or more different networks, maintains a combined Threat Cache and feeds Verified Threat messages from one to the other, potentially in both directions.
For example, a single or small group of universities could run a private Threat Network for open use by members of the University, with a single Network Bridge agent pulling in messages from a larger national or international Threat Network. The bridge would be configured to filter out any messages that reference hosts already listed on the university DNSBL server.
Optional Element: Security Model
One advantage of using IRC is that any of the existing security features that are currently part of IRC can also be applied to a threat network, including voice right, operators, channel management bots, SIRC and more.
At the most open, a single ThreatNet channel could be run within general IRC server with anyone free to participate and submit any entries they wish. This would make it trivially easy to set up an environment for testing use.
More secure than this, a bot could run the channel allowing anyone to join, but only giving voice permissions to agents joining from white-listed IP addresses.
On a larger scale, a large dedicated SIRC network could be created, with server or network-level accounts and monitoring bots to kick and issue Verified Threat events for any anti-social accounts connected to the Threat Network.
CONCLUSION
While not useful for long term blocking, the Decentralised Active Threat Network would dramatically improve the response time to threats, with the goal creating a powerful and effective "nervous system" for detecting and dealing with spammers, virus-infected or trojaned hosts, or other threats for which the damage could be greatly reduced by dealing with them immediately.
GLOSSARY OF TERMS
Active Threat
Any internet host that is currently engaged in anti-social or illegal behaviour such as spamming, virus transmission or acting as part of a DDOS or other attack.
Verified Threat
A host/ip that can be positively and reliably confirmed to be an Active Threat.
Threat Network
An communications network, populated by software agents, for the purpose of rapidly distributing information about, and responding to, Active Threats.
Threat Language
A protocol or message format used to describe a Verified Threat on a Threat Network.
Event Listener
A software agent which is connected to one or more Threat Networks and receives Verified Threat messages from the network.
Threat Cache
A stored list of "current" Active Threats, primarily used to prevent the flooding of duplicate messages onto the Threat Network.
Threat Filter
Any collection of IPs or IP networks against which a stream of events is checked, removing or keeping only a certain subset of all events.
Threat Provider
An software agent which connects to a Threat Network and submits Verified Threat messages to the network.
Network Bridge
A software agent that sits "between" two or more Threat Networks, transmitting new Verified Threat messages between them, after checking against an arbitrary number of Threat Filters.
Threat Response
Some action that undertaken to defend against or counter an Active Threat, in response to a Verified Threat message. This could include short-term DNSBL listing, firewall block entries, or actively disabling the internet connection of the host.
PARTICIPATION
The website http://ali.as/ will provide a link to the current forums of idea exchange for the ThreatNet project.
In addition, there is also a #threatnet channel on FreeNode for testing implementations of the reference implementation described above.
AUTHOR
Adam Kennedy <adamk@cpan.org>
SEE ALSO
ThreatNet, ThreatNet::Message, ThreatNet::Topic
COPYRIGHT
Copyright 2004 - 2008 Adam Kennedy.
All ThreatNet modules on CPAN are generally free software; you can redistribute them and/or modify them under the same terms as Perl itself.
The full text of the license can be found in the LICENSE file included with this documentation.