# Overview A brief overview ![](/files/-LqkPRu0bWV45RG8oYvh) The *Netcap* (NETwork CAPture) framework efficiently converts a stream of network packets into platform neutral type-safe structured audit records that represent specific protocols or custom abstractions. These audit records can be stored on disk or exchanged over the network, and are well suited as a data source for machine learning algorithms. Since parsing of untrusted input can be dangerous and network data is potentially malicious, implementation was performed in a programming language that provides a garbage collected memory safe runtime. It was developed for a series of experiments in my bachelor thesis: *Implementation and evaluation of secure and scalable anomaly-based network intrusion detection*. Currently, the thesis serves as documentation until the wiki is ready, it is included at the root of this repository (file: [mied18.pdf](https://github.com/dreadl0ck/netcap/blob/master/mied18.pdf)). Slides from my presentation at the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities are available on [researchgate](https://www.researchgate.net/project/Anomaly-based-Network-Security-Monitoring). The project won the 2nd Place at Kaspersky Labs SecurIT Cup 2018 in Budapest. *Netcap* uses Google's Protocol Buffers to encode its output, which allows accessing it across a wide range of programming languages. Alternatively, output can be emitted as comma separated values, which is a common input format for data analysis tools and systems. The tool is extensible and provides multiple ways of adding support for new protocols, while implementing the parsing logic in a memory safe way. It provides high dimensional data about observed traffic and allows the researcher to focus on experimenting with novel approaches for detecting malicious behavior in network environments, instead of fiddling with data collection mechanisms and post processing steps. It has a concurrent design that makes use of multi-core architectures. The name *Netcap* was chosen to be simple and descriptive. The command-line tool was designed with usability and readability in mind, and displays progress when processing packets. The latest version offers 58 audit record types of which 53 are protocol specific and 5 are flow models. ## Design Goals * memory safety when parsing untrusted input * ease of extension * output format interoperable with many different programming languages * concurrent design * output with small storage footprint on disk * maximum data availability * allow implementation of custom abstractions * rich platform and architecture support ## Framework Components Currently there are 8 applications: * net.capture (capture audit records live or from dumpfiles) * net.dump (dump with audit records in various formats) * net.label (tool for creating labeled CSV datasets from netcap data) * net.collect (collection server for distributed collection) * net.agent (sensor agent for distributed collection) * net.proxy (http reverse proxy for capturing traffic from web services) * net.util (utility tool for validating audit records and converting timestamps) * net.export (exporter for prometheus metrics) ## Use Cases * monitoring honeypots * monitoring medical / industrial devices * research on anomaly-based detection mechanisms * Forensic data analysis ## Demos A simple demonstration of generating audit records from a PCAP dump file, querying and displaying the collected information in various ways {% embed url="" %} And live operation decoding traffic from my wireless network interface, while I am surfing the web {% embed url="" %} Watch a quick demo of the deep neural network for classification of malicious behavior, on a small PCAP dump file with traffic from the LOKI Bot. First, the PCAP file is parsed with [netcap](https://github.com/dreadl0ck/netcap-tf-dnn/blob/master/github.com/dreadl0ck/netcap), in order to get audit records that will be labeled afterwards with the [netlabel](https://github.com/dreadl0ck/netcap#netlabel-command-line-tool) tool. The labeled CSV data for the TCP audit record type is then used for training (75%) and evaluation (25%) of the classification accuracy provided by the deep neural network. {% embed url="" %} ## License Netcap is licensed under the GNU General Public License v3, which is a very permissive open source license, that allows others to do almost anything they want with the project, except to distribute closed source versions. This license type was chosen with Netcaps research purpose in mind, and in the hope that it leads to further improvements and new capabilities contributed by other researchers on the long term. For more infos refer to the License page. ## Source Code Stats Stats for netcap v0.4, generated with cloc version 1.80 > $ zeus cloc ``` 332 text files. 332 unique files. 128 files ignored. github.com/AlDanial/cloc v 1.80 T=0.17 s (1167.8 files/s, 117408.0 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Go 196 2692 3655 13540 Markdown 8 106 0 517 ------------------------------------------------------------------------------- SUM: 204 2798 3655 14057 ------------------------------------------------------------------------------- ``` # Protocol Support An overview of supported protocols and available fields ![](/files/-LdpA6jh9XsIJkiZhBTA) ![](/files/-LdpA6jgtpL_Sfaz_G8d) ![](/files/-LdpA6jfxysogKTvoQca) # Specification The Netcap audit record format *Netcap* files have the file extension **.ncap** or **.ncap.gz** if compressed with gzip and contain serialized protocol buffers of one type. Naming of each file happens according to the naming in the [gopacket](https://godoc.org/github.com/google/gopacket) library: a short uppercase letter representation for common protocols, and a camel case version full word version for less common protocols. Audit records are modeled as protocol buffers. Each file contains a header that specifies which type of audit records is inside the file, what version of *Netcap* was used to generate it, what input source was used and what time it was created. Each audit record should be tagged with the timestamp the packet was seen, in the format *seconds.microseconds*. Output is written to a file that represents each data structure from the protocol buffers definition, i.e. *TCP.ncap*, *UDP.ncap*. For this purpose, the audit records are written as length delimited records into the file. ## Delimited Protocol Buffer Records The data format on disk consists of gzipped length-delimited byte records. Each delimited Protocol Buffer record is preceded by a variable-length encoded integer (varint) that specifies the length of the serialized protocol buffer record in bytes. A stream consists of a sequence of such records packed consecutively without additional padding. There are no checksums or compression involved in this processing step. ![Delimited protocol buffers](/files/-LqkPRrpLszNUluCwxRj) ## Data Compression Encoding the output as protocol buffers does not help much with reducing the size, compared to the CSV format. To further reduce the disk size required for storage, the data is gzipped prior to writing it into the file. This makes the resulting files around 70% smaller. Gzip is a common and well supported format, support for decoding it exists in almost every programming language. If this is not desired for e.g. direct access to the stored data, this can be toggled with the -comp command-line flag. ## Audit Records A piece of information produced by Netcap is called an audit record. Audit records are type safe structured data, encoded as protocol buffers. An audit record can describe a specific protocol, or other abstractions built on top of observations from the analyzed traffic. Netcap does currently not enforce the presence of any special fields for each audit record, however by convention each audit record should have a timestamp with microsecond precision. A record file contains a header followed by a list of length-delimited serialized audit records. Naming of the audit record file happens according to the encoder name and should signal whether the file contents are compressed by adding the .gz extension. ![](/files/-LqkPRrrtMgACgP3sxdO) # Installation Setup instructions ## Go Get Installation via go get: ``` $ go get -u github.com/dreadl0ck/netcap/... ``` ## Development Build To install the command-line tool: ``` $ go build -o $(go env GOPATH)/bin/netcap -i github.com/dreadl0ck/netcap/cmd ``` ## Cross Compilation To cross compile for other architectures, set the *GOARCH* and *GOOS* environment variables. For example to cross compile a binary for *linux amd64*: ``` $ GOARCH=amd64 GOOS=linux go build -o netcap -i github.com/dreadl0ck/netcap/cmd ``` ## Homebrew Install the *netcap* command-line tool with Homebrew: ``` $ brew tap dreadl0ck/formulas $ brew install netcap ``` ## Buildsystem *Netcap* uses the [zeus](https://github.com/dreadl0ck/zeus) buildsystem, it can be found on github along with installation instructions. However, the project can easily be installed without zeus. All shell scripts needed for installation can be found in the *zeus/generated* directory as standalone versions: ``` zeus/generated/install-netcap.sh zeus/generated/install-netlabel.sh zeus/generated/install-sensor.sh zeus/generated/install-server.sh ``` To install the *Netcap* and *Netlabel* command-line tool and the library with zeus, run: ``` $ zeus install ``` ## Tests To execute the unit tests, run the following from the project root: ``` $ go test -v -bench=. ./... ``` # Quickstart For those who can't wait to get their hands dirty. ## Basic Commands Read traffic live from interface, stop with *Ctrl-C* (*SIGINT*): ``` $ net.capture -iface eth0 ``` Read traffic from a dump file (supports PCAP or PCAPNG): ``` $ net.capture -r traffic.pcap ``` Read a netcap dumpfile and print to stdout as CSV: ``` $ net.dump -r TCP.ncap.gz ``` Show the available fields for a specific Netcap dump file: ``` $ net.dump -fields -r TCP.ncap.gz ``` Print only selected fields and output as CSV: ``` $ net.dump -r TCP.ncap.gz -select Timestamp,SrcPort,DstPort ``` Save CSV output to file: ``` $ net.dump -r TCP.ncap.gz -select Timestamp,SrcPort,DstPort > tcp.csv ``` Print output separated with tabs: ``` $ net.dump -r TPC.ncap.gz -tsv ``` Run with 24 workers and disable gzip compression and buffering: ``` $ net.capture -workers 24 -buf false -comp false -r traffic.pcapng ``` Parse pcap and write all data to output directory (will be created if it does not exist): ``` $ net.capture -r traffic.pcap -out traffic_ncap ``` Convert timestamps to UTC: ``` $ net.dump -r TCP.ncap.gz -select Timestamp,SrcPort,Dstport -utc ``` ## Show Audit Record File Header To display the header of the supplied audit record file, the -header flag can be used: ``` $ net.capture -r TCP.ncap.gz -header +----------+---------------------------------------+ | Field | Value | +----------+---------------------------------------+ | Created | 2018-11-15 04:42:22.411785 +0000 UTC | | Source | Wednesday-WorkingHours.pcap | | Version | v0.3.3 | | Type | NC_TCP | +----------+---------------------------------------+ ``` ## Print Structured Audit Records Audit records can be printed structured, this makes use of the *proto.MarshalTextString()* function. This is sometimes useful for debugging, but very verbose. ``` $ net.dump -r TCP.ncap.gz -struc ... NC_TCP Timestamp: "1499255023.848884" SrcPort: 80 DstPort: 49472 SeqNum: 1959843981 AckNum: 3666268230 DataOffset: 5 ACK: true Window: 1025 Checksum: 2348 PayloadEntropy: 7.836586993143013 PayloadSize: 1460 ... ``` ## Print as CSV This is the default behavior. First line contains all field names. ``` $ net.dump -r TCP.ncap.gz Timestamp,SrcPort,DstPort,SeqNum,AckNum,DataOffset,FIN,SYN,RST,PSH,ACK,URG,... 1499254962.234259,443,49461,1185870107,2940396492,5,false,false,false,true,true,false,... 1499254962.282063,49461,443,2940396492,1185870976,5,false,false,false,false,true,false,... ... ``` ## Print as Tab Separated Values To use a tab as separator, the *-tsv* flag can be supplied: ``` $ net.dump -r TCP.ncap.gz -tsv Timestamp SrcPort DstPort Length Checksum PayloadEntropy PayloadSize 1499254962.084372 49792 1900 145 34831 5.19616448 137 1499254962.084377 49792 1900 145 34831 5.19616448 137 1499254962.084378 49792 1900 145 34831 5.19616448 137 1499254962.084379 49792 1900 145 34831 5.19616448 137 ... ``` ## Print as Table The *-table* flag can be used to print output as a table. Every 100 entries the table is printed to stdout. ``` $ net.dump -r UDP.ncap.gz -table -select Timestamp,SrcPort,DstPort,Length,Checksum +--------------------+----------+----------+---------+-----------+ | Timestamp | SrcPort | DstPort | Length | Checksum | +--------------------+----------+----------+---------+-----------+ | 1499255691.722212 | 62109 | 53 | 43 | 38025 | | 1499255691.722216 | 62109 | 53 | 43 | 38025 | | 1499255691.722363 | 53 | 62109 | 59 | 37492 | | 1499255691.722366 | 53 | 62109 | 59 | 37492 | | 1499255691.723146 | 56977 | 53 | 43 | 7337 | | 1499255691.723149 | 56977 | 53 | 43 | 7337 | | 1499255691.723283 | 53 | 56977 | 59 | 6804 | | 1499255691.723286 | 53 | 56977 | 59 | 6804 | | 1499255691.723531 | 63427 | 53 | 43 | 17441 | | 1499255691.723534 | 63427 | 53 | 43 | 17441 | | 1499255691.723682 | 53 | 63427 | 87 | 14671 | ... ``` ## Print with Custom Separator Output can also be generated with a custom separator: ``` $ net.dump -r TCP.ncap.gz -sep ";" Timestamp;SrcPort;DstPort;Length;Checksum;PayloadEntropy;PayloadSize 1499254962.084372;49792;1900;145;34831;5.19616448;137 1499254962.084377;49792;1900;145;34831;5.19616448;137 1499254962.084378;49792;1900;145;34831;5.19616448;137 ... ``` ## Validate generated Output To ensure values in the generated CSV would not contain the separator string, the *-check* flag can be used. This will determine the expected number of separators for the audit record type, and print all lines to stdout that do not have the expected number of separator symbols. The separator symbol will be colored red with ansi escape sequences and each line is followed by the number of separators in red color. The *-sep* flag can be used to specify a custom separator. ``` $ net.util -r TCP.ncap.gz -check $ net.util -r TCP.ncap.gz -check -sep=";" ``` # Packet Collection This section focuses on gathering network packet information with netcap Packets are fetched from an input source (offline dump file or live from an interface) and distributed via round-robin to a pool of workers. Each worker dissects all layers of a packet and writes the generated *protobuf* audit records to the corresponding file. By default, the data is compressed with *gzip* to save storage space and buffered to avoid an overhead due to excessive *syscalls* for writing data to disk. ![Packet collection process](/files/-LqkPRtGLYDzFC-DiXme) ## Encoders Encoders take care of converting decoded packet data into protocol buffers for the audit records. Two types of encoders exist: the [Layer Encoder](https://github.com/dreadl0ck/netcap/blob/master/encoder/layerEncoder.go), which operates on *gopacket* layer types, and the [Custom Encoder](https://github.com/dreadl0ck/netcap/blob/master/encoder/customEncoder.go), for which any desired logic can be implemented, including decoding application layer protocols that are not yet supported by gopacket or protocols that require stream reassembly. ## Unknown Protocols Protocols that cannot be decoded will be dumped in the unknown.pcap file for later analysis, as this contains potentially interesting traffic that is not represented in the generated output. Separating everything that could not be understood makes it easy to reveal hidden communication channels, which are based on custom protocols. ## Error Log Errors that happen in the gopacket lib due to malformed packets or implementation errors are written to disk in the errors.log file, and can be checked by the analyst later. Each packet that had a decoding error on at least one layer will be added to the errors.pcap. An entry to the error log has the following format: ``` Error: Packet: ``` At the end of the error log, a summary of all errors and the number of their occurrences will be appended. ``` ... : ... ``` ## Inclusion and Exclusion of Encoders The *-encoders* flag can be used to list all available encoders. In case not all of them are desired, selective inclusion and exclusion is possible, by using the *-include* and *-exclude* flags. List all encoders: ``` $ netcap -encoders custom: + TLS + LinkFlow + NetworkFlow + TransportFlow + HTTP + Flow + Connection layer: + TCP + UDP + IPv4 + IPv6 + DHCPv4 + DHCPv6 + ICMPv4 + ICMPv6 + ICMPv6Echo ... ``` Include specific encoders (only those named will be used): ``` $ netcap -r traffic.pcap -include Ethernet,Dot1Q,IPv4,IPv6,TCP,UDP,DNS ``` Exclude encoders (this will prevent decoding of layers encapsulated by the excluded ones): ``` $ netcap -r traffic.pcap -exclude TCP,UDP ``` ## Applying Berkeley Packet Filters *Netcap* will decode all traffic it is exposed to, therefore it might be desired to set a berkeley packet filter, to reduce the workload imposed on *Netcap*. This is possible for both live and offline operation. In case a [BPF](https://www.kernel.org/doc/Documentation/networking/filter.txt) should be set for offline use, the [gopacket/pcap](https://godoc.org/github.com/google/gopacket/pcap) package with bindings to the *libpcap* will be used, since setting BPF filters is not yet supported by the native [pcapgo](https://godoc.org/github.com/google/gopacket/pcapgo) package. When capturing live from an interface: ``` $ netcap -iface en0 -bpf "host 192.168.1.1" ``` When reading offline dump files: ``` $ netcap -r traffic.pcap -bpf "host 192.168.1.1" ``` # Audit Record Labeling Label audit records for supervised machine learning ## Introduction The term labeling refers to the procedure of adding classification information to each audit record. For the purpose of intrusion detection this is usually a label stating whether the record is normal or malicious. This is called binary classification, since there are just two choices for the label (good / bad) \[BK14]. Efficient and precise creation of labeled datasets is important for supervised machine learning techniques. To create labeled data, Netcap parses logs produced by suricata and extracts information for each alert. The quality of labels therefore depends on the quality of the used ruleset. In the next step it iterates over the data generated by itself and maps each alert to the corresponding packets, connections or flows. This takes into account all available information on the audit records and alerts. More details on the mapping logic can be found in the implementation chapter. While labeling is usually performed by marking all packets of a known malicious IP address, Netcap implements a more granular and thus more precise approach of mapping labels for each record. Labeling happens asynchronously for each audit record type in a separate goroutine. ![](/files/-LqkPRsCaCdoXS5F4nTq) ## Netlabel command-line Tool In the following common operations with the netlabel tool on the command-line are presented and explained. The tool can be found in the *label/cmd* package. To display the available command-line flags, the *-h* flag must be used: ``` $ net.label -h Usage of netlabel: -collect append classifications from alert with duplicate timestamps to the generated label -description use attack description instead of classification for labels -disable-layers do not map layer types by timestamp -exclude string specify a comma separated list of suricata classifications that shall be excluded from the generated labeled csv -out string specify output directory, will be created if it does not exist -progress use progress bars -r string read specified file, can either be a pcap or netcap audit record file -sep string set separator string for csv output (default ",") -strict fail when there is more than one alert for the same timestamp ``` Scan input pcap and create labeled csv files by mapping audit records in the current directory: ``` $ net.label -r traffic.pcap ``` Scan input pcap and create output files by mapping audit records from the output directory: ``` $ net.label -r traffic.pcap -out output_dir ``` Abort if there is more than one alert for the same timestamp: ``` $ net.label -r taffic.pcap -strict ``` Display progress bar while processing input (experimental): ``` $ net.label -r taffic.pcap -progress ``` Append classifications for duplicate labels: ``` $ net.label -r taffic.pcap -collect ``` # HTTP Proxy Inspect traffic to web applications with a HTTP reverse proxy ## Motivation The **net.proxy** tool allows to quickly spin up monitoring of web applications and retrieving netcap audit records. Since currently, TCP stream reassembly is only supported for IPv4, netcap misses HTTP traffic over IPv6 when decoding traffic from raw packets. By using a simple reverse proxy for HTTP traffic, the operating system handles the stream reassembly and we can make sure no IPv6 traffic is missed. ## Usage Spin up a single proxy instance from the commandline: `$ net.proxy -local 127.0.0.1:4000 -remote http://google.com` Specifiy a custom config file for proxying multiple services: ``` $ net.proxy -config example_config.yml ``` The default config path is **net.proxy-config.yml**, so if this file exists in the folder where you execute the proxy, you do not need to specify it on the commandline. ## Configuration For proxying several services, you need to provide a config file, here is a simple example: ```yaml # Proxies map holds all reverse proxies proxies: service1: local: 127.0.0.1:443 remote: http://127.0.0.1:8080 tls: true service2: local: 127.0.0.1:9999 remote: http://192.168.1.20 service3: local: 127.0.0.1:7000 remote: https://google.com # CertFile for TLS secured connections certFile: "certs/cert.crt" # KeyFile for TLS secured connections keyFile: "certs/cert.key" # Logdir is used as destination for the logfile logdir: "logs" ``` ## Help ``` Usage of net.proxy: -version bool print netcap package version and exit -config string set config file path (default "net.proxy-config.yml") -debug set debug mode -dialTimeout int seconds until dialing to the backend times out (default 30) -idleConnTimeout int seconds until a connection times out (default 90) -local string set local endpoint -maxIdle int maximum number of idle connections (default 120) -remote string set remote endpoint -skipTlsVerify skip TLS verification -tlsTimeout int seconds until a TLS handshake times out (default 15) ``` # USB Capture Capture traffic sent via Universal Serial Bus (USB) protocol ## Live Capture USB live capture is now possible, currently the following Audit Records exist: USB and USBRequestBlockSetup. To capture USB traffic live on macOS, install wireshark and bring up the USB interface: ``` $ sudo ifconfig XHC20 up ``` Now attach netcap and set baselayer to USB: ``` $ net.cap -iface XHC20 -base usb ``` ## Offline from dumpfile To read offline USB traffic from a PCAP file use: ``` $ net.cap -r usb.pcap -base usb ``` Don't forget to set the **-payload** flag if you want to preserve the data being transmitted! # Payload Capture Capture full packet payloads It is now possible to capture payload data for the following protocols: **TCP, UDP, ModbusTCP, USB** This can be enabled with the **-payload** flag: ``` $ net.cap -r traffic.pcap -payload ``` Setting the flag works for both live and offlline capture, afterwards the raw payload bytes are stored in the **Payload** field of the audit records. # Distributed Collection Sensors and Collection Server ## Collection Server Using Netcap as a data collection mechanism, sensor agents can be deployed to export the traffic they see to a central collection server. This is especially interesting for internet of things (IoT) applications, since these devices are placed inside isolated networks and thus the operator does not have any information about the traffic the device sees. Although Go was not specifically designed for this application, it is an interesting language for embedded systems. Each binary contains the complete runtime, which increases the binary size but requires no installation of dependencies on the device itself. Data exporting currently takes place in batches over UDP sockets. Transferred data is compressed in transit and encrypted with the public key of the collection server. Asymmetric encryption was chosen, to avoid empowering an attacker who compromised a sensor, to decrypt traffic of all sensors communicating with the collection server. To increase the performance, in the future this could be replaced with using a symmetric cipher, together with a solid concept for key rotation and distribution. Sensor agents do not write any data to disk and instead keep it in memory before exporting it. ![](/files/-LqkPS5Oqze-4MArTJC8) As described in the concept chapter, sensors and the collection server use UDP datagrams for communication. Network communication was implemented using the go standard library. This section will focus on the procedure of encrypting the communication between sensor and collector. For encryption and decryption, cryptographic primitives from the [golang.org/x/crypto/nacl/box](https://godoc.org/golang.org/x/crypto/nacl/box) package are used. The NaCl (pronounced 'Salt') toolkit was developed by the reowned cryptographer Daniel J. Bernstein. The box package uses *Curve25519*, *XSalsa20* and *Poly1305* to encrypt and authenticate messages. It is important to note that the length of messages is not hidden. Netcap uses a thin wrapper around the functionality provided by the nacl package, the wrapper has been published here: [github.com/dreadl0ck/cryptoutils](https://www.github.com/dreadl0ck/cryptoutils). ## Batch Encryption The collection server generates a keypair, consisting of two 32 byte (256bit) keys, hex encodes them and writes the keys to disk. The created files are named *pub.key* and *priv.key*. Now, the servers public key can be shared with sensors. Each sensor also needs to generate a keypair, in order to encrypt messages to the collection server with their private key and the public key of the server. To allow the server to decrypt and authenticate the message, the sensor prepends its own public key to each message. ![NETCAP batch encryption](/files/-LqkPS5QAwJ9Oe_AWWAC) ## Batch Decryption When receiving an encrypted batch from a sensor, the server needs to trim off the first 32 bytes, to get the public key of the sensor. Now the message can be decrypted, and decompressed. The resulting bytes are serialized data for a batch protocol buffer. After unmarshalling them into the batch structure, the server can append the serialized audit records carried by the batch, into the corresponding audit record file for the provided client identifier. ![NETCAP batch decryption](/files/-LqkPS5SIfer4vlKb7NC) ## Usage Both sensor and client can be configured by using the *-addr* flag to specify an IP address and port. To generate a keypair for the server, the *-gen-keypair* flag must be used: ``` $ netcap-server -gen-keypair wrote keys $ ls priv.key pub.key ``` Now, the server can be started, the location of the file containing the private key must be supplied: ``` $ netcap-server -privkey priv.key -addr 127.0.0.1:4200 ``` The server will now be listening for incoming messages. Next, the sensor must be configured. The keypair for the sensor will be generated on startup, but the public key of the server must be provided: ``` $ netcap-sensor -pubkey pub.key -addr 127.0.0.1:4200 got 126 bytes of type NC_ICMPv6RouterAdvertisement expected [126] got size [73] for type NC_Ethernet got 73 bytes of type NC_Ethernet expected [73] got size [27] for type NC_ICMPv6 got size [126] for type NC_ICMPv6RouterAdvertisement got 126 bytes of type NC_ICMPv6RouterAdvertisement expected [126] got size [75] for type NC_IPv6 got 75 bytes of type NC_IPv6 expected [75] got 27 bytes of type NC_ICMPv6 expected [27] ``` The client will now collect the traffic live from the specified interface, and send it to the configured server, once a batch for an audit record type is complete. The server will log all received messages: ``` $ netcap-server -privkey priv.key -addr 127.0.0.1:4200 packet-received: bytes=2412 from=127.0.0.1:57368 decoded batch NC_Ethernet from client xyz new file xyz/Ethernet.ncap packet-received: bytes=2701 from=127.0.0.1:65050 decoded batch NC_IPv4 from client xyz new file xyz/IPv4.ncap ... ``` When stopping the server with a *SIGINT* (Ctrl-C), all audit record file handles will be flushed and closed properly. # Workers This is where the magic happens ## Introduction To make use of multi-core processors, processing of packets should happen in an asynchronous way. Since Netcap should be usable on a stream of packets, fetching of packets has to happen sequentially, but decoding them can be parallelized. The packets read from the input data source (PCAP file or network interface) are assigned to a configurable number of workers routines via round-robin. Each of those worker routines operates independently, and has all selected encoders loaded. It decodes all desired layers of the packet, and writes the encoded data into a buffer that will be flushed to disk after reaching its capacity. ## Worker [Workers](https://github.com/dreadl0ck/netcap/blob/master/collector/worker.go) are a core concept of *Netcap*, as they handle the actual task of decoding each packet. *Netcap* can be configured to run with the desired amount of workers, the default is 1000, since this configuration has shown the best results on the development machine. Increasing the number of workers also increases the number of runtime operations for goroutine scheduling, thus performance might decrease with a huge amount of workers. It is recommended to experiment with different configurations on the target system, and choose the one that performs best. Packet data fetched from the input source is distributed to a worker pool for decoding in round robin style. Each worker decodes all layers of a packet and calls all available custom encoders. After decoding of each layer, the generated protocol buffer instance is written into the *Netcap* data pipe. Packets that produced an error in the decoding phase or carry an unknown protocol are being written in the corresponding logs and dumpfiles. ![NETCAP worker](/files/-LqkPRrbQWfbxaDAhIFd) ## Buffering Each worker receives its data from an input channel. This channel can be buffered, by default the buffer size is 100, also because this configuration has shown the best results on the development machine. When the buffer size is set to zero, the operation of writing a packet into the channel blocks, until the goroutine behind it is ready for consumption. That means, the goroutine must finish the currently processed packet, until a new packet can be accepted. By configuring the buffer size for all routines to a specific number of packets, distributing packets among workers can continue even if a worker is not finished yet when new data arrives. New packets will be queued in the channel buffer, and writing in the channels will only block if the buffer is full. ![NETCAP buffered workers](/files/-LqkPRrd4dG9z5cIPNiA) ## Data Pipe The Netcap data pipe describes the way from a network packet that has been processed in a worker routine, to a serialized, delimited and compressed record into a file on disk. ![](/files/-LqkPRrgBVB27j8Y3Ppw) # Filtering and Export Process Netcap audit records and extract the data you are interested in ## Exporting Data with net.dump Netcap offers a simple interface to filter for specific fields and select only those of interest. Filtering and exporting specific fields can be performed with all available audit record types, over a uniform command-line interface. By default, output is generated as CSV with the field names added as first line. It is also possible to use a custom separator string. Fields are exported in the order they are named in the select statement. Sub structures of audit records (for example IPv4Options from an IPv4 packet), are converted to a human readable string representation. More examples for using this feature on the command-line can be found in the usage section. ![NETCAP filtering and export](/files/-LqkPRs_idlBPq7GGirv) Netcap offers a simple command-line interface to select fields of interest from the gathered audit records. ## Examples Show available header fields: ``` $ netcap -r UDP.ncap.gz -fields Timestamp,SrcPort,DstPort,Length,Checksum,PayloadEntropy,PayloadSize ``` Print all fields for the supplied audit record: ``` $ netcap -r UDP.ncap.gz 1331904607.100000,53,42665,120,41265,4.863994469989251,112 1331904607.100000,42665,53,53,1764,4.0625550894074385,45 1331904607.290000,51190,53,39,22601,3.1861758166070766,31 1331904607.290000,56434,53,39,37381,3.290856864924384,31 1331904607.330000,137,137,58,64220,3.0267194361875682,50 ... ``` Selecting fields will also define their order: ``` $ netcap -r UDP.ncap.gz -select Length,SrcPort,DstPort,Timestamp Length,SrcPort,DstPort,Timestamp 145,49792,1900,1499254962.084372 145,49792,1900,1499254962.084377 145,49792,1900,1499254962.084378 145,49792,1900,1499254962.084379 145,49792,1900,1499254962.084380 ... ``` Print selection in the supplied order and convert timestamps to UTC time: ``` $ netcap -r UDP.ncap.gz -select Timestamp,SrcPort,DstPort,Length -utc 2012-03-16 13:30:07.1 +0000 UTC,53,42665,120 2012-03-16 13:30:07.1 +0000 UTC,42665,53,53 2012-03-16 13:30:07.29 +0000 UTC,51190,53,39 2012-03-16 13:30:07.29 +0000 UTC,56434,53,39 2012-03-16 13:30:07.33 +0000 UTC,137,137,58 ... ``` To save the output into a new file, simply redirect the standard output: ``` $ netcap -r UDP.ncap.gz -select Timestamp,SrcPort,DstPort,Length -utc > UDP.csv ``` # Downloads A collection of cheatsheets and useful resources ## Publications ### Thesis {% file src="/files/-Le6rw78N\_Zi2DaiqPBi" %} ### Thesis Presentation {% file src="/files/-Le6rYKw68D8TWDxbyfs" %} ### SecurIT Cup 2018 Presentation {% file src="/files/-Le6rjP8HfE769-cZCd5" %} ## Cheatsheets ### List of all supported protocols and fields {% file src="/files/-Ldp9rTD9I6N3Jm3lQNs" %} Netcap - Overview Supported Protocols {% endfile %} ### Command Cheatsheet {% file src="/files/-LdpFCEFi7XkN8c5tCce" %} Netcap General Cheatsheet {% endfile %} # Internals Framework inner workings and Implementation details ## Packages ### cmd The cmd package contains the command-line application. It receives configuration param- eters from command-line flags, creates and configures a collector instance, and then starts collecting data from the desired source. #### label The label package contains the code for creating labeled datasets. For now, the suricata IDS / IPS engine is used to scan the input PCAP and generate alerts. In the future, support could also be added for using YARA. Alerts are then parsed with regular expressions and trans- formed into the label.SuricataAlert type. This could also be replaced by parsing suricatas eve.json event logs in upcoming versions. A suricata alert contains the following information: ```go // SuricataAlert is a summary structure of an alerts contents type SuricataAlert struct { Timestamp string Proto string SrcIP string SrcPort int DstIP string DstPort int Classification string Description string } ``` In the next iteration, the gathered alerts are mapped onto the collected data. For layer types which are not handled separately, this is currently by using solely the timestamp of the packet, since this is the only field required by Netcap, however multiple alerts might exist for the same timestamp. To detect this and throw an error, the -strict flag can be used. The default is to ignore duplicate alerts for the same timestamp, use the first encountered label and ignore the rest. Another option is to collect all labels that match the timestamp, and append them to the final label with the -collect flag. To allow filtering out classifications that shall be excluded, the -excluded flag can be used. Alerts matching the excluded classi- fication will then be ignored when collecting the generated alerts. Flow, Connection, HTTP and TLS records mapping logic also takes source and destination information into consider- ation. The created output files follow the naming convention: NetcapType labeled.csv. The label package includes a standalone command-line application in label/cmd. ### types The types package contains types.AuditRecord interface implementations for each supported protocol, to enable converting data to the CSV format. For this purpose, each protocol must provide a CSVRecord() \[]string and a CSVHeader() \[]string function. Additionally, a NetcapTimestamp() string function that returns the Netcap timestamp must be implemented. ### encoder The encoder package implements conversion of decoded network protocols to protocol buffers. This has to be defined for each supported protocol. Two types of encoders exist: The LayerEncoder and the CustomEncoder. #### Layer Encoder A LayerEncoder operates on a gopacket.Layer and has to provide the gopacket.LayerType constant, as well a handler function to receive the layer and the timestamp and convert it into a protocol buffer. #### Custom Encoder A CustomEncoder operates on a gopacket.Packet and is used to decode traffic into abstrac- tions such as Flows or Connections. To create it a name has to be supplied among three different handler functions to control initialization, decoding and deinitialization. Its handler function receives a gopacket.Packet interface type and returns a proto.Message. The postinit function is called after the initial initialization has taken place, the deinit function is used to teardown any additionally created structures for a clean exit. Both functions are optional and can be omitted by supplying nil as value. ### utils The utils package contains shared utility functions used by several other packages. ### collector The collector package provides an interface for fetching packets from a data source, this can either be a PCAP / PCAPNG file or directly from a named network interface. It is used to implement the command-line interface for Netcap. ### io Primitives for atomic maps and write operations ## Caveats Protocol buffers have a few caveats that developers and researchers should be aware of. First, there are no types for 16 bit signed (int16) and unsigned (uint16) integers in protobuf, also there is no type for unsigned 8 bit integers (uint8). This data type is seen a lot in network protocols, so the question arises how to represent it in protocol buffers. The non-fixed integer types use variable length encoding, so int32 is used instead. The variable-length encoding will take care of not sending the bytes that are not being used. Unfortunately, the mu type is too short for this purpose. Second, protocol buffers require all strings to be encoded as valid UTF-8, otherwise encoding to proto will fail. This means all input data that will be encoded as a string in protobuf must be checked to contain valid UTF-8, or they will create an error upon serialization and end up in the errors.pcap file. If this behavior is not desired strings must be filtered prior to setting them on the protocol buffer instances. Another thing that has to be kept in mind is that Netcap processes packets in parallel, thus the order in which packets are written to the dump file is not guaranteed. In experiments, no mixup was detected, and records were tracked in the correct order. However, under heavy load conditions or with a high number of workers, this might be different. Because of this caveat, the Netcap specification requires each record to preserve the timestamp, in order to allow sorting the packets afterwards, if required. ## Data Race Detection Builds In concurrent programming, shared resources need to be synchronized, in order to guarantee their state when modifying or reading them. If access is not synchronized, race conditions occur, which will lead to faulty program behavior. To avoid this and detect race conditions early in the development cycle, the go toolchain offers compiling the program with the race detector enabled. This will let the application crash with stack traces to assist the developer in debugging, if a data race occurs. Programs with active race detection are slower by the factor of 10 to 100. To compile a Go program with the race detection enabled the -race flag must be added to the compilation command. ## Unit Tests Unit tests have been implemented for parts of the core functionality. Currently there are benchmarks for reading pcap and pcapng data, as well as tests and benchmarks for common utility functions, such progress displaying and time conversions. The tests and benchmarks can be executed from the repository root by running: ``` $ go test -v -bench=. ./... ``` # Metrics Prometheus Metrics ## Introduction Netcap now support exporting prometheus metrics about its go runtime, the collection process and the audit records itself. This feature can be used with the **net.export** tool. ## Configuration Metrics are served by default on **127.0.0.1:7777/metrics**. Configure a prometheus instance to scrape it: ```yaml # reference: https://prometheus.io/docs/prometheus/latest/configuration/configuration/ global: scrape_interval: 15s scrape_timeout: 15s #evaluation_interval: 15s scrape_configs: # process_ metrics - job_name: netcap metrics_path: /metrics scheme: http static_configs: - targets: - 127.0.0.1:7777 ``` ## Usage Export a PCAP dumpfile and serve metrics . ``` $ net.export -r 2017-09-19-traffic-analysis-exercise.pcap ``` Capture and export traffic live from the named interface: ``` $ net.export -iface en0 ``` Export a specific audit record file: ``` $ net.export -r HTTP.ncap.gz ``` Export all audit record files in the current directory: ``` $ net.export . ``` ## Overview Dashboard Preview ![Grafana Dashboard Overview](/files/-LqkPRu3NbjO2GXz3Jei) ## TCP Dashboard Preview ![Grafana Dashboard TCP](/files/-LqkPRu5fN9Ey985AMOD) ## HTTP Dashboard Preview ![Grafana Dashboard HTTP](/files/-LqkPRu7v-GhYWl_OOHq) # Python Integration Read Netcap Audit records from Python ## Source Code The Python library for interacting with netcap audit records has been published here: {% embed url="" %} ## Usage ### Read into python dictionary Currently it is possible to retrieve the audit records as python dictionary: ```python #!/usr/bin/python import pynetcap as nc reader = nc.NCReader('pcaps/HTTP.ncap.gz') reader.read(dataframe=False) print("RECORDS:") print(reader.records) ``` ### Read into pandas dataframe Retrieving the audit records as pandas dataframe: ```python #!/usr/bin/python import pynetcap as nc reader = nc.NCReader('pcaps/HTTP.ncap.gz') reader.read(dataframe=True) print("[INFO] completed reading the audit record file:", reader.filepath) print("DATAFRAME:") print(reader.df) ``` # FAQ Frequently Asked Questions ## Does the tool work in outer space and under vaccum? I have no clue - please try and report the findings :) ## How can I contribute to the project? Please see the Contributing page. ## I have a question, how can I reach you? Contact me by mail: dreadl0ck \[at] protonmail \[dot] ch # Extension Implementing new audit records and features To add support for a new protocol or custom abstraction the following steps need to be performed. ## Protocol Buffer Definitions First, a type definition of the new audit record type must be added to the AuditRecord protocol buffers definitions, as well as a **Type enumeration** following the naming convention with the **NC prefix**. ## Encoder Implementation After recompiling the protocol buffers, a file for the new encoder named after the protocol must be created in the encoder package. The new file must contain a variable created with **CreateLayerEncoder** or **CreateCustomEncoder** depending on the desired encoder type. Depending on the choice of the encoder type, the new variable must be added to the customEncoderSlice in **encoder/customEncoder.go** or layerEncoderSlice in **encoder/layerEncoder.go**. ## Audit Record Interface Implementation Next, the interface for conversion to CSV and JSON and exporting metrics must be implemented in the types package, by creating a new file with the protocol name and implementing the **CSVHeader() \[]string, CSVRecord() \[]string** and **NetcapTimestamp() string** functions of the types.AuditRecord interface. If the new protocol contains sub-structures, functions to convert them to strings need to be implemented as well. ## Add Initializer Finally, the **InitRecord(typ types.Type) (record proto.Message)** function in netcap.go needs to be updated, to initialize the structure for the new type. # Contributing Contributing to the Netcap project ## Issues & Bug Reports Please include the Netcap version, the exact error messages and as much log output as possible! Try to answer these questions in your Bug Report: * What version of Netcap, which OS, which version of OS did you use? * What did you want to do? * What happened instead? * What output did you get? ## Pull Requests Before submitting your pull request please make sure the unit tests execute without errors. ## Feature Requests You have an idea for a new feature? Create a feature draft and open an issue to discuss it. # License License type and terms ## GNU General Public License v3.0 Netcap is licensed under the GNU General Public License v3, which is a very permissive open source license, that allows others to do almost anything they want with the project, except to distribute closed source versions. Permissions of this strong copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. This license type was chosen with Netcaps research purpose in mind, and in the hope that it leads to further improvements and new capabilities contributed by other researchers on the long term. The license can be found here: ## Usage Disclaimer Netcap was developed in a short timeframe as a research project and thus was neither tested nor developed to run in a production environment. The project may contain bugs, that have not yet been discovered. Error handling is not very graceful, in many cases that could have been handled otherwise, the program panics in order to assist in debugging with a stack trace. Until there are further unit tests and the error handling is more robust, using Netcap for other purposes than research is not recommended! # Overview A brief overview ![](/files/-M4pSO6i5GQOHGNfVsY6) The *Netcap* (NETwork CAPture) framework efficiently converts a stream of network packets into platform neutral type-safe structured audit records that represent specific protocols or custom abstractions. These audit records can be stored on disk or exchanged over the network, and are well suited as a data source for machine learning algorithms. Since parsing of untrusted input can be dangerous and network data is potentially malicious, implementation was performed in a programming language that provides a garbage collected memory safe runtime. It was developed for a series of experiments in my bachelor thesis: *Implementation and evaluation of secure and scalable anomaly-based network intrusion detection*. The thesis is included at the root of this repository (file: [mied18.pdf](https://github.com/dreadl0ck/netcap/blob/master/mied18.pdf)) and can be used as an introduction to the framework, its philosophy and architecture. However, be aware that the command-line interface was refactored heavily and the thesis examples refer to very early versions. This documentation contains the latest API and usage examples. Slides from my presentation at the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities are available on [researchgate](https://www.researchgate.net/project/Anomaly-based-Network-Security-Monitoring). The project won the 2nd Place at Kaspersky Labs SecurIT Cup 2018 in Budapest. *Netcap* uses Google's Protocol Buffers to encode its output, which allows accessing it across a wide range of programming languages. Alternatively, output can be emitted as comma separated values, which is a common input format for data analysis tools and systems. The tool is extensible and provides multiple ways of adding support for new protocols, while implementing the parsing logic in a memory safe way. It provides high dimensional data about observed traffic and allows the researcher to focus on experimenting with novel approaches for detecting malicious behavior in network environments, instead of fiddling with data collection mechanisms and post processing steps. It has a concurrent design that makes use of multi-core architectures. The name *Netcap* was chosen to be simple and descriptive. The command-line tool was designed with usability and readability in mind, and displays progress when processing packets. The latest version offers 66 audit record types of which 55 are protocol specific and 8 are custom abstractions, such as flows or transferred files. ## Design Goals * memory safety when parsing untrusted input * ease of extension * output format interoperable with many different programming languages * concurrent design * output with small storage footprint on disk * gather everything, separate what can be understood from what can't * allow implementation of custom abstractions * rich platform and architecture support ## Framework Components The framework consists of 9 logically separate tools compiled into a single binary: * capture (capture audit records live or from dumpfiles) * dump (dump with audit records in various formats) * label (tool for creating labeled CSV datasets from netcap data) * collect (collection server for distributed collection) * agent (sensor agent for distributed collection) * proxy (http reverse proxy for capturing traffic from web services) * util (utility tool for validating audit records and converting timestamps) * export (exporter for prometheus metrics) * transform (maltego transformation plugin) ## Use Cases * monitoring honeypots * monitoring medical / industrial devices * research on anomaly-based detection mechanisms * Forensic data analysis ## Demos A simple demonstration of generating audit records from a PCAP dump file, querying and displaying the collected information in various ways {% embed url="" %} Working with PCAPs {% endembed %} And live operation decoding traffic from my wireless network interface, while I am surfing the web {% embed url="" %} Live Capture {% endembed %} Exploring HTTP audit records {% embed url="" %} HTTP Audit Records {% endembed %} ### Deep Learning Watch a quick demo of the deep neural network for classification of malicious behavior, on a small PCAP dump file with traffic from the LOKI Bot. First, the PCAP file is parsed with [netcap](https://github.com/dreadl0ck/netcap-tf-dnn/blob/master/github.com/dreadl0ck/netcap), in order to get audit records that will be labeled afterwards with the [netlabel](https://github.com/dreadl0ck/netcap#netlabel-command-line-tool) tool. The labeled CSV data for the TCP audit record type is then used for training (75%) and evaluation (25%) of the classification accuracy provided by the deep neural network. {% embed url="" %} Deep Learning with Tensorflow {% endembed %} ## License Netcap is licensed under the GNU General Public License v3, which is a very permissive open source license, that allows others to do almost anything they want with the project, except to distribute closed source versions. This license type was chosen with Netcaps research purpose in mind, and in the hope that it leads to further improvements and new capabilities contributed by other researchers on the long term. For more infos refer to the License page. ## Source Code Stats Stats for netcap v0.5, generated with cloc version 1.80 > $ zeus cloc ``` 444 text files. 444 unique files. 158 files ignored. github.com/AlDanial/cloc v 1.84 T=0.26 s (1090.4 files/s, 116481.5 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Go 277 4191 4788 21031 Markdown 9 123 0 503 YAML 1 5 4 14 ------------------------------------------------------------------------------- SUM: 287 4319 4792 21548 ------------------------------------------------------------------------------- ``` # Audit Records An overview of supported protocols and available fields The following markdown overview was generated using: ``` $ net capture -overview ``` ## NETCAP Overview v0.5 > Documentation: [docs.netcap.io](https://docs.netcap.io) > > ### LayerEncoders | Name | NumFields | Fields | | --------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TCP | 25 | Timestamp, SrcPort, DstPort, SeqNum, AckNum, DataOffset, FIN, SYN, RST, PSH, ACK, URG, ECE, CWR, NS, Window, Checksum, Urgent, Padding, Options, PayloadEntropy, PayloadSize, Payload, SrcIP, DstIP | | UDP | 10 | Timestamp, SrcPort, DstPort, Length, Checksum, PayloadEntropy, PayloadSize, Payload, SrcIP, DstIP | | IPv4 | 17 | Timestamp, Version, IHL, TOS, Length, Id, Flags, FragOffset, TTL, Protocol, Checksum, SrcIP, DstIP, Padding, Options, PayloadEntropy, PayloadSize | | IPv6 | 12 | Timestamp, Version, TrafficClass, FlowLabel, Length, NextHeader, HopLimit, SrcIP, DstIP, PayloadEntropy, PayloadSize, HopByHop | | DHCPv4 | 20 | Timestamp, Operation, HardwareType, HardwareLen, HardwareOpts, Xid, Secs, Flags, ClientIP, YourClientIP, NextServerIP, RelayAgentIP, ClientHWAddr, ServerName, File, Options, SrcIP, DstIP, SrcPort, DstPort | | DHCPv6 | 11 | Timestamp, MsgType, HopCount, LinkAddr, PeerAddr, TransactionID, Options, SrcIP, DstIP, SrcPort, DstPort | | ICMPv4 | 7 | Timestamp, TypeCode, Checksum, Id, Seq, SrcIP, DstIP | | ICMPv6 | 5 | Timestamp, TypeCode, Checksum, SrcIP, DstIP | | ICMPv6Echo | 5 | Timestamp, Identifier, SeqNumber, SrcIP, DstIP | | ICMPv6NeighborSolicitation | 5 | Timestamp, TargetAddress, Options, SrcIP, DstIP | | ICMPv6RouterSolicitation | 4 | Timestamp, Options, SrcIP, DstIP | | DNS | 22 | Timestamp, ID, QR, OpCode, AA, TC, RD, RA, Z, ResponseCode, QDCount, ANCount, NSCount, ARCount, Questions, Answers, Authorities, Additionals, SrcIP, DstIP, SrcPort, DstPort | | ARP | 10 | Timestamp, AddrType, Protocol, HwAddressSize, ProtAddressSize, Operation, SrcHwAddress, SrcProtAddress, DstHwAddress, DstProtAddress | | Ethernet | 6 | Timestamp, SrcMAC, DstMAC, EthernetType, PayloadEntropy, PayloadSize | | Dot1Q | 5 | Timestamp, Priority, DropEligible, VLANIdentifier, Type | | Dot11 | 14 | Timestamp, Type, Proto, Flags, DurationID, Address1, Address2, Address3, Address4, SequenceNumber, FragmentNumber, Checksum, QOS, HTControl | | NTP | 19 | Timestamp, LeapIndicator, Version, Mode, Stratum, Poll, Precision, RootDelay, RootDispersion, ReferenceID, ReferenceTimestamp, OriginTimestamp, ReceiveTimestamp, TransmitTimestamp, ExtensionBytes, SrcIP, DstIP, SrcPort, DstPort | | SIP | 11 | Timestamp, Version, Method, Headers, IsResponse, ResponseCode, ResponseStatus, SrcIP, DstIP, SrcPort, DstPort | | IGMP | 15 | Timestamp, Type, MaxResponseTime, Checksum, GroupAddress, SupressRouterProcessing, RobustnessValue, IntervalTime, SourceAddresses, NumberOfGroupRecords, NumberOfSources, GroupRecords, Version, SrcIP, DstIP | | LLC | 6 | Timestamp, DSAP, IG, SSAP, CR, Control | | IPv6HopByHop | 4 | Timestamp, Options, SrcIP, DstIP | | SCTP | 7 | Timestamp, SrcPort, DstPort, VerificationTag, Checksum, SrcIP, DstIP | | SNAP | 3 | Timestamp, OrganizationalCode, Type | | LinkLayerDiscovery | 5 | Timestamp, ChassisID, PortID, TTL, Values | | ICMPv6NeighborAdvertisement | 6 | Timestamp, Flags, TargetAddress, Options, SrcIP, DstIP | | ICMPv6RouterAdvertisement | 9 | Timestamp, HopLimit, Flags, RouterLifetime, ReachableTime, RetransTimer, Options, SrcIP, DstIP | | EthernetCTP | 2 | Timestamp, SkipCount | | EthernetCTPReply | 4 | Timestamp, Function, ReceiptNumber, Data | | LinkLayerDiscoveryInfo | 8 | Timestamp, PortDescription, SysName, SysDescription, SysCapabilities, MgmtAddress, OrgTLVs, Unknown | | IPSecAH | 7 | Timestamp, Reserved, SPI, Seq, AuthenticationData, SrcIP, DstIP | | IPSecESP | 6 | Timestamp, SPI, Seq, LenEncrypted, SrcIP, DstIP | | Geneve | 12 | Timestamp, Version, OptionsLength, OAMPacket, CriticalOption, Protocol, VNI, Options, SrcIP, DstIP, SrcPort, DstPort | | IPv6Fragment | 9 | Timestamp, NextHeader, Reserved1, FragmentOffset, Reserved2, MoreFragments, Identification, SrcIP, DstIP | | VXLAN | 9 | Timestamp, ValidIDFlag, VNI, GBPExtension, GBPDontLearn, GBPApplied, GBPGroupPolicyID, SrcIP, DstIP | | USB | 20 | Timestamp, ID, EventType, TransferType, Direction, EndpointNumber, DeviceAddress, BusID, TimestampSec, TimestampUsec, Setup, Data, Status, UrbLength, UrbDataLength, UrbInterval, UrbStartFrame, UrbCopyOfTransferFlags, IsoNumDesc, Payload | | LCM | 13 | Timestamp, Magic, SequenceNumber, PayloadSize, FragmentOffset, FragmentNumber, TotalFragments, ChannelName, Fragmented, SrcIP, DstIP, SrcPort, DstPort | | MPLS | 7 | Timestamp, Label, TrafficClass, StackBottom, TTL, SrcIP, DstIP | | Modbus | 12 | Timestamp, TransactionID, ProtocolID, Length, UnitID, Payload, Exception, FunctionCode, SrcIP, DstIP, SrcPort, DstPort | | OSPF | 16 | Timestamp, Version, Type, PacketLength, RouterID, AreaID, Checksum, AuType, Authentication, LSAs, LSU, LSR, DbDesc, HelloV2, SrcIP, DstIP | | OSPF | 16 | Timestamp, Version, Type, PacketLength, RouterID, AreaID, Checksum, Instance, Reserved, Hello, DbDesc, LSR, LSU, LSAs, SrcIP, DstIP | | BFD | 21 | Timestamp, Version, Diagnostic, State, Poll, Final, ControlPlaneIndependent, AuthPresent, Demand, Multipoint, DetectMultiplier, MyDiscriminator, YourDiscriminator, DesiredMinTxInterval, RequiredMinRxInterval, RequiredMinEchoRxInterval, AuthHeader, SrcIP, DstIP, SrcPort, DstPort | | GRE | 21 | Timestamp, ChecksumPresent, RoutingPresent, KeyPresent, SeqPresent, StrictSourceRoute, AckPresent, RecursionControl, Flags, Version, Protocol, Checksum, Offset, Key, Seq, Ack, Routing, SrcIP, DstIP, SrcPort, DstPort | | FDDI | 5 | Timestamp, FrameControl, Priority, SrcMAC, DstMAC | | EAP | 6 | Timestamp, Code, Id, Length, Type, TypeData | | VRRP | 12 | Timestamp, Version, Type, VirtualRtrID, Priority, CountIPAddr, AuthType, AdverInt, Checksum, IPAdresses, SrcIP, DstIP | | EAPOL | 4 | Timestamp, Version, Type, Length | | EAPOLKey | 22 | Timestamp, KeyDescriptorType, KeyDescriptorVersion, KeyType, KeyIndex, Install, KeyACK, KeyMIC, Secure, MICError, Request, HasEncryptedKeyData, SMKMessage, KeyLength, ReplayCounter, Nonce, IV, RSC, ID, MIC, KeyDataLength, EncryptedKeyData | | CiscoDiscovery | 5 | Timestamp, Version, TTL, Checksum, Values | | CiscoDiscoveryInfo | 27 | Timestamp, CDPHello, DeviceID, Addresses, PortID, Capabilities, Version, Platform, IPPrefixes, VTPDomain, NativeVLAN, FullDuplex, VLANReply, VLANQuery, PowerConsumption, MTU, ExtendedTrust, UntrustedCOS, SysName, SysOID, MgmtAddresses, Location, PowerRequest, PowerAvailable, SparePairPoe, EnergyWise, Unknown | | USBRequestBlockSetup | 6 | Timestamp, RequestType, Request, Value, Index, Length | | NortelDiscovery | 7 | Timestamp, IPAddress, SegmentID, Chassis, Backplane, State, NumLinks | | CIP | 12 | Timestamp, Response, ServiceID, ClassID, InstanceID, Status, AdditionalStatus, Data, SrcIP, DstIP, SrcPort, DstPort | | Ethernet/IP | 12 | Timestamp, Command, Length, SessionHandle, Status, SenderContext, Options, CommandSpecific, SrcIP, DstIP, SrcPort, DstPort | | SMTP | 9 | Timestamp, IsEncrypted, IsResponse, ResponseLines, Command, SrcIP, DstIP, SrcPort, DstPort | | Diameter | 13 | Timestamp, Version, Flags, MessageLen, CommandCode, ApplicationID, HopByHopID, EndToEndID, AVPs, SrcIP, DstIP, SrcPort, DstPort | > ### CustomEncoders | Name | NumFields | Fields | | -------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TLSClientHello | 27 | Timestamp, Type, Version, MessageLen, HandshakeType, HandshakeLen, HandshakeVersion, Random, SessionIDLen, SessionID, CipherSuiteLen, ExtensionLen, SNI, OSCP, CipherSuites, CompressMethods, SignatureAlgs, SupportedGroups, SupportedPoints, ALPNs, Ja3, SrcIP, DstIP, SrcMAC, DstMAC, SrcPort, DstPort | | TLSServerHello | 27 | Timestamp, Version, Random, SessionID, CipherSuite, CompressionMethod, NextProtoNeg, NextProtos, OCSPStapling, TicketSupported, SecureRenegotiationSupported, SecureRenegotiation, AlpnProtocol, Ems, SupportedVersion, SelectedIdentityPresent, SelectedIdentity, Cookie, SelectedGroup, Extensions, SrcIP, DstIP, SrcMAC, DstMAC, SrcPort, DstPort, Ja3S | | HTTP | 18 | Timestamp, Proto, Method, Host, UserAgent, Referer, ReqCookies, ResCookies, ReqContentLength, URL, ResContentLength, ContentType, StatusCode, SrcIP, DstIP, ReqContentEncoding, ResContentEncoding, ServerName | | Flow | 17 | TimestampFirst, LinkProto, NetworkProto, TransportProto, ApplicationProto, SrcMAC, DstMAC, SrcIP, SrcPort, DstIP, DstPort, TotalSize, AppPayloadSize, NumPackets, UID, Duration, TimestampLast | | Connection | 17 | TimestampFirst, LinkProto, NetworkProto, TransportProto, ApplicationProto, SrcMAC, DstMAC, SrcIP, SrcPort, DstIP, DstPort, TotalSize, AppPayloadSize, NumPackets, UID, Duration, TimestampLast | | DeviceProfile | 7 | Timestamp, MacAddr, DeviceManufacturer, NumDeviceIPs, NumContacts, NumPackets, Bytes | | File | 12 | Timestamp, Name, Length, Hash, Location, Ident, Source, ContentType, SrcIP, DstIP, SrcPort, DstPort | | POP3 | 7 | Timestamp, Client, Server, AuthToken, User, Pass, NumMails | # Specification The Netcap audit record format *Netcap* files have the file extension **.ncap** or **.ncap.gz** if compressed with gzip and contain serialized protocol buffers of one type. Naming of each file happens according to the naming in the [gopacket](https://godoc.org/github.com/google/gopacket) library: a short uppercase letter representation for common protocols, and a camel case version full word version for less common protocols. Audit records are modeled as protocol buffers. Each file contains a header that specifies which type of audit records is inside the file, what version of *Netcap* was used to generate it, what input source was used and what time it was created. Each audit record should be tagged with the timestamp the packet was seen, in the format *seconds.microseconds*. Output is written to a file that represents each data structure from the protocol buffers definition, i.e. *TCP.ncap*, *UDP.ncap*. For this purpose, the audit records are written as length delimited records into the file. ## Delimited Protocol Buffer Records The data format on disk consists of gzipped length-delimited byte records. Each delimited Protocol Buffer record is preceded by a variable-length encoded integer (varint) that specifies the length of the serialized protocol buffer record in bytes. A stream consists of a sequence of such records packed consecutively without additional padding. There are no checksums or compression involved in this processing step. ![Delimited protocol buffers](/files/-LqkPRrpLszNUluCwxRj) ## Data Compression Encoding the output as protocol buffers does not help much with reducing the size, compared to the CSV format. To further reduce the disk size required for storage, the data is gzipped prior to writing it into the file. This makes the resulting files around 70% smaller. Gzip is a common and well supported format, support for decoding it exists in almost every programming language. If this is not desired for e.g. direct access to the stored data, this can be toggled with the **-comp** command-line flag. ## Audit Records A piece of information produced by Netcap is called an audit record. Audit records are type safe structured data, encoded as protocol buffers. An audit record can describe a specific protocol, or other abstractions built on top of observations from the analyzed traffic. Netcap does currently not enforce the presence of any special fields for each audit record, however by convention each audit record should have a timestamp with microsecond precision. A record file contains a header followed by a list of length-delimited serialized audit records. Naming of the audit record file happens according to the encoder name and should signal whether the file contents are compressed by adding the .gz extension. ![](/files/-LqkPRrrtMgACgP3sxdO) # Installation Setup instructions ## Binary Distributions Compiled versions for macOS, Linux and Windows are available on GitHub: {% embed url="" %} NETCAP GitHub Releases Page {% endembed %} ## Go Get Installation via go get: ``` $ go get -u github.com/dreadl0ck/netcap/... ``` ## Manual Build ``` $ go build -ldflags "-s -w" -o /usr/local/bin/net -i github.com/dreadl0ck/netcap/cmd ``` ## Reproducible Builds via Go Modules In order to provide stable and reproducible builds, Go modules are used to pin the versions of source code dependencies to specific versions. Go has included support for versioned modules as proposed [here](https://golang.org/design/24301-versioned-go) since `1.11`. The initial prototype `vgo` was [announced](https://research.swtch.com/vgo) in February 2018. In July 2018, versioned modules [landed](https://groups.google.com/d/msg/golang-dev/a5PqQuBljF4/61QK4JdtBgAJ) in the main Go repository. They are used by default by the go toolchain starting from version `1.13` . You can read about Go modules here: {% embed url="" %} {% embed url="" %} ## Development Build To install the command-line tool: ``` $ go build -o /usr/local/bin/net -i github.com/dreadl0ck/netcap/cmd ``` ## Cross Compilation To cross compile for other architectures, set the *GOARCH* and *GOOS* environment variables. For example to cross compile a binary for *linux amd64*: ``` $ GOARCH=amd64 GOOS=linux go build -o bin/net -i github.com/dreadl0ck/netcap/cmd ``` ## Homebrew On macOS, you can install the *netcap* command-line tool with Homebrew: ``` $ brew tap dreadl0ck/formulas $ brew install netcap ``` ## Buildsystem *Netcap* uses the [zeus](https://github.com/dreadl0ck/zeus) build system, it can be found on GitHub along with installation instructions: {% embed url="" %} ZEUS Build System GitHub {% endembed %} To install the *Netcap* and *Netlabel* command-line tool and the library with zeus, run: ``` $ zeus install ``` # Kali Linux Installing NETCAP on Kali ## Prepare Kali Image This part covers basic setup of Kali Linux, you can skip this if you already have a configured installation. Grab a fresh Kali image from: > Be careful with old kali images where you log in as the root user, and your default shell is still bash and not zshell, you will need to adjust the commands slightly. If you are using a virtual machine, make sure to supply sufficient resources. Maltego and Netcap are both resource intensive programs, so plan accordingly. It is possible to run on smaller files (10-50MB) and analyse simple graphs with a two core / 2GB RAM VM, but its not a pleasant experience and might crash if you run a larger set of transforms at the same time. I recommend at least 4 cores and 4-8GB RAM, but avoid allocating more than half of your host system resources. After logging in with the default user **kali** and password **kali**, open a terminal prompt. Set language for your keyboard (add to \~/.zshrc to persist), I've chosen the German layout here: ``` $ setxkbmap de ``` Update to latest version of Kali: ``` $ apt update $ apt upgrade -y ``` In case you are using a VMWare machine, ensure to update **open-vm-tools-desktop** and **fuse**, so that the shared clipboard and mounting volumes works: ``` $ sudo apt update $ sudo apt install -y --reinstall open-vm-tools-desktop fuse $ sudo reboot -f ``` Configure any volumes you want to mount and use the script provided by the kali authors to mount it: ``` $ cat </dev/null sudo vmhgfs-fuse -o allow_other -o auto_unmount ".host:/\${folder}" "\${vmwpath}" done sleep 2s EOF $ sudo chmod +x /usr/local/sbin/mount-shared-folders $ sudo mount-shared-folders [i] Mounting PCAPs (/mnt/hgfs/PCAPs) ``` ## Install NETCAP You have two options: install a binary release, or compile from source. ### Install binary release Download the latest release: ### Compile from source #### Install Go I recommend to install Go via snap on debian systems, the aptitude packages are often outdated. ``` # install snap package manager $ sudo apt install snapd $ sudo systemctl start snapd $ sudo systemctl enable snapd # install go $ sudo snap install go --classic # ensure apparmor service is running and will be restarted automatically # to prevent errors when running binaries installed via snap after reboot $ sudo systemctl start apparmor $ sudo systemctl enable apparmor ``` add the following directories as a **prefix** to your PATH in the **\~/.zshrc** file (optionally as well to **/root/.zshrc**): ``` export PATH="/home/kali/go/bin:/snap/bin:/usr/local/bin:$PATH" ``` Optional: To have the correct PATH with all binaries produced by the go compiler and netcap when using sudo as well, update the **secure\_path** override using: ``` $ sudo visudo ... Defaults secure_path="/home/kali/go/bin:/snap/bin:/usr/local/bin:" ``` Source the rc file and verify the go compiler binary is found by the shell: ``` $ source ~/.zshrc $ go version go version go1.16.3 linux/amd64 ``` #### Install Dependencies We need the libpcap development headers: ``` $ sudo apt install libpcap-dev ``` now its time to fetch the NETCAP source code and use the build scripts to compile. ``` $ mkdir -p /home/kali/go/src/github.com/dreadl0ck $ cd /home/kali/go/src/github.com/dreadl0ck $ git clone https://github.com/dreadl0ck/netcap ``` You have two options again: Build with bindings for DPI or compile without. Compiling with the DPI bindings will dynamilly load libndpi and libprotoident at runtime, and therefore these have to be installed on the system. If you do not need the DPI functioniality for now, you can simply compile netcap without. The scripts will also invoke setcap to give the netcap binary permissions to attach to a network interface for live capture. #### Compile with DPI ``` $ cd /home/kali/go/src/github.com/dreadl0ck/netcap $ zeus/scripts/install-debian.sh ``` This will fetch and compile libprotoident and DPI, and then compile NETCAP. The DPI libs are pretty big and compilation will take 10-20mins depending on your hardware. Time to get a coffee. #### Compile without DPI ``` $ cd /home/kali/go/src/github.com/dreadl0ck/netcap $ zeus/generated/install-linux-nodpi.sh ``` Either way, you should now have the **net** binary in your PATH, verify by running: ``` $ net / | _______ ______ _10 |_ _______ ______ ______ / / \ / / \ / 01/ | / / | / / \ / / \ 0010100 /|/011010 /|101010/ /0101010/ 001010 |/100110 | 01 | 00 |00 00 | 10 | __ 00 | / 10 |00 | 01 | 10 | 01 |01001010/ 00 |/ |01 \_____ /0101000 |00 |__10/| 10 | 00 |00/ / | 10 00/ 00/ / |00 00 |00/ 00/ 00/ 10/ 0101000/ 0010/ 0010010/ 0010100/ 1010100/ 00 | Network Protocol Analysis Framework 00 | created by Philipp Mieden, 2018 00/ v0.5.13 available subcommands: > capture capture audit records > util general util toool > proxy http proxy > label apply labels to audit records > export exports audit records > dump utility to read audit record files > collect collector for audit records from agents > transform maltego plugin > help display this help usage: ./net [flags] or: ./net [-h] to get help for the subcommand ``` ### Tab completion Run the following to install the tab completion on your system and enable it for the current shell: ``` # step into the cloned repository $ cd /home/kali/go/src/github.com/dreadl0ck/netcap $ sudo mkdir -p /usr/local/etc/bash_completion.d $ autoload -U +X compinit && compinit $ autoload -U +X bashcompinit && bashcompinit $ sudo cp cmd/net /usr/local/etc/bash_completion.d/net $ sudo chown -R kali /usr/local/etc/bash_completion.d $ . /usr/local/etc/bash_completion.d/net ``` To persist it, append the following **at the end** of your **\~/.zshrc** and **/root/.zshrc** files: ``` autoload -U +X compinit && compinit autoload -U +X bashcompinit && bashcompinit . /usr/local/etc/bash_completion.d/net ``` ### Databases To fetch the netcap databases for data enrichment, first install the git large file storage extension: ``` $ sudo apt install git-lfs ``` Create the filesystem path and ensure the permissions are correct for the kali user: ``` $ sudo mkdir -p /usr/local/etc/netcap $ sudo chown -R kali /usr/local/etc/netcap ``` Now you can use the following command to clone the latest version of the database repository from to the correct place on the filesystem. The tool will ask for confirmation before starting the download and displays the expected file size: ``` $ net util -clone-dbs This will fetch 3.3 GB of data. Proceed? [Y/n]: Cloning into '/usr/local/etc/netcap'... remote: Enumerating objects: 1031, done. remote: Counting objects: 100% (252/252), done. remote: Compressing objects: 100% (152/152), done. remote: Total 1031 (delta 77), reused 252 (delta 77), pack-reused 779 Receiving objects: 100% (1031/1031), 510.95 MiB | 9.35 MiB/s, done. Resolving deltas: 100% (325/325), done. cloned netcap-dbs repository to /usr/local/etc/netcap done! Downloaded databases to /usr/local/etc/netcap/ ``` Now that you have the databases, you are good to go. Remember you can update them using the following command, from any location on the filesystem: ``` $ net util -update-dbs Already up to date. ``` The databases are rebuilt from their sources daily at midnight. You can read more about the resolvers that make use of the DBs here: {% content-ref url="/pages/-M4pSNs9Ewumxyv3nEjO" %} [Resolvers](/resolvers) {% endcontent-ref %} ### Maltego Start Maltego, register an account for the community edition and authenticate. Next, load the netcap configuration for Maltego, by switching to the **Import | Export** Tab and hit **Import Config** ![Import Config](/files/-MaEAhJfI86sU-aGB2Hf) If you installed from source, navigate to the following path and load the **maltego.mtz** configuration archive: **/home/kali/go/src/github.com/dreadl0ck/netcap/maltego** Otherwise, just download the latest version from github.com: ![Load netcap.mtz config](/files/-MaE9kl8aOnf55GKQrkk) ![Confirm import](/files/-MaE9xaANpoltu9TKdIG) ![Successful import](/files/-MaE9sK8OFKJr162f-FE) ### Configuring the file type matcher preference In order for Maltego to sucessfully detect the type of files you copy and paste into it, we need to disable using the generic **maltego.Phrase** matcher, because it will always match first. Select **Manage Entities**, search for the Phrase entity, edit it by clicking on the three little dots and deselect the shown checkbox in the **Advanced Settings**: ![Search Phrase entity](/files/-MaES-WdUUreQSyaKZlp) ![Disable use of regex converter for Phrase entity](/files/-MaESCxuWIRLdKuHlfRZ) Make sure the changes are saved by hitting **OK**. Now you should be able to copy and paste files into Maltego as described below. #### Importing PCAP or audit record files via Copy and Paste Drag and Drop of entities does not seem possible for Maltego on Kali yet (it works on macOS), so the most convenient way to load a PCAP file into Maltego is by selecting the file in the file explorer, hit Ctrl-C or CMD-C (VMWare on macOS) to copy the path to the clipboard, and then paste that path directly into a Maltego graph. The NETCAP configuration registered regular expressions to handle files that end on pcap or pcapng, so these should be detected as a **netcap.PCAP** entity automatically. The same should work as well for .ncap and .ncap.gz audit record files. If sucessful, the imported entity should look like this: ![PCAP imported into Maltego](/files/-MaERCCIzQ5G06StYavN) #### Importing PCAP files into Maltego manually Alternatively to importing files by copy and paste, you can add new entity of type netcap.PCAP to the graph manually, then double click it and set the mandatory field containing the path of the PCAP file on disk. ![Windows > Entity Palette > Search: PCAP > Drag and Drop entity into graph](/files/-MaEThS1mmTq-0zDHVbZ) Double click the file, switch to the Properties Tab and set a name and file path: ![Populate netcap.PCAP entity information manually](/files/-MaEU8_-ibwzZRuKX4dh) Either way, after importing the file executing a right click should show you the following NETCAP transforms: ![Transforms available on netcap.PCAP](/files/-MaEUWnZ1VJSDQhJjsD2) Running the **To Audit Records \[NETCAP]** transform will start NETCAP to process the pcap file! Afterwards, you should see audit records in Maltego: ![Audit records in Maltego](/files/-MaE_veFx7Li2iM3kHzR) Clone the exploitdb repository, so the transforms for opening them work: ``` $ cd /usr/local/etc/netcap $ git clone https://github.com/offensive-security/exploitdb.git ``` ### Installing IDA For the '**Open File in Disassembler**' transform to open extracted binary files in the IDA dissassembler to work, we need to install IDA, or move your existing installation to the expected place on the filesystem. To install, grab a download link from: At the time of this writing it is: ``` $ wget https://out7.hex-rays.com/files/idafree76_linux.run $ chmod +x idafree76_linux.run $ ./idafree76_linux.run [click Next in graphical installer] ``` Next, add **/usr/local/bin/ida** to your PATH in **\~/.zshrc**: ``` export PATH="/usr/local/bin/ida:$PATH" ``` Move the downloaded files there and source the **\~/.zshrc**: ``` $ sudo mv /usr/local/bin/idafree-7.6 /usr/local/bin/ida $ source ~/.zshrc # confirm that you can open ida $ ida64 [launches gui] ``` ### Utils Optional, but very useful to inspect data generated by netcap on the commandline are **tree** and **batcat**, you can install them with: ``` $ sudo snap install batcat $ sudo apt install tree ``` For example, you can inspect extracted TCP streams with ANSI colors using batcat. Everything that is colored in red has been transferred by the client, everything in blue is from the server, just like in wireshark. Enter a directory with netcap audit records where the capture tool was executed the **-conns** flag (enabled by default) and run: ``` $ batcat tcp/world-wide-web-http/* ``` ![Batcat: like cat, just with wings](/files/-MaEF5vY7ZrLe5ILT1_y) > Tip: use batcat -A to view binary connection data in the terminal Lets examine the tree command output to understand what files and directories are produced by NETCAP. Move into a directory that was created as output by the capture tool (**-out** flag, defaults to current directory) and run: ``` $ tree -h . ``` ![Tree of NETCAP output](/files/-MaEG6Y2u9ycpRxwd2MB) You can see there are two different file types on the top level: * .log Log files from different components of NETCAP, for diagnostic purposes. The main log file **netcap.log** always contains the log output that was written to the console when the NETCAP engine was executed. * .ncap.gz NETCAP audit records compressed with gzip. In order to read them, you need to use the **net dump** sub command, e.g: ![Reading NETCAP audit records](/files/-MaEHA1jfCgS5bA69xOY) The following directories can be generated, depending on the configuration: * files (or custom name, provided via **-fileStorage** flag, default name is files) The output directory of the files extracted from network connections, structured according to the detected content MIME type. * tcp (generated when **-conns** flag is set, enabled by default) Extracted **TCP** connection data, structured based on the service names obtained by looking up the used port numbers in the IANA database. Colorized with ANSI escape sequences, red is client, blue is server. * udp (generated when **-conns** flag is set, enabled by default) Extracted **UDP** connection data, structured based on the service names obtained by looking up the used port numbers in the IANA database. Colorized with ANSI escape sequences, red is client, blue is server. Also **codium** is a nice IDE and text editor to inspect source code: ``` $ sudo snap install codium ``` # Quickstart For those who can't wait to get their hands dirty. ## Capture traffic to create audit records Read traffic live from interface, stop with *Ctrl-C* (*SIGINT*): ``` $ net capture -iface eth0 ``` Read traffic from a dump file (supports PCAP or PCAPNG): ``` $ net capture -read traffic.pcap ``` ## Read audit records Read a netcap dumpfile and print to stdout as CSV: ``` $ net dump -read TCP.ncap.gz ``` Show the available fields for a specific Netcap dump file: ``` $ net dump -fields -read TCP.ncap.gz ``` Print only selected fields and output as CSV: ``` $ net dump -read TCP.ncap.gz -select Timestamp,SrcPort,DstPort ``` Save CSV output to file: ``` $ net dump -read TCP.ncap.gz -select Timestamp,SrcPort,DstPort > tcp.csv ``` Print output separated with tabs: ``` $ net dump -read TPC.ncap.gz -tsv ``` Run with 24 workers and disable gzip compression and buffering: ``` $ net capture -workers 24 -buf false -comp false -read traffic.pcapng ``` Parse pcap and write all data to output directory (will be created if it does not exist): ``` $ net capture -read traffic.pcap -out traffic_ncap ``` Convert timestamps to UTC: ``` $ net dump -read TCP.ncap.gz -select Timestamp,SrcPort,Dstport -utc ``` ## Show Audit Record File Header To display the header of the supplied audit record file, the -header flag can be used: ``` $ net capture -read TCP.ncap.gz -header +----------+---------------------------------------+ | Field | Value | +----------+---------------------------------------+ | Created | 2018-11-15 04:42:22.411785 +0000 UTC | | Source | Wednesday-WorkingHours.pcap | | Version | v0.3.3 | | Type | NC_TCP | +----------+---------------------------------------+ ``` ## Print Structured Audit Records Audit records can be printed structured, this makes use of the *proto.MarshalTextString()* function. This is sometimes useful for debugging, but very verbose. ``` $ net dump -read TCP.ncap.gz -struc ... NC_TCP Timestamp: "1499255023.848884" SrcPort: 80 DstPort: 49472 SeqNum: 1959843981 AckNum: 3666268230 DataOffset: 5 ACK: true Window: 1025 Checksum: 2348 PayloadEntropy: 7.836586993143013 PayloadSize: 1460 ... ``` ## Print as CSV This is the default behavior. First line contains all field names. ``` $ net dump -read TCP.ncap.gz Timestamp,SrcPort,DstPort,SeqNum,AckNum,DataOffset,FIN,SYN,RST,PSH,ACK,URG,... 1499254962.234259,443,49461,1185870107,2940396492,5,false,false,false,true,true,false,... 1499254962.282063,49461,443,2940396492,1185870976,5,false,false,false,false,true,false,... ... ``` ## Print as Tab Separated Values To use a tab as separator, the *-tsv* flag can be supplied: ``` $ net dump -read TCP.ncap.gz -tsv Timestamp SrcPort DstPort Length Checksum PayloadEntropy PayloadSize 1499254962.084372 49792 1900 145 34831 5.19616448 137 1499254962.084377 49792 1900 145 34831 5.19616448 137 1499254962.084378 49792 1900 145 34831 5.19616448 137 1499254962.084379 49792 1900 145 34831 5.19616448 137 ... ``` ## Print as Table The *-table* flag can be used to print output as a table. Every 100 entries the table is printed to stdout. ``` $ net dump -read UDP.ncap.gz -table -select Timestamp,SrcPort,DstPort,Length,Checksum +--------------------+----------+----------+---------+-----------+ | Timestamp | SrcPort | DstPort | Length | Checksum | +--------------------+----------+----------+---------+-----------+ | 1499255691.722212 | 62109 | 53 | 43 | 38025 | | 1499255691.722216 | 62109 | 53 | 43 | 38025 | | 1499255691.722363 | 53 | 62109 | 59 | 37492 | | 1499255691.722366 | 53 | 62109 | 59 | 37492 | | 1499255691.723146 | 56977 | 53 | 43 | 7337 | | 1499255691.723149 | 56977 | 53 | 43 | 7337 | | 1499255691.723283 | 53 | 56977 | 59 | 6804 | | 1499255691.723286 | 53 | 56977 | 59 | 6804 | | 1499255691.723531 | 63427 | 53 | 43 | 17441 | | 1499255691.723534 | 63427 | 53 | 43 | 17441 | | 1499255691.723682 | 53 | 63427 | 87 | 14671 | ... ``` ## Print with Custom Separator Output can also be generated with a custom separator: ``` $ net dump -read TCP.ncap.gz -sep ";" Timestamp;SrcPort;DstPort;Length;Checksum;PayloadEntropy;PayloadSize 1499254962.084372;49792;1900;145;34831;5.19616448;137 1499254962.084377;49792;1900;145;34831;5.19616448;137 1499254962.084378;49792;1900;145;34831;5.19616448;137 ... ``` ## Validate generated CSV output To ensure values in the generated CSV would not contain the separator string, the *-check* flag can be used. This will determine the expected number of separators for the audit record type, and print all lines to stdout that do not have the expected number of separator symbols. The separator symbol will be colored red with ansi escape sequences and each line is followed by the number of separators in red color. The *-sep* flag can be used to specify a custom separator. ``` $ net util -read TCP.ncap.gz -check $ net util -read TCP.ncap.gz -check -sep=";" ``` # Configuration Adjusting framework parameters ## Command-line Flags Each subcommand has a dedicated set of flags for configuration. List the flag names, a short description and their default values with: ``` $ net -h ``` ## Environment All default values for flags can be overriden via environment variables, by using the flag name and prefixing it with "NC\_", for example lets overwrite the **-read** flag from net capture: ``` $ NC_READ=/home/user/traffic.pcap net capture ``` Since the provide the value via the environment, passing it via flag is no longer necessary. This is generally useful to enable or disable features globally on your system. ## Configuration File Additionally, the configuration can be provided as a config file via the **-config** flag. To retrieve a sane default configuration for the subcommand you want to execute, use the **-gen-config** flag and redirect the output into a file: ``` $ net capture -gen-config > capture.conf ``` The config file will look something like this, using the **name value** syntax to set values: ```bash ... # toggle promiscous mode for live capture promisc true # don't print infos to stdout quiet false # reassemble TCP connections reassemble-connections true # resolve ips to domains via the operating systems default dns resolver reverse-dns false # use serviceDB for device profiling serviceDB false # configure snaplen for live capture from interface snaplen 1514 # print netcap package version and exit version false # wait for all connections to finish processing before cleanup wait-conns true # number of workers workers 12 # write incomplete response writeincomplete false ... ``` > Lines starting with # are treated as comments, blank lines are being ignored. Adjust the parameters of interest and pass the config file: ``` $ net capture -config capture.conf ``` ## Resolver Database The environment variable **NC\_DATABASE\_SOURCE** can be used to overwrite the default path for the resolver databases **/usr/local/etc/netcap/db**. Read more about the resolvers package here: {% content-ref url="/pages/-M4pSNs9Ewumxyv3nEjO" %} [Resolvers](/resolvers) {% endcontent-ref %} # Bash Completion Tab completion for the shell ## Installation Completions for the command-line is provided via the bash-completion package which is available for most linux distros and macOS. On macOS you can install it with brew: ``` $ brew install bash-completion ``` on linux use the package manager of your distro. Then add the completion file **cmd/net** to: * macOS: /usr/local/etc/bash\_completion.d/ * Linux: /etc/bash\_completion.d/ and source it with: * macOS: . /usr/local/etc/bash\_completion.d/net * Linux: . /etc/bash\_completion.d/net If you use zeus, simply execute the following in the project root to install the completion script: ``` $ zeus install-completions ``` or move and source the file manually from the project root: ``` $ cp cmd/net /usr/local/etc/bash_completion.d/net && . /usr/local/etc/bash_completion.d/net ``` Afterwards you should receive predictions when hitting tab in the shell, for subcommands and flags. For flags that expect a path on the filesystem, path completion is available and will only display files with the expected datatype (based on the file extension). To use completion with **zsh** run the following: ``` autoload -U +X compinit && compinit autoload -U +X bashcompinit && bashcompinit cp cmd/net /usr/local/etc/bash_completion.d/net && . /usr/local/etc/bash_completion.d/net ``` # Packet Collection This section focuses on gathering network packet information with netcap Packets are fetched from an input source (offline dump file or live from an interface) and distributed via round-robin to a pool of workers. Each worker dissects all layers of a packet and writes the generated *protobuf* audit records to the corresponding file. By default, the data is compressed with *gzip* to save storage space and buffered to avoid an overhead due to excessive *syscalls* for writing data to disk. ![Packet collection process](/files/-M4zJRRXSeaBZOlGLN8w) ## Encoders Encoders take care of converting decoded packet data into protocol buffers for the audit records. Two types of encoders exist: the [Layer Encoder](https://github.com/dreadl0ck/netcap/blob/master/encoder/layerEncoder.go), which operates on *gopacket* layer types, and the [Custom Encoder](https://github.com/dreadl0ck/netcap/blob/master/encoder/customEncoder.go), for which any desired logic can be implemented, including decoding application layer protocols that are not yet supported by gopacket or protocols that require stream reassembly. ## Unknown Protocols Protocols that cannot be decoded will be dumped in the unknown.pcap file for later analysis, as this contains potentially interesting traffic that is not represented in the generated output. Separating everything that could not be understood makes it easy to reveal hidden communication channels, which are based on custom protocols. ## Error Log Errors that happen in the gopacket lib due to malformed packets or implementation errors are written to disk in the errors.log file, and can be checked by the analyst later. Each packet that had a decoding error on at least one layer will be added to the errors.pcap. An entry to the error log has the following format: ``` Error: Packet: ``` At the end of the error log, a summary of all errors and the number of their occurrences will be appended. ``` ... : ... ``` ## Inclusion and Exclusion of Encoders The *-encoders* flag can be used to list all available encoders. In case not all of them are desired, selective inclusion and exclusion is possible, by using the *-include* and *-exclude* flags. List all encoders: ``` $ net capture -encoders custom: 11 + TLSClientHello + TLSServerHello + LinkFlow + NetworkFlow + TransportFlow + HTTP + Flow + Connection + DeviceProfile + File + POP3 layer: 55 + TCP + UDP + IPv4 + IPv6 + DHCPv4 + DHCPv6 + ICMPv4 + ICMPv6 + ICMPv6Echo + ICMPv6NeighborSolicitation + ICMPv6RouterSolicitation + DNS + ARP + Ethernet + Dot1Q + Dot11 + NTP + SIP + IGMP + LLC + IPv6HopByHop + SCTP + SNAP + LinkLayerDiscovery + ICMPv6NeighborAdvertisement + ICMPv6RouterAdvertisement + EthernetCTP + EthernetCTPReply + LinkLayerDiscoveryInfo + IPSecAH + IPSecESP + Geneve + IPv6Fragment + VXLAN + USB + LCM + MPLS + Modbus + OSPF + OSPF + BFD + GRE + FDDI + EAP + VRRP + EAPOL + EAPOLKey + CiscoDiscovery + CiscoDiscoveryInfo + USBRequestBlockSetup + NortelDiscovery + CIP + Ethernet/IP + SMTP + Diameter ... ``` Include specific encoders (only those named will be used): ``` $ net capture -read traffic.pcap -include Ethernet,Dot1Q,IPv4,IPv6,TCP,UDP,DNS ``` Exclude encoders (this will prevent decoding of layers encapsulated by the excluded ones): ``` $ net capture -read traffic.pcap -exclude TCP,UDP ``` ## Applying Berkeley Packet Filters *Netcap* will decode all traffic it is exposed to, therefore it might be desired to set a berkeley packet filter, to reduce the workload imposed on *Netcap*. This is possible for both live and offline operation. In case a [BPF](https://www.kernel.org/doc/Documentation/networking/filter.txt) should be set for offline use, the [gopacket/pcap](https://godoc.org/github.com/google/gopacket/pcap) package with bindings to the *libpcap* will be used, since setting BPF filters is not yet supported by the native [pcapgo](https://godoc.org/github.com/google/gopacket/pcapgo) package. When capturing live from an interface: ``` $ net capture -iface en0 -bpf "host 192.168.1.1" ``` When reading offline dump files: ``` $ net capture -read traffic.pcap -bpf "host 192.168.1.1" ``` # Audit Record Labeling Label audit records for supervised machine learning ## Introduction The term labeling refers to the procedure of adding classification information to each audit record. For the purpose of intrusion detection this is usually a label stating whether the record is normal or malicious. This is called binary classification, since there are just two choices for the label (good / bad). Another option is to use multi-class labels which could represent attack names or categories. Efficient and precise creation of labeled datasets is important for supervised machine learning techniques. To create labeled data, Netcap parses logs produced by suricata and extracts information for each alert. The quality of labels therefore depends on the quality of the used ruleset. In the next step it iterates over the data generated by itself and maps each alert to the corresponding packets, connections or flows. This takes into account all available information on the audit records and alerts. More details on the mapping logic can be found in the implementation chapter. While labeling is usually performed by marking all packets of a known malicious IP address, Netcap implements a more granular and thus more precise approach of mapping labels for each record. Labeling happens asynchronously for each audit record type in a separate goroutine. ![Labeling audit records with alerts from suricata](/files/-LqkPRsCaCdoXS5F4nTq) ## Labeling with Suricata For labeling with suricata please install suricata first and make sure it can be found in your **$PATH**. The installation guide can be found here: {% embed url="" %} The suricata config file is expected by default at **/usr/local/etc/suricata/suricata.yaml**, but you can overwrite this path with the **-suricata-config** flag. ```go // SuricataAlert is a summary structure for an alert type SuricataAlert struct { Timestamp string Proto string SrcIP string SrcPort int DstIP string DstPort int Classification string Description string } ``` The label tool expects being passed a packet capture dump with the **-read** flag, which will then be scanned with suricata to retrieve the alerts from the suricata **fast.log** file with regular expressions. > Future versions could use the eve.json log for this. Inside of the provided output directory (**-out** or current directory by default) the audit records generated for the provided PCAP file are expected to be present. That means you need to generate them first before using the label tool. When running the tool, labeled CSV files will be created for the alerts produced by suricata, adding the attack class or description (depending on the configuration), as a last element to every line. ## Labeling with custom attack information Custom attack information can be loaded as a CSV file. The data is expected to have the following fields: ```go type AttackInfo struct { Num int `csv:"num"` Name string `csv:"name"` Start time.Time `csv:"start"` End time.Time `csv:"end"` IPs []string `csv:"ips"` Proto string `csv:"proto"` Notes string `csv:"notes"` Category string `csv:"category"` } ``` The time format for the start and end markers is: ``` 2006/1/2 15:04:05 ``` Audit records will be labeled as a part of an attack if all of the following conditions are met: * at least one of the ips from the attackinfo is either source or destination of the audit record * the audit record has a timestamp within the attack period or matches it exactly This specification resulted from a specific dataset from a research project and can be easily updated if you have different requirements or different data. ## Usage Scan input pcap and create labeled csv files by mapping audit records in the current directory: ``` $ net label -read traffic.pcap ``` Scan input pcap and create output files by mapping audit records from the output directory: ``` $ net label -read traffic.pcap -out output_dir ``` Abort if there is more than one alert for the same timestamp: ``` $ net label -read taffic.pcap -strict ``` Display progress bar while processing input (experimental): ``` $ net label -read taffic.pcap -progress ``` Append classifications for duplicate labels: ``` $ net label -read taffic.pcap -collect ``` # HTTP Proxy Inspect traffic to web applications with a HTTP reverse proxy ## Motivation The **proxy** tool allows to quickly spin up monitoring of web applications and retrieving netcap audit records. Since currently, TCP stream reassembly is only supported for IPv4, netcap misses HTTP traffic over IPv6 when decoding traffic from raw packets. Also there is currently no support implemented for decoding HTTP2 over TCP or QUIC. By using a simple reverse proxy for HTTP traffic, the operating system handles the stream reassembly and we can make sure no IPv6 and / or HTTP2 traffic is missed. ## Usage Spin up a single proxy instance from the commandline: `$ net proxy -local 127.0.0.1:4000 -remote http://google.com` Specifiy a custom config file for proxying multiple services with the **-proxy-config** flag: ``` $ net proxy -proxy-config example_config.yml ``` The default config path is **net.proxy-config.yml**, so if this file exists in the folder where you execute the proxy, you do not need to specify it on the commandline. ## Configuration For proxying several services, you need to provide a config file, here is a simple example: ```yaml # Proxies map holds all reverse proxies proxies: service1: local: 127.0.0.1:443 remote: http://127.0.0.1:8080 tls: true service2: local: 127.0.0.1:9999 remote: http://192.168.1.20 service3: local: 127.0.0.1:7000 remote: https://google.com # CertFile for TLS secured connections certFile: "certs/cert.crt" # KeyFile for TLS secured connections keyFile: "certs/cert.key" # Logdir is used as destination for the logfile logdir: "logs" ``` ## Help ```erlang Usage of net proxy: -version bool print netcap package version and exit -config string set config file path (default "net.proxy-config.yml") -debug set debug mode -dialTimeout int seconds until dialing to the backend times out (default 30) -idleConnTimeout int seconds until a connection times out (default 90) -local string set local endpoint -maxIdle int maximum number of idle connections (default 120) -remote string set remote endpoint -skipTlsVerify skip TLS verification -tlsTimeout int seconds until a TLS handshake times out (default 15) ``` # USB Capture Capture traffic sent via Universal Serial Bus (USB) protocol ## Live Capture USB live capture is now possible, currently the following Audit Records exist: USB and USBRequestBlockSetup. To capture USB traffic live on macOS, install wireshark and bring up the USB interface: ``` $ sudo ifconfig XHC20 up ``` Now attach netcap and set baselayer to USB: ``` $ net.cap -iface XHC20 -base usb ``` ## Offline from dumpfile To read offline USB traffic from a PCAP file use: ``` $ net.cap -r usb.pcap -base usb ``` Don't forget to set the **-payload** flag if you want to preserve the data being transmitted! ## Audit Records The **USB** and **USBRequestBlockSetup** audit records contain the following fields: ```erlang message USB { string Timestamp = 1; uint64 ID = 2; int32 EventType = 3; int32 TransferType = 4; int32 Direction = 5; int32 EndpointNumber = 6; int32 DeviceAddress = 7; int32 BusID = 8; int64 TimestampSec = 9; int32 TimestampUsec = 10; bool Setup = 11; bool Data = 12; int32 Status = 13; uint32 UrbLength = 14; uint32 UrbDataLength = 15; uint32 UrbInterval = 16; uint32 UrbStartFrame = 17; uint32 UrbCopyOfTransferFlags = 18; uint32 IsoNumDesc = 19; bytes Payload = 20; } message USBRequestBlockSetup { string Timestamp = 1; int32 RequestType = 2; int32 Request = 3; int32 Value = 4; int32 Index = 5; int32 Length = 6; } ``` � # Payload Capture Capture full packet payloads It is now possible to capture payload data for the following protocols: **TCP, UDP, ModbusTCP, USB** This can be enabled with the **-payload** flag: ``` $ net capture -read traffic.pcap -payload ``` Setting the flag works for both live and offlline capture, afterwards the raw payload bytes are stored in the **Payload** field of the audit records. You can use the **-struc** flag with the **dump** tool to see the payload in the command-line: ``` $ net dump -read TCP.ncap.gz -struc ``` # Distributed Collection Sensors and Collection Server ## Collection Server Using Netcap as a data collection mechanism, sensor agents can be deployed to export the traffic they see to a central collection server. This is especially interesting for internet of things (IoT) applications, since these devices are placed inside isolated networks and thus the operator does not have any information about the traffic the device sees. Although Go was not specifically designed for this application, it is an interesting language for embedded systems. Each binary contains the complete runtime, which increases the binary size but requires no installation of dependencies on the device itself. Data exporting currently takes place in batches over UDP sockets. Transferred data is compressed in transit and encrypted with the public key of the collection server. Asymmetric encryption was chosen, to avoid empowering an attacker who compromised a sensor, to decrypt traffic of all sensors communicating with the collection server. To increase the performance, in the future this could be replaced with using a symmetric cipher, together with a solid concept for key rotation and distribution. Sensor agents do not write any data to disk and instead keep it in memory before exporting it. ![](/files/-M4zJvVWwpzXy0DInGoO) As described in the concept chapter, sensors and the collection server use UDP datagrams for communication. Network communication was implemented using the go standard library. This section will focus on the procedure of encrypting the communication between sensor and collector. For encryption and decryption, cryptographic primitives from the [golang.org/x/crypto/nacl/box](https://godoc.org/golang.org/x/crypto/nacl/box) package are used. The NaCl (pronounced 'Salt') toolkit was developed by the reowned cryptographer Daniel J. Bernstein. The box package uses *Curve25519*, *XSalsa20* and *Poly1305* to encrypt and authenticate messages. It is important to note that the length of messages is not hidden. Netcap uses a thin wrapper around the functionality provided by the nacl package, the wrapper has been published here: [github.com/dreadl0ck/cryptoutils](https://www.github.com/dreadl0ck/cryptoutils). ## Batch Encryption The collection server generates a keypair, consisting of two 32 byte (256bit) keys, hex encodes them and writes the keys to disk. The created files are named *pub.key* and *priv.key*. Now, the servers public key can be shared with sensors. Each sensor also needs to generate a keypair, in order to encrypt messages to the collection server with their private key and the public key of the server. To allow the server to decrypt and authenticate the message, the sensor prepends its own public key to each message. ![NETCAP batch encryption](/files/-LqkPS5QAwJ9Oe_AWWAC) ## Batch Decryption When receiving an encrypted batch from a sensor, the server needs to trim off the first 32 bytes, to get the public key of the sensor. Now the message can be decrypted, and decompressed. The resulting bytes are serialized data for a batch protocol buffer. After unmarshalling them into the batch structure, the server can append the serialized audit records carried by the batch, into the corresponding audit record file for the provided client identifier. ![](/files/-M4zJzA2Mfa_lRQXuU1q) ## Usage Both sensor and client can be configured by using the *-addr* flag to specify an IP address and port. To generate a keypair for the server, the *-gen-keypair* flag must be used: ``` $ net collect -gen-keypair wrote keys $ ls priv.key pub.key ``` Now, the server can be started, the location of the file containing the private key must be supplied: ```bash $ net collect -privkey priv.key -addr 127.0.0.1:4200 ``` The server will now be listening for incoming messages. Next, the sensor must be configured. The keypair for the sensor will be generated on startup, but the public key of the server must be provided: ``` $ net agent -pubkey pub.key -addr 127.0.0.1:4200 got 126 bytes of type NC_ICMPv6RouterAdvertisement expected [126] got size [73] for type NC_Ethernet got 73 bytes of type NC_Ethernet expected [73] got size [27] for type NC_ICMPv6 got size [126] for type NC_ICMPv6RouterAdvertisement got 126 bytes of type NC_ICMPv6RouterAdvertisement expected [126] got size [75] for type NC_IPv6 got 75 bytes of type NC_IPv6 expected [75] got 27 bytes of type NC_ICMPv6 expected [27] ``` The client will now collect the traffic live from the specified interface, and send it to the configured server, once a batch for an audit record type is complete. The server will log all received messages: ``` $ net collect -privkey priv.key -addr 127.0.0.1:4200 packet-received: bytes=2412 from=127.0.0.1:57368 decoded batch NC_Ethernet from client xyz new file xyz/Ethernet.ncap packet-received: bytes=2701 from=127.0.0.1:65050 decoded batch NC_IPv4 from client xyz new file xyz/IPv4.ncap ... ``` When stopping the server with a *SIGINT* (Ctrl-C), all audit record file handles will be flushed and closed properly. The agent uses the **$USER** environment variable to identify the workstation where the audit records are created. This will be replaced with a unique identifier in a future release. # Workers This is where the magic happens ## Introduction To make use of multi-core processors, processing of packets should happen in an asynchronous way. Since Netcap should be usable on a stream of packets, fetching of packets has to happen sequentially, but decoding them can be parallelized. The packets read from the input data source (PCAP file or network interface) are assigned to a configurable number of workers routines via round-robin. Each of those worker routines operates independently, and has all selected encoders loaded. It decodes all desired layers of the packet, and writes the encoded data into a buffer that will be flushed to disk after reaching its capacity. ## Worker [Workers](https://github.com/dreadl0ck/netcap/blob/master/collector/worker.go) are a core concept of *Netcap*, as they handle the actual task of decoding each packet. *Netcap* can be configured to run with the desired amount of workers, the default is 1000, since this configuration has shown the best results on the development machine. Increasing the number of workers also increases the number of runtime operations for goroutine scheduling, thus performance might decrease with a huge amount of workers. It is recommended to experiment with different configurations on the target system, and choose the one that performs best. Packet data fetched from the input source is distributed to a worker pool for decoding in round robin style. Each worker decodes all layers of a packet and calls all available custom encoders. After decoding of each layer, the generated protocol buffer instance is written into the *Netcap* data pipe. Packets that produced an error in the decoding phase or carry an unknown protocol are being written in the corresponding logs and dumpfiles. > Note: by default the number of workers is set to the numbers of cores of your machine! You can use the **-workers** flag to overwrite this value. ![NETCAP worker](/files/-LqkPRrbQWfbxaDAhIFd) ## Buffering Each worker receives its data from an input channel. This channel can be buffered, by default the buffer size is 100, also because this configuration has shown the best results on the development machine. When the buffer size is set to zero, the operation of writing a packet into the channel blocks, until the goroutine behind it is ready for consumption. That means, the goroutine must finish the currently processed packet, until a new packet can be accepted. By configuring the buffer size for all routines to a specific number of packets, distributing packets among workers can continue even if a worker is not finished yet when new data arrives. New packets will be queued in the channel buffer, and writing in the channels will only block if the buffer is full. ![NETCAP buffered workers](/files/-LqkPRrd4dG9z5cIPNiA) ## Data Pipe The Netcap data pipe describes the way from a network packet that has been processed in a worker routine, to a serialized, delimited and compressed record into a file on disk. ![](/files/-LqkPRrgBVB27j8Y3Ppw) # Filtering and Export Process Netcap audit records and extract the data you are interested in ## Exporting Data with net dump Netcap offers a simple interface to filter for specific fields and select only those of interest. Filtering and exporting specific fields can be performed with all available audit record types, over a uniform command-line interface. By default, output is generated as CSV with the field names added as first line. It is also possible to use a custom separator string. Fields are exported in the order they are named in the select statement. Sub structures of audit records (for example IPv4Options from an IPv4 packet), are converted to a human readable string representation. More examples for using this feature on the command-line can be found in the usage section. ![](/files/-M4zf-fqn5vKXXSv1SAY) Netcap offers a simple command-line interface to select fields of interest from the gathered audit records. ## Examples Show available header fields: ``` $ net dump -read UDP.ncap.gz -fields Timestamp,SrcPort,DstPort,Length,Checksum,PayloadEntropy,PayloadSize ``` Print all fields for the supplied audit record: ``` $ net dump -read UDP.ncap.gz 1331904607.100000,53,42665,120,41265,4.863994469989251,112 1331904607.100000,42665,53,53,1764,4.0625550894074385,45 1331904607.290000,51190,53,39,22601,3.1861758166070766,31 1331904607.290000,56434,53,39,37381,3.290856864924384,31 1331904607.330000,137,137,58,64220,3.0267194361875682,50 ... ``` Selecting fields will also define their order: ``` $ net dump -read UDP.ncap.gz -select Length,SrcPort,DstPort,Timestamp Length,SrcPort,DstPort,Timestamp 145,49792,1900,1499254962.084372 145,49792,1900,1499254962.084377 145,49792,1900,1499254962.084378 145,49792,1900,1499254962.084379 145,49792,1900,1499254962.084380 ... ``` Print selection in the supplied order and convert timestamps to UTC time: ``` $ net dump -read UDP.ncap.gz -select Timestamp,SrcPort,DstPort,Length -utc 2012-03-16 13:30:07.1 +0000 UTC,53,42665,120 2012-03-16 13:30:07.1 +0000 UTC,42665,53,53 2012-03-16 13:30:07.29 +0000 UTC,51190,53,39 2012-03-16 13:30:07.29 +0000 UTC,56434,53,39 2012-03-16 13:30:07.33 +0000 UTC,137,137,58 ... ``` To save the output into a new file, simply redirect the standard output: ``` $ net dump -read UDP.ncap.gz -select Timestamp,SrcPort,DstPort,Length -utc > UDP.csv ``` # Data Compression Save storage space To reduce the amount of disk space used for storing the audit records, netcap compresses them by default with **gzip**. Compressed files have the extension **.ncap.gz.** For this purpose Netcap currently uses the following gzip implementation: {% embed url="" %} This implementation will split the data into blocks that are compressed in parallel, which can be useful for compressing big amounts of data. The output is a standard gzip file. The gzip decompression is modified so it decompresses ahead of the current reader. This means that reads will be non-blocking and CRC calculation also takes place in a separate goroutine. This design implements input buffering to the compressor which has a nice performance effect: writes to the compressor only block if the compressor is already compressing the number of blocks specified. This reduces waiting time for the workers which they can instead use to decode packets. To get any performance gains, you should at least be compressing more than 1 megabyte of data at the time. You should at least have a block size of 100k and at least a number of blocks that match the number of cores your would like to utilize, but about twice the number of blocks would be the best. The default configuration uses 1MB block size and 2x NumCPUs as the number of blocks. Netcap only uses the parallel gzip implementation for reading and writing audit records, as only there the required amounts of data are reached to allow a speedup. For tasks where the data size can vary heavily, such as decompressing HTTP requests and responses, the standard library **compress/gzip** is used instead. # Internals Framework inner workings and Implementation details ## Packages You can browse the source and sub packages on GoDev: ### cmd The cmd package contains the command-line application. It receives configuration parameters from command-line flags, creates and configures a collector instance, and then starts collecting data from the desired source. #### label [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/l](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector)abel The label package contains the code for creating labeled datasets. For now, the suricata IDS / IPS engine is used to scan the input PCAP and generate alerts. In the future, support could also be added for using YARA. Alerts are then parsed with regular expressions and transformed into the **label.SuricataAlert** type. This could also be replaced by parsing suricatas eve.json event logs in upcoming versions. A suricata alert contains the following information: ```go // SuricataAlert is a summary structure of an alerts contents type SuricataAlert struct { Timestamp string Proto string SrcIP string SrcPort int DstIP string DstPort int Classification string Description string } ``` In the next iteration, the gathered alerts are mapped onto the collected data. For layer types which are not handled separately, this is currently by using solely the timestamp of the packet, since this is the only field required by Netcap, however multiple alerts might exist for the same timestamp. To detect this and throw an error, the **-strict** flag can be used. The default is to ignore duplicate alerts for the same timestamp, use the first encountered label and ignore the rest. Another option is to collect all labels that match the timestamp, and append them to the final label with the **-collect** flag. To allow filtering out classifications that shall be excluded, the **-excluded** flag can be used. Alerts matching the excluded classi- fication will then be ignored when collecting the generated alerts. Flow, Connection, HTTP and TLS records mapping logic also takes source and destination information into consider- ation. The created output files follow the naming convention: **\\_labeled.csv**. ### types [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/t](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector)ypes The types package contains types.AuditRecord interface implementations for each supported protocol, to enable converting data to the CSV format. For this purpose, each protocol must provide a CSVRecord() \[]string and a CSVHeader() \[]string function. Additionally, a NetcapTimestamp() string function that returns the Netcap timestamp must be implemented. ### encoder [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/e](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector)ncoder The encoder package implements conversion of decoded network protocols to protocol buffers. This has to be defined for each supported protocol. Two types of encoders exist: The LayerEncoder and the CustomEncoder. #### Layer Encoder A LayerEncoder operates on a gopacket.Layer and has to provide the gopacket.LayerType constant, as well a handler function to receive the layer and the timestamp and convert it into a protocol buffer. #### Custom Encoder A CustomEncoder operates on a gopacket.Packet and is used to decode traffic into abstrac- tions such as Flows or Connections. To create it a name has to be supplied among three different handler functions to control initialization, decoding and deinitialization. Its handler function receives a gopacket.Packet interface type and returns a proto.Message. The postinit function is called after the initial initialization has taken place, the deinit function is used to teardown any additionally created structures for a clean exit. Both functions are optional and can be omitted by supplying nil as value. ### resolvers [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/r](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector)esolvers Resolvers for lookup of various external information, such as geolocation, domain names, hardware addresses, port numbers etc ### dpi [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/d](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector)pi Deep Packet Inspection integration, using a fork of **mushorg/go-dpi** that was extended to identify the full range of protocols offered by **nDPI** and **libprotoident**. Both libraries are loaded dynamically at runtime and is invoked via C bindings. The fork can be found here: {% embed url="" %} Deep Packet Inspection Package {% endembed %} ### delimited [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/d](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector)elimited Primitives for reading and writing length delimited binary data ### utils [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/u](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector)tils The utils package contains shared utility functions used by several other packages. ### collector [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/collector](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector) The collector package provides an interface for fetching packets from a data source, this can either be a PCAP / PCAPNG file or directly from a named network interface. It is used to implement the command-line interface for Netcap. {% hint style="info" %} Warning: Do not use multiple instances of a collector in parallel! This is not supported yet. Once it is possible, this warning will be removed. {% endhint %} ### io [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/io](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector) Primitives for atomic maps and write operations ## Caveats Protocol buffers have a few caveats that developers and researchers should be aware of. First, there are no types for 16 bit signed (int16) and unsigned (uint16) integers in protobuf, also there is no type for unsigned 8 bit integers (uint8). This data type is seen a lot in network protocols, so the question arises how to represent it in protocol buffers. The non-fixed integer types use variable length encoding, so int32 is used instead. The variable-length encoding will take care of not sending the bytes that are not being used. Unfortunately, the mu type is too short for this purpose. Second, protocol buffers require all strings to be encoded as valid UTF-8, otherwise encoding to proto will fail. This means all input data that will be encoded as a string in protobuf must be checked to contain valid UTF-8, or they will create an error upon serialization and end up in the errors.pcap file. If this behavior is not desired strings must be filtered prior to setting them on the protocol buffer instances. Another thing that has to be kept in mind is that Netcap processes packets in parallel, thus the order in which packets are written to the dump file is not guaranteed. In experiments, no mixup was detected, and records were tracked in the correct order. However, under heavy load conditions or with a high number of workers, this might be different. Because of this caveat, the Netcap specification requires each record to preserve the timestamp, in order to allow sorting the packets afterwards, if required. ## Data Race Detection Builds In concurrent programming, shared resources need to be synchronized, in order to guarantee their state when modifying or reading them. If access is not synchronized, race conditions occur, which will lead to faulty program behavior. To avoid this and detect race conditions early in the development cycle, the go toolchain offers compiling the program with the race detector enabled. This will let the application crash with stack traces to assist the developer in debugging, if a data race occurs. Programs with active race detection are slower by the factor of 10 to 100. To compile a Go program with the race detection enabled the **-race** flag must be added to the compilation command. To compile a netcap binary with the race detection enabled use: ``` $ zeus install-race ``` # Metrics Prometheus Metrics ## Introduction Netcap now supports exporting prometheus metrics about its go runtime, the collection process and the audit records itself. These data points can be used to gain insights about the collection performance or discover security related events. [Prometheus](https://github.com/prometheus) is an open-source systems monitoring and alerting toolkit originally built at [SoundCloud](https://soundcloud.com/). Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user [community](https://prometheus.io/community). It is now a standalone open source project and maintained independently of any company. {% embed url="" %} Prometheus Homepage {% endembed %} To visualize the captured data I recomment the open source analytics and monitoring solution Grafana: {% embed url="" %} Grafana Homepage {% endembed %} This feature can be used with the **export** tool, which behaves similar to **capture** but is able to operate on pcaps, audit records and network interfaces. ## Configuration Metrics are served by default on [**127.0.0.1:7777/metrics**](http://127.0.0.1:7777/metrics). Configure a prometheus instance to scrape it: ```yaml # reference: https://prometheus.io/docs/prometheus/latest/configuration/configuration/ global: scrape_interval: 15s scrape_timeout: 15s #evaluation_interval: 15s scrape_configs: # process_ metrics - job_name: netcap metrics_path: /metrics scheme: http static_configs: - targets: - 127.0.0.1:7777 ``` {% hint style="info" %} Tip: The latest prometheus config documentation can be found at: [https://prometheus.io/docs/prometheus/latest/configuration/configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) {% endhint %} Run the export tool to capture live from an interface: ``` net export -iface en0 ``` {% hint style="info" %} Tip: Use `$ net capture -interfaces` to get a list of available interfaces to choose from {% endhint %} Go to or to the port you configured alternatively, to check if your prometheus instance is scraping data correctly. Now that we have some data at hands, lets use Grafana to visualize it! You can setup Grafana on macOS via brew: ``` $ brew install grafana ``` {% hint style="info" %} Tip: On macOS, Grafanas default config is at **/usr/local/etc/grafana/grafana.ini** and installed plugins are stored at **/usr/local/opt/grafana/share/grafana/data/plugins**. {% endhint %} Start the prometheus server and pass the previously created config: ``` $ prometheus --config.file prometheus/prometheus.yml ``` You need to install the pie chart plugin for grafana: ``` $ cd /usr/local/opt/grafana/share/grafana/data/plugins $ git clone https://github.com/grafana/piechart-panel.git --branch release-1.4.0 ``` Start the grafana server: ``` $ grafana-server --homepath /usr/local/opt/grafana/share/grafana ``` Now download the NETCAP Dashboard and import it into Grafana: {% file src="/files/-M5bLwIZdA6CfbX1Q1mP" %} Go to **Settings > Datasources** and a prometheus datasource, either with the default port 9090 or the one you choose in the config. You should be good to go! ## Usage Export a PCAP dumpfile and serve metrics . ``` $ net export -read 2017-09-19-traffic-analysis-exercise.pcap ``` Capture and export traffic live from the named interface: ``` $ net export -iface en0 ``` Export a specific audit record file: ``` $ net export -read HTTP.ncap.gz ``` Export all audit record files in the current directory: ``` $ net export . ``` ## Overview Dashboard Preview ![Grafana Dashboard Overview](/files/-LqkPRu3NbjO2GXz3Jei) ## TCP Dashboard Preview ![Grafana Dashboard TCP](/files/-LqkPRu5fN9Ey985AMOD) ## HTTP Dashboard Preview ![Grafana Dashboard HTTP](/files/-LqkPRu7v-GhYWl_OOHq) # Resolvers Lookup everything! ## Motivation Lots of information is not available on first sight, and we need to combine our data with knowledge from other data sources to make it easier to understand for humans. Think of resolving ip addresses to geolocations, hardware addreses to manufacturers, domains to ip addresses and vice versa, or simply identifying the service name associated with a given port number. Or consider filtering ip addresses or domain names against a whitelist, to eliminate known legitimate traffic. The resolvers package provides primitives for such tasks, and if possible, caches results in memory for better performance. ## Design External data sources are stored in a central directory on the system, which defaults to **/usr/local/etc/netcap/db** but can be overridden using the **NC\_DATABASE\_SOURCE** environment variable. Database files: * *domain-whitelist.csv* * *GeoLite2-City.mmdb* * *GeoLite2-ASN.mmdb* * *ja3fingerprint.json* * *macaddress.io-db.json* * *service-names-port-numbers.csv* * *ja3UserAgents.json* * *ja3erDB.json* ## Configuration By default, all resolvers are disabled. You need to use the **-reverse-dns**, **-local-dns**, **-macDB**, **-ja3DB**, **-serviceDB** and **-geoDB** to enable what you want to use, or configure it via environment variables or config file, as described in: {% content-ref url="/pages/-M4pT437hgZNXapaXZPa" %} [Configuration](/configuration) {% endcontent-ref %} ## Quickstart You can download a bundled version of all databases except for the MaxMind GeoLite, here: {% file src="/files/-M6158bOQF\_mI5dN6UjM" %} ## DNS Reverse DNS lookups can be used to identify the domains associated with an address. By default the standard system resolver will be contacted for this. ### Passive / Local DNS Passive DNS will read the hosts mapping from a file and load it into memory, instead of looking up encountered adresses by contacting a resolver. This can be used to provide names for known hosts in your network for example. To avoid producing lookups that leave the network, you can generate a hosts mapping based on the DNS traffic in your dumpfile using tshark: ``` $ tshark -r traffic.pcap -q -z hosts ``` And provide it to netcaps resolver via a **hosts** file in the database directory. ## Domain Whitelisting To filter known legitimate domains away, the alexa top 1 million can be used for example. {% embed url="" %} You can download the CSV file here: {% embed url="" %} Rename it to **domain-whitelist.csv** and move it into the database path: ``` $ mv top-1m.csv /usr/local/etc/netcap/db/domain-whitelist.csv ``` ## Geolocation To determine the geolocation for a given host, the MaxMind GeoLite database is used. The lite database is freely available, but you have to register on their website to download it. {% embed url="" %} GeoLite2 MaxMind {% endembed %} Geolocation lookups can provide the Country, City and ASN where an ip adress is registered. Download the databases and move them into the database path. ## Vendor Identification To identify the vendor for a given MAC address, the **macaddress.io** JSON database is used. At the time of this writing it contains 39,041 tracked address blocks and 28,961 unique vendors. {% embed url="" %} MacAddress.io database {% endembed %} ## Service Identification Resolving port numbers to service names is done according to the CSV mapping from IANA, which contains 6104 records for TCP and UDP services at the time of this writing: {% embed url="" %} IANA service names and ports {% endembed %} ## TLS Fingerprints To identify hosts that use TLS connections, the Ja3 fingerprint database from **Trisul** is used: {% embed url="" %} For more fingerprints, you can load other databases additionally. For example from **ja3erDB**: {% embed url="" %} Ja3er JSON database downloads {% endembed %} # TLS Fingerprinting Identify client and server that are using encrypted connections ## TLS Audit Records Watch a quick demo of creating and exploring the **TLSClientHello** audit records on the command-line {% embed url="" %} ## JA3 JA3 is a technique developed by Salesforce, to fingerprint the TLS client and server hellos. The official python implementation can be found [here](https://github.com/salesforce/ja3). More details can be found in their blog post: {% embed url="" %} JA3 blog post from salesforce {% endembed %} Support for JA3 and JA3S in netcap is implemented via: {% embed url="" %} JA3(S) go package {% endembed %} The *TLSClientHello* and *TLSServerHello* audit records, as well as the *DeviceProfiles* provide JA3 hashes. ## JA3 Details JA3 gathers the decimal values of the bytes for the following fields: **SSL Version, Accepted Ciphers, List of Extensions, Elliptic Curves, and Elliptic Curve Formats**. It then concatenates those values together in order, using a “,” to delimit each field and a “-” to delimit each value in each field. **The field order is as follows:** ``` SSLVersion,Ciphers,Extensions,EllipticCurves,EllipticCurvePointFormats ``` **Example:** ``` 769,47–53–5–10–49161–49162–49171–49172–50–56–19–4,0–10–11,23–24–25,0 ``` If there are no SSL Extensions in the Client Hello, the fields are left empty. **Example:** ``` 769,4–5–10–9–100–98–3–6–19–18–99,,, ``` These strings are then MD5 hashed to produce an easily consumable and shareable 32 character fingerprint. This is the JA3 SSL Client Fingerprint. JA3 is a much more effective way to detect malicious activity over SSL than IP or domain based IOCs. Since JA3 detects the client application, it doesn’t matter if malware uses DGA (Domain Generation Algorithms), or different IPs for each C2 host, or even if the malware uses Twitter for C2, JA3 can detect the malware itself based on how it communicates rather than what it communicates to. JA3 is also an excellent detection mechanism in locked-down environments where only a few specific applications are allowed to be installed. In these types of environments one could build a whitelist of expected applications and then alert on any other JA3 hits. For more details on what you can see and do with JA3 and JA3S, please see this Shmoocon 2018 talk: ### Client Hello Audit Record ```erlang message TLSClientHello { string Timestamp = 1; int32 Type = 2; int32 Version = 3; int32 MessageLen = 4; int32 HandshakeType = 5; uint32 HandshakeLen = 6; int32 HandshakeVersion = 7; bytes Random = 8; uint32 SessionIDLen = 9; bytes SessionID = 10; int32 CipherSuiteLen = 11; int32 ExtensionLen = 12; string SNI = 13; bool OSCP = 14; repeated int32 CipherSuites = 15; repeated int32 CompressMethods = 16; repeated int32 SignatureAlgs = 17; repeated int32 SupportedGroups = 18; repeated int32 SupportedPoints = 19; repeated string ALPNs = 20; string Ja3 = 21; string SrcIP = 22; string DstIP = 23; string SrcMAC = 24; string DstMAC = 25; int32 SrcPort = 26; int32 DstPort = 27; repeated int32 Extensions = 28; } ``` ## JA3S Details JA3S is JA3 for the Server side of the SSL/TLS communication and fingerprints how servers respond to particular clients. JA3S uses the following field order: ``` SSLVersion,Cipher,SSLExtension ``` With JA3S it is possible to fingerprint the entire cryptographic negotiation between client and it's server by combining JA3 + JA3S. That is because servers will respond to different clients differently but will always respond to the same client the same. For the Trickbot example: ``` JA3 = 6734f37431670b3ab4292b8f60f29984 ( Fingerprint of Trickbot ) JA3S = 623de93db17d313345d7ea481e7443cf ( Fingerprint of Command and Control Server Response ) ``` For the Emotet example: ``` JA3 = 4d7a28d6f2263ed61de88ca66eb011e3 ( Fingerprint of Emotet ) JA3S = 80b3a14bccc8598a1f3bbe83e71f735f ( Fingerprint of Command and Control Server Response ) ``` In these malware examples, the command and control server always responds to the malware client in exactly the same way, it does not deviate. So even though the traffic is encrypted and one may not know the command and control server's IPs or domains as they are constantly changing, we can still identify, with reasonable confidence, the malicious communication by fingerprinting the TLS negotiation between client and server. Again, please be aware that these are examples, not indicative of all versions ever, and are intended to illustrate what is possible. ### Server Hello Audit Record ```erlang message TLSServerHello { string Timestamp = 1; int32 Version = 2; bytes Random = 3; bytes SessionID = 4; int32 CipherSuite = 5; int32 CompressionMethod = 6; bool NextProtoNeg = 7; repeated string NextProtos = 8; bool OCSPStapling = 9; bool TicketSupported = 10; bool SecureRenegotiationSupported = 11; bytes SecureRenegotiation = 12; string AlpnProtocol = 13; bool Ems = 14; repeated bytes Scts = 15; int32 SupportedVersion = 16; bool SelectedIdentityPresent = 18; int32 SelectedIdentity = 19; bytes Cookie = 20; int32 SelectedGroup = 21; repeated int32 Extensions = 22; string SrcIP = 23; string DstIP = 24; string SrcMAC = 25; string DstMAC = 26; int32 SrcPort = 27; int32 DstPort = 28; string Ja3s = 29; } ``` # Reassembly TCP stream reassembly ## Implementation For reassembling TCP streams the gopacket/reassembly implementation is used. This allows to parse application layer protocols such as HTTP and POP3. The reassembly package currrently only implements reassembling stream over IPv4. To overcome this limitation for HTTP capture, you can use the **proxy** tool. {% content-ref url="/pages/-Ld3UN5WUZqlTlm2MG5s" %} [HTTP Proxy](/http-proxy) {% endcontent-ref %} ## Architecture The gopacket reassembly implementation leaves several options for using it. Netcap currently uses one dedicated assembler for each worker and a single shared connection pool for all streams. Another option would be using a dedicated assembler for each worker for each L7 protocol with a shared stream pool for that specific protocol. This would potentially decrease lock contention for the reassembly, and might be implemented to improve performance in future versions. ## Configuration The following fields of the **encoder.Config** affect the TCP stream reassembly: ```go // Interval to apply connection flushes FlushEvery int // Do not use IPv4 defragger NoDefrag bool // Dont verify the packet checksums Checksum bool // Dont check TCP options NoOptCheck bool // Ignore TCP state machine errors IgnoreFSMerr bool // TCP state machine allow missing init in three way handshake AllowMissingInit bool // Toggle debug mode Debug bool // Dump packet contents as hex for debugging HexDump bool // Wait until all connections finished processing when receiving shutdown signal WaitForConnections bool // Write incomplete HTTP responses to disk when extracting files WriteIncomplete bool ``` ## Debugging To see debug output for the reassembly, run with the **-debug** flag and check the **reassembly.log** file. For more general troubleshooting advice, please refer to the Troubleshooting page: {% content-ref url="/pages/-M4pSNs8UVjcQS6cDVc-" %} [Troubleshooting](/troubleshooting) {% endcontent-ref %} # Deep Packet Inspection Identify applications and categories ## Libprotoident NETCAP has support for using **libprotoident** (v[2.0.14](https://github.com/wanduow/libprotoident/releases/tag/2.0.14-1)), to identify 45 application categories and 500+ applications and protocols! The full list of supported protocols can be found here: {% embed url="" %} Libprotoident Supported Protocols {% endembed %} **libprotoident** is maintained by the WAND group, you can download and install the library here: {% embed url="" %} Libprotoident Source Code {% endembed %} ## nDPI Furthermore **nDPI** (v3.0) can be used to identify 244 applications, they are listed here: {% embed url="" %} nDPI Supported Protocols {% endembed %} **nDPI** is mainted by **ntop**, and can be downloaded here: {% embed url="" %} nDPI Source Code {% endembed %} The results from all heuristic engines (lPI, nDPI and go heuristics) get dedpulicated automatically. Future versions could create a certainity score based on the number of votes from different heuristics. DPI is currently used to indicate which applications have been seen for which **IPProfile**, when using the **DeviceProfile** encoder. Read more about DeviceProfiles here: {% content-ref url="/pages/-M4zgij9Nh\_pcelAun7g" %} [Device Profiles](/device-profiles) {% endcontent-ref %} ## Platform Support NETCAPs DPI integration is currently only available on linux and macOS. # Live Capture Capture from a network interface To capture packets live, simple use the **-iface** flag: ``` $ net capture -iface en0 ``` Use the **-interfaces** flag to list all available intefaces and their MTUs: ``` $ net capture -interfaces ┌───────┬─────────┬───────────────────────────┬───────────────────┬───────┐ │ Index │ Name │ Flags │ HardwareAddr │ MTU │ ├───────┼─────────┼───────────────────────────┼───────────────────┼───────┤ │ 1 │ lo0 │ up|loopback|multicast │ │ 16384 │ │ 2 │ gif0 │ pointtopoint|multicast │ │ 1280 │ │ 3 │ stf0 │ 0 │ │ 1280 │ │ 4 │ en5 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 5 │ ap1 │ broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 6 │ en0 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 7 │ en4 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 8 │ en1 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 9 │ en2 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 10 │ en3 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 11 │ bridge0 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 12 │ p2p0 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 2304 │ │ 13 │ awdl0 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1484 │ │ 14 │ llw0 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 15 │ utun0 │ up|pointtopoint|multicast │ │ 1380 │ │ 16 │ utun1 │ up|pointtopoint|multicast │ │ 2000 │ │ 17 │ utun2 │ up|pointtopoint|multicast │ │ 1380 │ │ 18 │ utun3 │ up|pointtopoint|multicast │ │ 1380 │ │ 19 │ utun4 │ up|pointtopoint|multicast │ │ 1380 │ │ 20 │ utun5 │ up|pointtopoint|multicast │ │ 1380 │ │ 21 │ utun6 │ up|pointtopoint|multicast │ │ 1380 │ │ 22 │ utun7 │ up|pointtopoint|multicast │ │ 1380 │ └───────┴─────────┴───────────────────────────┴───────────────────┴───────┘ ``` ## Promiscous Mode Netcap uses promiscous mode by default, which requires root permissions. You can toggle this behavior with the **-promisc** flag: ``` $ net capture -iface en0 -promisc=false ``` ## Windows For windows, things work a little bit different. First, download & install the latest version of **WinPcap**: {% embed url="" %} Next, open a CMD prompt and run: ```aspnet C:\>getmac /fo csv /v "Connection Name","Network Adapter","Physical Address","Transport Name" "Ethernet0","Intel(R) PRO/1000 MT Network Connection","00-0C-29-BB-EC-9B","\Device\Tcpip_{B1B1E59F-FA8F-4A7B-B28C-7A26F6E00F5A}" ``` Note down the Identifier for your adapter of interest (here: **Ethernet0**), in this example the identifier is: ```aspnet \Device\Tcpip_{B1B1E59F-FA8F-4A7B-B28C-7A26F6E00F5A} ``` To capture traffic on the interface, you must prefix the interface **ID ({B1B1E59F-FA8F-4A7B-B28C-7A26F6E00F5A})** with **\Device\NPF\_** This leaves us with the final command: ```aspnet net.exe capture -iface \Device\NPF_{B1B1E59F-FA8F-4A7B-B28C-7A26F6E00F5A} ``` {% hint style="info" %} Info: When you stop the packet capture on windows with ctrl-C you will see several errors of the format: `failed to remove file remove XXXXX.ncap.gz: The process cannot access the file because it is being used by another process.` This happens because NETCAP creates and opens files for all supported audit records types on startup, and closes them when packet capture is finished or interrupted. Since it often happens that not all supported protocols appeared in the data stream, NETCAP opens the audit record files after closing again, to check if they are empty (=only contain the NETCAP header), and if so, removes the empty audit record files. Unfortunately, windows does not allow closing and opening a file from the same process within such small time interval, which leads to the shown error. As a consequence, empty audit record files are not removed automatically on windows. If you know a workaround for this, please let me know. {% endhint %} # Maltego Integration Graphical link analysis to the rescue! ## Introduction **Maltego** is an open source intelligence (OSINT) and graphical link analysis tool for gathering and connecting information for investigative tasks. {% embed url="" %} It allows to transform data using external knowledge and visualize the results in a graph topology. Transforms are small pieces of code that automatically fetch data from different sources and return the results as visual entities in the desktop client. Transforms are the central elements of Maltego which enable its users to unleash the full potential of the software whilst using a point-and-click logic to run analyses. Netcap provides a set of entities and transformations to analyze packet capture dump files in Maltego! The current implementation focuses on behavorial analysis of entities within the traffic dump. ## Installation Ensure netcap **>= v0.5** is installed and can be found in **$PATH**: ``` $ net -version v0.5 ``` Ensure the **net** binary is placed in **/usr/local/bin**: ``` $ which net /usr/local/bin/net ``` {% hint style="info" %} Transformations in maltego have to specify a working directory. Currently /**usr/local** is used for this so make sure the current user has sufficient right to enter the directory. No data will be written there, all logs from transformations that invoke the netcap core are written into dedicated directories for each processed pcap file. {% endhint %} Next, download install the maltego transformations and enities for netcap: Currently there are **20 entities** and **42 transformations** implemented. You can download them here: {% file src="/files/-M5WMXmGHlrZKgvb0PLA" %} Import them into Maltego in the "**Import / Export Config**" tab under "**Import Config**". ## Loading PCAP files into Maltego To load a pcap file into maltego you have two options: 1\) Drag and Drop the file into a maltego graph. The files type will be **maltego.File** by default, and the path to the file on disk is set as a note on the entity. Now, change the entities type to **netcap.PCAP**, and double click it to open the detail view. Copy the filesystem path from the **Notes** tab into the **path** property of the PCAP entity. 2\) Create a new **netcap.PCAP** entity and set the **path** property to the path of your pcap file on disk ## Running Transformations Right click an entity and start typing **Get** into the search bar to see all available transformations for the selected type. Alternatively you can also use the **Run View** in the **Windows** tab to see and launch available transformations with a single click. Transformations are usually bound to specific entities. For example, the **netcap.DeviceProfiles** entity, which represents the device profiles that have been derived from an input PCAP file, currently only offers the following two transformations: ![](/files/-M5Xfon6c4TAO_Kbh1IN) **GetDeviceProfilesWithDPI** enables deep packet inspection, which requires to install dependencies and can slow down the processing drastically. When not using DPI the transformations making use of this data will simply return no results. To add the actual devices to the graph, use the **GetDevices** transformation. ![](/files/-M5XenFiN06my_CIY5Mu) This bring the first usable entities to the graph, of type **netcap.Device**. A device has contacts and addresses it has been using to access network services by itself. When selecting a device entity, you will see the following transformations: ![](/files/-M5XfSCvVACoq0Ki1VfO) The generated entities will be of type **netcap.IPAddr** and contain information about the host, as well as a set of transforms to further drill down and investigate. When selecting an entity of type **netcap.IPAddr**, the following transformations are offered: ![netcap.IPAddr transformations](/files/-M5XdRgKRJe1lBrsszv-) ## Configuration Netcap offers an **OpenFile** maltego transform, which will pass filetypes except for executables to the default system application for the corresponding file format. On macOS the open utility will be used for this and on the linux the default is gio open. You can override the application used for this by setting **NC\_MALTEGO\_OPEN\_FILE**. ## Examples Search for DHCP information from the selected hosts: ![](/files/-M5XhHXTxQ-07HJGtQLH) Add Server Names provided as SNI on the TLS handshake: ![](/files/-M5Xhaq2Un74mbFGA4tD) Use Deep Packet Inspection to list all identified application categories: ![](/files/-M5XhpGDioQY16Px9mQd) Extraction of a POP3 authentication token: ![](/files/-M5XmdbCPzyLQcI9jkm8) ## Gallery When working with larger amount of nodes, the organic topology can be useful: ![Graph during an investigation (organic topology)](/files/-M4pT4QzVTRyAYY_MIhK) Example of interaction with a PHP webshell: ![PHP webshell interaction](/files/-M4pT4R0oxG4V3OIhy5P) Example of an exploit abusing a HTTP parameter command injection vulnerability: ![HTTP parameter command injection](/files/-M4pT4R2VkF2sr1iKZSE) Graph during an investigation where the attacker has been identified and further information is gathered: ![Dataset investigation](/files/-M4zlcbKtUlsLz79kL2e) ![Dataset investigation](/files/-M4zm2PH-Rh4-kt99rX5) ![Flow Graph](/files/-M4zloIdCPwzDrcwoYtu) ## Detail Views ![](/files/-M5Xh0dpS4dTdt0bzCF-) ![](/files/-M5XiU2R3LIJMsfM4GOX) ![](/files/-M5XimlxAjqbP6NdGMv0) ![](/files/-M5XipirHhG9fK26tfBE) ![](/files/-M5XiuwQt33Bk69C3g6r) ![](/files/-M5Xj0MFh3cX41uaEuLY) # Logging Logging options ## Quiet mode Netcap writes a general summary to stdout, if you wish to disable output entirely use the **-quiet** flag: ``` $ net capture -quiet ``` When the quiet mode is used, the output is instead written into the **netcap.log** file in the directory where netcap is executed from. ## Decoding errors Errors when parsing packets are logged by default into the **errors.log** file in the current directory. Each log entry contains a hex dump of the entire packet and the error message or stack trace. ## Log files in debug mode The following log file are produced when running with the **-debug** flag: * debug.log: general debug messages * reassembly.log: tcp stream reassembly debug logs # Packet Contexts Preserve information about other layers on audit records Netcap v0.4.3 added PacketContexts, a new feature to preserve additional information on audit records. This need originates from a core concept of Netcap: separating the results based on the different protocols, which results in audit records like TCP not providing any IP address information, since they only provide information about the TCP protocol which operates at the Transport Layer. A packet context looks as follows: ```erlang message PacketContext { string SrcIP = 1; string DstIP = 2; string SrcPort = 3; string DstPort = 4; } ``` Many audit record types (e.g: IPv4, IPv6, TCP, UDP, ICMPv4, ICMPv6, SCTP, DNS, DHCP, SIP etc) now have an addional field called context, which will contain a PacketContext that describes the flow where the packet originated from, if context capture is enabled. When generating a CSV representation the fields from the PacketContext are flattened, which means they will be shown as if they are direct member of the dumped audit record, e.g: ``` $ net dump -read UDP.ncap.gz -fields Timestamp,SrcPort,DstPort,Length,Checksum,PayloadEntropy,PayloadSize,Payload,SrcIP,DstIP ``` If the audit record already has information that would be duplicated by the PacketContext (for example Port information for UDP), this information is cleared on the context to avoid repetition. Context capture is enabled by default and can be controlled using the **-context** flag. # Industrial Control Systems ICS / SCADA threat hunting ## Protocol Support Netcap offers audit records for the following protocols seen in industrial control systems: * Ethernet/IP * CIP - Common Industrial Protocol * Modbus / ModbusTCP The encoders are enabled by default. ## Modbus ```erlang message Modbus { string Timestamp = 1; int32 TransactionID = 2; // Identification of a MODBUS Request/Response transaction int32 ProtocolID = 3; // It is used for intra-system multiplexing int32 Length = 4; // Number of following bytes (includes 1 byte for UnitIdentifier + Modbus data length int32 UnitID = 5; // Identification of a remote slave connected on a serial line or on other buses bytes Payload = 6; bool Exception = 7; int32 FunctionCode = 8; PacketContext Context = 9; } ``` ## CIP ```erlang message CIP { string Timestamp = 1; bool Response = 2; // false if request, true if response int32 ServiceID = 3; // The service specified for the request uint32 ClassID = 4; // request only uint32 InstanceID = 5; // request only int32 Status = 6; // Response only repeated uint32 AdditionalStatus = 7; // Response only bytes Data = 8; // Command data for request, reply data for response PacketContext Context = 9; } ``` ## ENIP ```erlang message ENIP { string Timestamp = 1; uint32 Command = 2; uint32 Length = 3; uint32 SessionHandle = 4; uint32 Status = 5; bytes SenderContext = 6; uint32 Options = 7; ENIPCommandSpecificData CommandSpecific = 8; PacketContext Context = 9; } ``` # File Extraction Extract transferred files and save them to disk ## Introduction Various protocols allow transferring files (e.g: HTTP, POP3) and some are made for the sole purpose of transferring files (FTP, SMB etc). From a network security monitoring perspective, transferred files are interesting because they can contain malicious software or prohibited content. Netcap extracts files from HTTP and saves them to disk, for both HTTP responses and HTTP requests. It uses the **File** audit record type to model the extracted information. > Future versions will add file extraction support for other protocols as well. ## File Audit Records The audit record definition for a file looks like this: ```erlang message File { string Timestamp = 1; string Name = 2; int64 Length = 3; string Hash = 4; string Location = 5; string Ident = 6; string Source = 7; string ContentType = 8; PacketContext Context = 9; string Host = 10; string ContentTypeDetected = 11; } ``` As can be seen, the content type indicated by the HTTP header is included, as well as the content type that was detected. In addition, the source of the File is specified (e.g: from HTTP, Mail attachment etc), as well the identifier of the connection where it originated from. The Hash field currently holds an MD5 hash of the file, Location points to the path on disk where the file is stored. > This will likely be replaced with a stronger hash function in the future. ## Usage To enable file capture, set the **-fileStorage** flag and supply a path to store the files to (will be created if it does not exist): ``` $ net capture -read traffic.pcap -fileStorage files ``` After capturing, lets inspect the directory contents: ``` $ tree files files ├── application │   └── x-gzip │   └── unknown-193.24.227.12->216.66.80.30-80->60075.gz ├── image │   └── x-icon │   └── favicon.ico-193.24.227.12->216.66.80.30-80->60076.ico └── text └── html ├── unknown-193.24.227.12->216.66.80.30-80->55031.html ├── unknown-193.24.227.12->216.66.80.30-80->55032.html ├── unknown-193.24.227.12->216.66.80.30-80->55033.html └── unknown-80.237.133.136->192.168.110.10-80->1152.html 6 directories, 6 files ``` As you can see, files are sorted by their MIME types retrieved from classifying them using the go standard library and named after the TCP connection they originated from. By default, only complete requests and responses are captured, if you also want to extract incomplete data, use the **-writeincomplete** flag: ``` $ net capture -read traffic.pcap -fileStorage files -writeincomplete ``` Dumping a File on the commandline looks like this: ``` $ net dump -read File.ncap.gz -struc NC_File Timestamp: "2015-03-08 14:05:29.664213 +0000 UTC" Name: "ads.bmp" Length: 126 Hash: "2d5a035011854b04a456b244b15a583b" Location: "files/image/bmp/ads.bmp-80.239.178.178->192.168.0.51-80->41214.bmp" Ident: "80.239.178.178->192.168.0.51-80->41214" Source: "HTTP RESPONSE from /ads.bmp" Context: < SrcIP: "192.168.0.51" DstIP: "80.239.178.178" SrcPort: "41214" DstPort: "80" > ContentTypeDetected: "image/bmp" ... ``` For properly exploring files for each host I recommend using the Maltego Integration: {% content-ref url="/pages/-M4pT434l2NNMvO9nGCE" %} [Maltego Integration](/maltego-integration) {% endcontent-ref %} ![](/files/-M5ky4VdFrnqCMdYGA2l) # Email Extraction Extract transferred emails ## Motivation Emails are a key communication mechanism that holds plenty of digital evidence, starting from Mail header information about the sender and route, to transferred files via attachments. Netcap currently extracts Email fetched over POP3 and support IMAP is in the making. ## POP3 A POP3 audit record contains information about the addresses involved, as well as authentication information. The fetched mails are provided as an array of **Mail** instances. ```erlang message POP3 { string Timestamp = 1; string ClientIP = 2; string ServerIP = 3; string AuthToken = 4; string User = 5; string Pass = 6; repeated Mail Mails = 7; } ``` A Mail instance has the following fields: ```erlang message Mail { string ReturnPath = 1; string DeliveryDate = 2; string From = 3; string To = 4; string CC = 5; string Subject = 6; string Date = 7; string MessageID = 8; string References = 9; string InReplyTo = 10; string ContentLanguage = 11; bool HasAttachments = 12; string XOriginatingIP = 13; string ContentType = 14; string EnvelopeTo = 15; repeated MailPart Body = 16; } ``` For exploring captured emails, the Maltego Integration can be used: {% content-ref url="/pages/-M4pT434l2NNMvO9nGCE" %} [Maltego Integration](/maltego-integration) {% endcontent-ref %} ![](/files/-M5kyIbH_wnm4b1scsMm) ![](/files/-M5kyNom5uMyxBCOnl-H) ## SMTP For SMTP an audit record is also available, though mail extraction has not been implemented yet: ```erlang message SMTP { string Timestamp = 1; bool IsEncrypted = 2; bool IsResponse = 3; repeated SMTPResponse ResponseLines = 4; SMTPCommand Command = 5; PacketContext Context = 6; } ``` # Device Profiles Behavorial Profiling with Netcap ## Motivation Which device on the network uses which IP address? Which addresses / devices did it contact? How are devices related to each other, how does communication flow? Identifying devices within a network is a good starting point for any investigation and helps to understand complex situations and relations quickly. The **DeviceProfile** custom encoder implements exactly this, and is enabled from v0.5 on by default. > Note: DeviceProfile currently get written when processing all traffic is done - that means when using live capture, the profiles will be available when processing stopped. Future versions will implement a flushing mechanism similar to the one for Flows / Connections. DeviceProfiles rely heavily on local resolvers to be set up and configured. You can use the **DeviceProfile** audit records without resolvers, but you will get less information. Read more about the resolvers here: {% content-ref url="/pages/-M4pSNs9Ewumxyv3nEjO" %} [Resolvers](/resolvers) {% endcontent-ref %} Analyzing DeviceProfiles can be done using Maltego, for example: {% content-ref url="/pages/-M4pT434l2NNMvO9nGCE" %} [Maltego Integration](/maltego-integration) {% endcontent-ref %} ![DeviceProfiles and their used IP addresses from an industrial automation system](/files/-M5lEVm_iZaECOSQ137P) ## DeviceProfile Audit Records Lets look at the protocol buffer definition for a device profile: ```erlang message DeviceProfile { string MacAddr = 1; string DeviceManufacturer = 2; repeated IPProfile DeviceIPs = 3; repeated IPProfile Contacts = 4; int64 NumPackets = 5; string Timestamp = 6; // first seen uint64 Bytes = 7; } ``` As you can see, a DeviceProfile is a summary structure built around the hardware address of a physical device. It captures the addresses that have been used, as well as the contacted addresses in form of IPProfiles, among other meta information, like the number of packets or the hardware manufacturer. Lets take a closer look at an IPProfile: ```erlang message IPProfile { string Addr = 1; int64 NumPackets = 2; string Geolocation = 3; repeated string DNSNames = 4; string TimestampFirst = 5; string TimestampLast = 6; repeated string Applications = 7; map Ja3 = 8; // ja3 to lookup map Protocols = 9; uint64 Bytes = 10; map DstPorts = 11; // Ports to bytes map SrcPorts = 12; // Ports to bytes map SNIs = 13; } ``` This is the information associated with a single ip address. Note how in addition to general meta data like the number of packets, bytes and timestamps, there is also information retrieved from the new resolvers API, namely the geolocation and dns names. Additionally, the results from Deep Packet Inspection for all flows seen from or towards this IP are added as well, in addition to the Server Name Indicators seen and flow statistics for each seen port number from this address. To enhance encrypted telemetry, Ja3 fingerprints seen for this host are mapped to lookup results from the Ja3 database. # Python Integration Read Netcap Audit records from Python ## Source Code The Python library for interacting with netcap audit records has been published here: {% embed url="" %} ## Usage ### Read into python dictionary Currently it is possible to retrieve the audit records as python dictionary: ```python #!/usr/bin/python import pynetcap as nc reader = nc.NCReader('pcaps/HTTP.ncap.gz') reader.read(dataframe=False) print("RECORDS:") print(reader.records) ``` ### Read into pandas dataframe Retrieving the audit records as pandas dataframe: ```python #!/usr/bin/python import pynetcap as nc reader = nc.NCReader('pcaps/HTTP.ncap.gz') reader.read(dataframe=True) print("[INFO] completed reading the audit record file:", reader.filepath) print("DATAFRAME:") print(reader.df) ``` # Changelog Detailed Version History Information ## v0.5 - April 2020 ### Fixed * multiple bugs in the stream reassembly * several panics during parsing in gopacket ### Changed * CLI interface refactored: single binary app with subcommands, stripped size \~**17MB** * Updated units tests * Documentation updates * Updated Docker containers for **Ubuntu** and **Alpine** * Compiled with **Go 1.14.2** * removed custom audit records Link-, Network- and TransportFlow ### New Features * **Maltego** integration * **File** audit records * **Diameter** protocol audit records * **SMTP** audit records * **POP3** support for extracting Mails * **JA3S** support and separate audit record for **TLSServerHello** * New configuration options: via **environment variables** or **configuration** file * Resolvers package for **Geolocation**, **DNS** and **Service** lookups and **whitelisting** * Deep Packet Inspection via **nDPI** and **libprotoident** * **DeviceProfile** Audit records, to capture the behavior of a single device within a traffic dump * Added an integration for **bash-completion** support # Troubleshooting Look behind the curtain ## First Aid * are all files in place? * does the current user have sufficient rights to access them? * does the current user have sufficient rights to access the current working directory? ## Pitfalls * the go flag package implementation only allows to set boolean value using the **-name=value** syntax (e.g: **-debug=true**), but strings can be set also using a space instead with the **-name value** syntax (e.g: **-read traffic.pcap**) ## Debug Mode Use the **-debug** flag to generate debug logs. The files will be created in the directory from which the netcap process has been started. The reassembly engine logs data into **reassembly.log** and all other debug messages go into **debug.log.** > Remember that netcap logs packet decoding errors into **errors.log!** ## Advanced Debugging In order to use advanced debugging features, you will need to recompile the code and make a few changes. Some primitives in the core library have a second implementation that spawns an additional goroutine for each invocation to time out the call. This is useful to debug hangs in the reassembly or due to invocation of external code, for example the DPI integration of **nDPI** and **libprotoident**. List of \*Timeout primitives: * handlePacketTimeout(p \*packet) in collector.go * AssembleWithContextTimeout(...) in tcpConnection.go * GetProtocolsTimeout(packet gopacket.Packet) in dpi.go Simply replace the calls to the original versions with the \*Timeout primitives, and if one of those will block for longer than the configured thresholds, the call will be interrupted and an error logged. ## Race Detection Builds To debug synchronization problems and data races you can compile a version with the **-race** flag set for the go compiler and see if the program crashes due to a race condition. There is a command implemented for that in the build scripts: ``` $ zeus install-race ``` # Unit Tests Netcap has tests for its core functionality ## Prerequisites Some of the tests operate on a dump file that is not in the repository. You can download it with: ``` $ zeus download-test-pcap ``` which will basically just invoke: ``` wget https://weberblog.net/wp-content/uploads/2020/02/The-Ultimate-PCAP.7z ``` Now unpack the file and move it to the tests folder in the project root. ## Unit Tests Unit tests have been implemented for parts of the core functionality. Currently there are basic tests for reading pcap data from files and traffic live from an interface, as well as tests and benchmarks for common utility functions, such progress displaying and time conversions. The tests and benchmarks can be executed from the repository root by executing the following from the project root: ``` $ go test -v ./... === RUN TestCountRecords --- PASS: TestCountRecords (0.18s) === RUN TestReader --- PASS: TestReader (0.01s) === RUN TestWriter --- PASS: TestWriter (0.06s) PASS ok github.com/dreadl0ck/netcap 0.862s ? github.com/dreadl0ck/netcap/cmd [no test files] ? github.com/dreadl0ck/netcap/cmd/agent [no test files] ? github.com/dreadl0ck/netcap/cmd/capture [no test files] ? github.com/dreadl0ck/netcap/cmd/collect [no test files] ? github.com/dreadl0ck/netcap/cmd/dump [no test files] ? github.com/dreadl0ck/netcap/cmd/export [no test files] ? github.com/dreadl0ck/netcap/cmd/label [no test files] ? github.com/dreadl0ck/netcap/cmd/proxy [no test files] ? github.com/dreadl0ck/netcap/cmd/split [no test files] ? github.com/dreadl0ck/netcap/cmd/transform [no test files] ? github.com/dreadl0ck/netcap/cmd/util [no test files] === RUN TestCollectPCA done in 2.595847118s --- PASS: TestCollectPCAP (2.60s) PASS ok github.com/dreadl0ck/netcap/collector 3.242s === RUN TestCorruptedWriter TestCorruptedWriter: delimited_test.go:46: Put record returned expected error: BAD --- PASS: TestCorruptedWriter (0.00s) === RUN TestGoodWriter --- PASS: TestGoodWriter (0.00s) === RUN TestCorruptedReader TestCorruptedReader: delimited_test.go:87: Next record returned expected error: unexpected EOF --- PASS: TestCorruptedReader (0.00s) === RUN TestGoodReader --- PASS: TestGoodReader (0.00s) === RUN TestRoundTrip TestRoundTrip: delimited_test.go:136: After writing: buffer="\x04Some\x02of\x04what\x01a\x04fool\x06thinks\x05often\bremains." len=42 --- PASS: TestRoundTrip (0.00s) PASS ok github.com/dreadl0ck/netcap/delimited 0.526s ? github.com/dreadl0ck/netcap/dpi [no test files] ? github.com/dreadl0ck/netcap/encoder [no test files] ? github.com/dreadl0ck/netcap/io [no test files] ? github.com/dreadl0ck/netcap/label [no test files] ? github.com/dreadl0ck/netcap/maltego [no test files] ? github.com/dreadl0ck/netcap/metrics [no test files] ? github.com/dreadl0ck/netcap/resolvers [no test files] === RUN TestMarshal --- PASS: TestMarshal (0.00s) PASS ok github.com/dreadl0ck/netcap/types 0.668s === RUN TestTimeToString --- PASS: TestTimeToString (0.00s) === RUN TestStringToTime --- PASS: TestStringToTime (0.00s) PASS ok github.com/dreadl0ck/netcap/utils 0.932s ``` ## Benchmarks Run the benchmarks using: ``` $ go test -bench=. ./... | grep -E "Bench|pkg" pkg: github.com/dreadl0ck/netcap/collector BenchmarkReadPcapNG-12 1265539 844 ns/op 1249 B/op 1 allocs/op BenchmarkReadPcapNGZeroCopy-12 2028283 640 ns/op 0 B/op 0 allocs/op BenchmarkReadPcap-12 7557667 137 ns/op 106 B/op 1 allocs/op pkg: github.com/dreadl0ck/netcap/types BenchmarkMarshal-12 9817819 110 ns/op 64 B/op 1 allocs/op BenchmarkUnmarshal-12 8703766 134 ns/op 40 B/op 2 allocs/op pkg: github.com/dreadl0ck/netcap/utils BenchmarkTimeToStringOld-12 5283726 229 ns/op 64 B/op 4 allocs/op BenchmarkTimeToString-12 8273997 136 ns/op 80 B/op 3 allocs/op BenchmarkStringToTime-12 8842005 137 ns/op 32 B/op 1 allocs/op BenchmarkStringToTimeFieldsFunc-12 6809409 185 ns/op 32 B/op 1 allocs/op BenchmarkProgressOld-12 54425902 21.0 ns/op 0 B/op 0 allocs/op BenchmarkProgress-12 23389420 45.9 ns/op 16 B/op 2 allocs/op ``` ## Race Detection Tests Run the tests with race detection enabled: ``` $ go test -race -v ./... ``` # Extension Implementing new audit records and features To add support for a new protocol or custom abstraction the following steps need to be performed. ## Protocol Buffer Definitions First, a type definition of the new audit record type must be added to the AuditRecord protocol buffers definitions, as well as a **Type enumeration** following the naming convention with the **NC prefix**. First, make sure you have code generator plugin(s) that NETCAP is using to accelerate the protocol buffer en- and decoding. Get the plugins with: ```go $ go get github.com/gogo/protobuf/... ``` The framework for this can be found here: {% embed url="" %} Recompile the protocol buffers with: ```go $ zeus gen-proto-dev ``` This will create the type definitions for your new audit record in the **types** package. ## Encoder Implementation After recompiling the protocol buffers, a file for the new encoder named after the protocol must be created in the encoder package. The new file must contain a variable created with **CreateLayerEncoder** or **CreateCustomEncoder** depending on the desired encoder type. Lets take a brief look at a very simple **LayerEncoder**, for example for the ARP protocol: ```go package encoder import ( "github.com/dreadl0ck/gopacket" "github.com/dreadl0ck/gopacket/layers" "github.com/dreadl0ck/netcap/types" "github.com/golang/protobuf/proto" ) var arpEncoder = CreateLayerEncoder( types.Type_NC_ARP, layers.LayerTypeARP, func(layer gopacket.Layer, timestamp string) proto.Message { if arp, ok := layer.(*layers.ARP); ok { return &types.ARP{ Timestamp: timestamp, AddrType: int32(arp.AddrType), Protocol: int32(arp.Protocol), HwAddressSize: int32(arp.HwAddressSize), ProtAddressSize: int32(arp.ProtAddressSize), Operation: int32(arp.Operation), SrcHwAddress: arp.SourceHwAddress, SrcProtAddress: arp.SourceProtAddress, DstHwAddress: arp.DstHwAddress, DstProtAddress: arp.DstProtAddress, } } return nil } ) ``` �Since **ARP** can be decoded by **gopacket** already, all we have to do is check if the packet has the **ARP** layer, and if yes, convert it to the **types.ARP** audit record and return it. The constructor for a **LayerEncoder** needs the type enumeration for the new audit record, as well as the **gopacket.LayerType**, followed by the actual encoder function. This function will be called for every network packet. As you can see, **LayerEncoders** are tied to **gopacket**. If you want to implement custom decoding logic or support for a new protocol, you essentially have two options: * implement protocol decoding in **gopacket**, then use a **LayerEncoder** in netcap * implement protocol decoding in a **CustomEncoder** A **CustomEncoder** works the same way but offers more flexibility for the implementation, like functions for initialisation and teardown. The CustomEncoder constructor signature looks as follows: ```go func CreateCustomEncoder( t types.Type, name string, postinit func(*CustomEncoder) error, handler CustomEncoderHandler, deinit func(*CustomEncoder) error ) *CustomEncoder ``` The **CustomEncoderHandler** will simply receive the raw **gopacket.Packet** and return a **proto.Message**: ```go CustomEncoderHandler = func(p gopacket.Packet) proto.Message ``` Depending on the choice of the encoder type, the new variable must be added to the customEncoderSlice in **encoder/customEncoder.go** or layerEncoderSlice in **encoder/layerEncoder.go**. ## Audit Record Interface Implementation Next, the interface for conversion to CSV and JSON and exporting metrics must be implemented in the types package, by creating a new file with the protocol name and implementing the **types.AuditRecord** interface: ```go // AuditRecord is the interface for basic operations with NETCAP audit records // this includes dumping as CSV or JSON or prometheus metrics // and provides access to the timestamp of the audit record type AuditRecord interface { // returns CSV values CSVRecord() []string // returns CSV header fields CSVHeader() []string // used to retrieve the timestamp of the audit record for labeling Time() string // Src returns the source of an audit record // for Layer 2 records this shall be the MAC address // for Layer 3+ records this shall be the IP address Src() string // Dst returns the source of an audit record // for Layer 2 records this shall be the MAC address // for Layer 3+ records this shall be the IP address Dst() string // increments the metric for the audit record Inc() // returns the audit record as JSON JSON() (string, error) // can be implemented to set additional information for each audit record // important: // - MUST be implemented on a pointer of an instance // - the passed in packet context MUST be set on the Context field of the current audit record SetPacketContext(ctx *PacketContext) } ``` If the new protocol contains sub-structures, functions to convert them to strings need to be implemented as well. Take a look at other encoders that have lots of substructures, for example **DNS**. ## Add Initializer Finally, the **InitRecord(typ types.Type) (record proto.Message)** function in netcap.go needs to be updated, to initialize the structure for the new type. # Downloads A collection of cheatsheets and useful resources ## Releases You can find the latest release on the releases page on GitHub: {% embed url="" %} NETCAP GitHub Releases Page {% endembed %} ## Publications In this paper, we explore Graph based analysis using Maltego to visualise data from NETCAP during a forensic investigation: {% file src="/files/-M5lnlgF79aR3khnqCiy" %} Behavorial Profiling From Network Packet Captures {% endfile %} ### Thesis {% file src="/files/-M5lnKCoA1AP7qKlW\_Zb" %} Implementation and Evaluation of secure and scalable anomaly-based Network Intrusion Detection {% endfile %} ### Thesis Presentation {% file src="/files/-M0KJn1Kokg4BOxvWSjZ" %} ### SecurIT Cup 2018 Presentation {% file src="/files/-M5ln9oeQgK7N1WcV6Rz" %} ## External Publications The authors used the framework to process their recorded PCAP dumps: {% embed url="" %} ## Cheatsheets ### List of all supported protocols and fields {% file src="/files/-M60zpQQtkygr86V87Bh" %} ### Command Cheatsheet {% file src="/files/-M5loDsA\_HO6ISWg-Bis" %} # Docker Containers ## Docker Hub There are ubuntu and alpine linux docker containers available with netcap and dependencies (e.g: libprotoident, nDPI) preinstalled. {% embed url="" %} NETCAP on docker hub {% endembed %} ## Pull Containers To get the v0.5 ubuntu container: ``` $ docker pull dreadl0ck/netcap:ubuntu-v0.5 ``` To get the v0.5 alpine container: ``` $ docker pull dreadl0ck/netcap:alpine-v0.5 ``` ## Run Containers To run the v0.5 ubuntu container: ``` $ docker run -it dreadl0ck/netcap:ubuntu-v0.5 bash ``` To run the v0.5 alpine container: ``` $ docker run -it dreadl0ck/netcap:alpine-v0.5 ash ``` > Tip: You can use the docker run **-v** flag to mount a volume with your packet captures into the container # FAQ Frequently Asked Questions ## What can I use this for? This is a framework for capturing and analysing network packets. For common protocols, chances are high that NETCAP already supports parsing them. If not, it's simple to implement. NETCAP uses its own data storage format, that focuses on efficiency and applies compression by default. The data format allows accessing the extracted information across all major programming languages as type safe structured data, which makes it easy to implement quality software on top the NETCAP core engine. ## How can I contribute to the project? Please see the Contributing page. {% content-ref url="/pages/-Le3y6RcIsJTEFMmDv01" %} [Contributing](/contributing) {% endcontent-ref %} ## I have a question, how can I reach you? Contact me by mail: ``` dreadl0ck [at] protonmail [dot] ch ``` # Contributing Contributing to the Netcap project Contributions welcome! If you have an idea or questions, please feel free to reach out! ## Issues & Bug Reports Please include the Netcap version, the exact error messages and as much log output as possible! Try to answer these questions in your Bug Report: * What version of Netcap, which OS, which version of OS did you use? * What did you want to do? * What happened instead? * What output did you get? ## Pull Requests Before submitting your pull request please make sure the unit tests execute without errors. ## Feature Requests You have an idea for a new feature? Create a feature draft and open an issue to discuss it. # License License type and terms ## GNU General Public License v3.0 Netcap is licensed under the GNU General Public License v3, which is a very permissive open source license, that allows others to do almost anything they want with the project, except to distribute closed source versions. Permissions of this strong copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights. This license type was chosen with Netcaps research purpose in mind, and in the hope that it leads to further improvements and new capabilities contributed by other researchers on the long term. The license can be found here: ## Usage Disclaimer Netcap was developed in a short timeframe as a research project and thus was neither tested nor developed to run in a production environment. The project may contain bugs, that have not yet been discovered. Error handling is not very graceful, in many cases that could have been handled otherwise, the program panics in order to assist in debugging with a stack trace. Until there are further unit tests and the error handling is more robust, using Netcap for other purposes than research is not recommended! # Overview A brief overview ![](/files/RYB96GciVx899YfpzEjI) The *Netcap* (NETwork CAPture) framework efficiently converts a stream of network packets into platform neutral type-safe structured audit records that represent specific protocols or custom abstractions. These audit records can be stored on disk or exchanged over the network, and are well suited as a data source for machine learning algorithms. Since parsing of untrusted input can be dangerous and network data is potentially malicious, implementation was performed in a programming language that provides a garbage collected memory safe runtime. It was developed for a series of experiments in my bachelor thesis: *Implementation and evaluation of secure and scalable anomaly-based network intrusion detection*. The thesis is included at the root of this repository (file: [mied18.pdf](https://github.com/dreadl0ck/netcap/blob/master/mied18.pdf)) and can be used to as an introduction to the framework, its philosphy and architecture. However, be aware that the command-line interface was refactored heavily and the thesis examples refer to very early versions. This documentation contains the latest API and usage examples. Slides from my presentation at the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities are available on [researchgate](https://www.researchgate.net/project/Anomaly-based-Network-Security-Monitoring). The project won the 2nd Place at Kaspersky Labs SecurIT Cup 2018 in Budapest. *Netcap* uses Google's Protocol Buffers to encode its output, which allows accessing it across a wide range of programming languages. Alternatively, output can be emitted as comma separated values, which is a common input format for data analysis tools and systems. The tool is extensible and provides multiple ways of adding support for new protocols, while implementing the parsing logic in a memory safe way. It provides high dimensional data about observed traffic and allows the researcher to focus on experimenting with novel approaches for detecting malicious behavior in network environments, instead of fiddling with data collection mechanisms and post processing steps. It has a concurrent design that makes use of multi-core architectures. The name *Netcap* was chosen to be simple and descriptive. The command-line tool was designed with usability and readability in mind, and displays progress when processing packets. The latest version offers 66 audit record types of which 55 are protocol specific and 8 are custom abstractions, such as flows or transferred files. ## Design Goals * memory safety when parsing untrusted input * ease of extension * output format interoperable with many different programming languages * concurrent design * output with small storage footprint on disk * gather everything, separate what can be understood from what can't * allow implementation of custom abstractions * rich platform and architecture support ## Framework Components The framework consists of 9 logically separate tools compiled into a single binary: * capture (capture audit records live or from dumpfiles) * dump (dump with audit records in various formats) * label (tool for creating labeled CSV datasets from netcap data) * collect (collection server for distributed collection) * agent (sensor agent for distributed collection) * proxy (http reverse proxy for capturing traffic from web services) * util (utility tool for validating audit records and converting timestamps) * export (exporter for prometheus metrics) * transform (maltego transformation plugin) ## Use Cases * monitoring honeypots * monitoring medical / industrial devices * research on anomaly-based detection mechanisms * Forensic data analysis ## Demos A simple demonstration of generating audit records from a PCAP dump file, querying and displaying the collected information in various ways {% embed url="" %} And live operation decoding traffic from my wireless network interface, while I am surfing the web {% embed url="" %} Exploring HTTP audit records {% embed url="" %} ### Deep Learning Watch a quick demo of the deep neural network for classification of malicious behavior, on a small PCAP dump file with traffic from the LOKI Bot. First, the PCAP file is parsed with [netcap](https://github.com/dreadl0ck/netcap-tf-dnn/blob/master/github.com/dreadl0ck/netcap), in order to get audit records that will be labeled afterwards with the [netlabel](https://github.com/dreadl0ck/netcap#netlabel-command-line-tool) tool. The labeled CSV data for the TCP audit record type is then used for training (75%) and evaluation (25%) of the classification accuracy provided by the deep neural network. {% embed url="" %} ## License Netcap is licensed under the GNU General Public License v3, which is a very permissive open source license, that allows others to do almost anything they want with the project, except to distribute closed source versions. This license type was chosen with Netcaps research purpose in mind, and in the hope that it leads to further improvements and new capabilities contributed by other researchers on the long term. For more infos refer to the License page. ## Source Code Stats Stats for netcap v0.5, generated with cloc version 1.80 > $ zeus cloc ``` 444 text files. 444 unique files. 158 files ignored. github.com/AlDanial/cloc v 1.84 T=0.26 s (1090.4 files/s, 116481.5 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Go 277 4191 4788 21031 Markdown 9 123 0 503 YAML 1 5 4 14 ------------------------------------------------------------------------------- SUM: 287 4319 4792 21548 ------------------------------------------------------------------------------- ``` # Audit Records An overview of supported protocols and available fields The following markdown overview was generated using: ``` $ net capture -overview ``` ## NETCAP Overview v0.5 > Documentation: [docs.netcap.io](https://docs.netcap.io) > > #### LayerEncoders > > #### CustomEncoders | Name | NumFields | Fields | | -------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TLSClientHello | 27 | Timestamp, Type, Version, MessageLen, HandshakeType, HandshakeLen, HandshakeVersion, Random, SessionIDLen, SessionID, CipherSuiteLen, ExtensionLen, SNI, OSCP, CipherSuites, CompressMethods, SignatureAlgs, SupportedGroups, SupportedPoints, ALPNs, Ja3, SrcIP, DstIP, SrcMAC, DstMAC, SrcPort, DstPort | | TLSServerHello | 27 | Timestamp, Version, Random, SessionID, CipherSuite, CompressionMethod, NextProtoNeg, NextProtos, OCSPStapling, TicketSupported, SecureRenegotiationSupported, SecureRenegotiation, AlpnProtocol, Ems, SupportedVersion, SelectedIdentityPresent, SelectedIdentity, Cookie, SelectedGroup, Extensions, SrcIP, DstIP, SrcMAC, DstMAC, SrcPort, DstPort, Ja3S | | HTTP | 18 | Timestamp, Proto, Method, Host, UserAgent, Referer, ReqCookies, ResCookies, ReqContentLength, URL, ResContentLength, ContentType, StatusCode, SrcIP, DstIP, ReqContentEncoding, ResContentEncoding, ServerName | | Flow | 17 | TimestampFirst, LinkProto, NetworkProto, TransportProto, ApplicationProto, SrcMAC, DstMAC, SrcIP, SrcPort, DstIP, DstPort, TotalSize, AppPayloadSize, NumPackets, UID, Duration, TimestampLast | | Connection | 28 | TimestampFirst, LinkProto, NetworkProto, TransportProto, ApplicationProto, SrcMAC, DstMAC, SrcIP, SrcPort, DstIP, DstPort, TotalSize, AppPayloadSize, NumPackets, Duration, TimestampLast, BytesClientToServer, BytesServerToClient, NumFINFlags, NumRSTFlags, NumACKFlags, NumSYNFlags, NumURGFlags, NumECEFlags, NumPSHFlags, NumCWRFlags, NumNSFlags, MeanWindowSize, Applications | | DeviceProfile | 8 | Timestamp, MacAddr, DeviceManufacturer, NumDeviceIPs, NumContacts, NumPackets, Bytes, Applications | | File | 12 | Timestamp, Name, Length, Hash, Location, Ident, Source, ContentType, SrcIP, DstIP, SrcPort, DstPort | | POP3 | 7 | Timestamp, Client, Server, AuthToken, User, Pass, NumMails | | Name | NumFields | Fields | | --------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TCP | 25 | Timestamp, SrcPort, DstPort, SeqNum, AckNum, DataOffset, FIN, SYN, RST, PSH, ACK, URG, ECE, CWR, NS, Window, Checksum, Urgent, Padding, Options, PayloadEntropy, PayloadSize, Payload, SrcIP, DstIP | | UDP | 10 | Timestamp, SrcPort, DstPort, Length, Checksum, PayloadEntropy, PayloadSize, Payload, SrcIP, DstIP | | IPv4 | 17 | Timestamp, Version, IHL, TOS, Length, Id, Flags, FragOffset, TTL, Protocol, Checksum, SrcIP, DstIP, Padding, Options, PayloadEntropy, PayloadSize | | IPv6 | 12 | Timestamp, Version, TrafficClass, FlowLabel, Length, NextHeader, HopLimit, SrcIP, DstIP, PayloadEntropy, PayloadSize, HopByHop | | DHCPv4 | 20 | Timestamp, Operation, HardwareType, HardwareLen, HardwareOpts, Xid, Secs, Flags, ClientIP, YourClientIP, NextServerIP, RelayAgentIP, ClientHWAddr, ServerName, File, Options, SrcIP, DstIP, SrcPort, DstPort | | DHCPv6 | 11 | Timestamp, MsgType, HopCount, LinkAddr, PeerAddr, TransactionID, Options, SrcIP, DstIP, SrcPort, DstPort | | ICMPv4 | 7 | Timestamp, TypeCode, Checksum, Id, Seq, SrcIP, DstIP | | ICMPv6 | 5 | Timestamp, TypeCode, Checksum, SrcIP, DstIP | | ICMPv6Echo | 5 | Timestamp, Identifier, SeqNumber, SrcIP, DstIP | | ICMPv6NeighborSolicitation | 5 | Timestamp, TargetAddress, Options, SrcIP, DstIP | | ICMPv6RouterSolicitation | 4 | Timestamp, Options, SrcIP, DstIP | | DNS | 22 | Timestamp, ID, QR, OpCode, AA, TC, RD, RA, Z, ResponseCode, QDCount, ANCount, NSCount, ARCount, Questions, Answers, Authorities, Additionals, SrcIP, DstIP, SrcPort, DstPort | | ARP | 10 | Timestamp, AddrType, Protocol, HwAddressSize, ProtAddressSize, Operation, SrcHwAddress, SrcProtAddress, DstHwAddress, DstProtAddress | | Ethernet | 6 | Timestamp, SrcMAC, DstMAC, EthernetType, PayloadEntropy, PayloadSize | | Dot1Q | 5 | Timestamp, Priority, DropEligible, VLANIdentifier, Type | | Dot11 | 14 | Timestamp, Type, Proto, Flags, DurationID, Address1, Address2, Address3, Address4, SequenceNumber, FragmentNumber, Checksum, QOS, HTControl | | NTP | 19 | Timestamp, LeapIndicator, Version, Mode, Stratum, Poll, Precision, RootDelay, RootDispersion, ReferenceID, ReferenceTimestamp, OriginTimestamp, ReceiveTimestamp, TransmitTimestamp, ExtensionBytes, SrcIP, DstIP, SrcPort, DstPort | | SIP | 11 | Timestamp, Version, Method, Headers, IsResponse, ResponseCode, ResponseStatus, SrcIP, DstIP, SrcPort, DstPort | | IGMP | 15 | Timestamp, Type, MaxResponseTime, Checksum, GroupAddress, SupressRouterProcessing, RobustnessValue, IntervalTime, SourceAddresses, NumberOfGroupRecords, NumberOfSources, GroupRecords, Version, SrcIP, DstIP | | LLC | 6 | Timestamp, DSAP, IG, SSAP, CR, Control | | IPv6HopByHop | 4 | Timestamp, Options, SrcIP, DstIP | | SCTP | 7 | Timestamp, SrcPort, DstPort, VerificationTag, Checksum, SrcIP, DstIP | | SNAP | 3 | Timestamp, OrganizationalCode, Type | | LinkLayerDiscovery | 5 | Timestamp, ChassisID, PortID, TTL, Values | | ICMPv6NeighborAdvertisement | 6 | Timestamp, Flags, TargetAddress, Options, SrcIP, DstIP | | ICMPv6RouterAdvertisement | 9 | Timestamp, HopLimit, Flags, RouterLifetime, ReachableTime, RetransTimer, Options, SrcIP, DstIP | | EthernetCTP | 2 | Timestamp, SkipCount | | EthernetCTPReply | 4 | Timestamp, Function, ReceiptNumber, Data | | LinkLayerDiscoveryInfo | 8 | Timestamp, PortDescription, SysName, SysDescription, SysCapabilities, MgmtAddress, OrgTLVs, Unknown | | IPSecAH | 7 | Timestamp, Reserved, SPI, Seq, AuthenticationData, SrcIP, DstIP | | IPSecESP | 6 | Timestamp, SPI, Seq, LenEncrypted, SrcIP, DstIP | | Geneve | 12 | Timestamp, Version, OptionsLength, OAMPacket, CriticalOption, Protocol, VNI, Options, SrcIP, DstIP, SrcPort, DstPort | | IPv6Fragment | 9 | Timestamp, NextHeader, Reserved1, FragmentOffset, Reserved2, MoreFragments, Identification, SrcIP, DstIP | | VXLAN | 9 | Timestamp, ValidIDFlag, VNI, GBPExtension, GBPDontLearn, GBPApplied, GBPGroupPolicyID, SrcIP, DstIP | | USB | 20 | Timestamp, ID, EventType, TransferType, Direction, EndpointNumber, DeviceAddress, BusID, TimestampSec, TimestampUsec, Setup, Data, Status, UrbLength, UrbDataLength, UrbInterval, UrbStartFrame, UrbCopyOfTransferFlags, IsoNumDesc, Payload | | LCM | 13 | Timestamp, Magic, SequenceNumber, PayloadSize, FragmentOffset, FragmentNumber, TotalFragments, ChannelName, Fragmented, SrcIP, DstIP, SrcPort, DstPort | | MPLS | 7 | Timestamp, Label, TrafficClass, StackBottom, TTL, SrcIP, DstIP | | Modbus | 12 | Timestamp, TransactionID, ProtocolID, Length, UnitID, Payload, Exception, FunctionCode, SrcIP, DstIP, SrcPort, DstPort | | OSPF | 16 | Timestamp, Version, Type, PacketLength, RouterID, AreaID, Checksum, AuType, Authentication, LSAs, LSU, LSR, DbDesc, HelloV2, SrcIP, DstIP | | OSPF | 16 | Timestamp, Version, Type, PacketLength, RouterID, AreaID, Checksum, Instance, Reserved, Hello, DbDesc, LSR, LSU, LSAs, SrcIP, DstIP | | BFD | 21 | Timestamp, Version, Diagnostic, State, Poll, Final, ControlPlaneIndependent, AuthPresent, Demand, Multipoint, DetectMultiplier, MyDiscriminator, YourDiscriminator, DesiredMinTxInterval, RequiredMinRxInterval, RequiredMinEchoRxInterval, AuthHeader, SrcIP, DstIP, SrcPort, DstPort | | GRE | 21 | Timestamp, ChecksumPresent, RoutingPresent, KeyPresent, SeqPresent, StrictSourceRoute, AckPresent, RecursionControl, Flags, Version, Protocol, Checksum, Offset, Key, Seq, Ack, Routing, SrcIP, DstIP, SrcPort, DstPort | | FDDI | 5 | Timestamp, FrameControl, Priority, SrcMAC, DstMAC | | EAP | 6 | Timestamp, Code, Id, Length, Type, TypeData | | VRRP | 12 | Timestamp, Version, Type, VirtualRtrID, Priority, CountIPAddr, AuthType, AdverInt, Checksum, IPAdresses, SrcIP, DstIP | | EAPOL | 4 | Timestamp, Version, Type, Length | | EAPOLKey | 22 | Timestamp, KeyDescriptorType, KeyDescriptorVersion, KeyType, KeyIndex, Install, KeyACK, KeyMIC, Secure, MICError, Request, HasEncryptedKeyData, SMKMessage, KeyLength, ReplayCounter, Nonce, IV, RSC, ID, MIC, KeyDataLength, EncryptedKeyData | | CiscoDiscovery | 5 | Timestamp, Version, TTL, Checksum, Values | | CiscoDiscoveryInfo | 27 | Timestamp, CDPHello, DeviceID, Addresses, PortID, Capabilities, Version, Platform, IPPrefixes, VTPDomain, NativeVLAN, FullDuplex, VLANReply, VLANQuery, PowerConsumption, MTU, ExtendedTrust, UntrustedCOS, SysName, SysOID, MgmtAddresses, Location, PowerRequest, PowerAvailable, SparePairPoe, EnergyWise, Unknown | | USBRequestBlockSetup | 6 | Timestamp, RequestType, Request, Value, Index, Length | | NortelDiscovery | 7 | Timestamp, IPAddress, SegmentID, Chassis, Backplane, State, NumLinks | | CIP | 12 | Timestamp, Response, ServiceID, ClassID, InstanceID, Status, AdditionalStatus, Data, SrcIP, DstIP, SrcPort, DstPort | | Ethernet/IP | 12 | Timestamp, Command, Length, SessionHandle, Status, SenderContext, Options, CommandSpecific, SrcIP, DstIP, SrcPort, DstPort | | SMTP | 9 | Timestamp, IsEncrypted, IsResponse, ResponseLines, Command, SrcIP, DstIP, SrcPort, DstPort | | FTP | 18 | Timestamp, SrcIP, DstIP, SrcPort, DstPort, IsResponse, Command, Argument, ResponseCode, ResponseMessage, Filename, TransferMode, DataConnectionMode, DataIP, DataPort, Username, IsControl, FileSize | | IRC | 18 | Timestamp, SrcIP, DstIP, SrcPort, DstPort, Prefix, Command, Parameters, Message, IsDCC, DCCType, DCCFilename, DCCIP, DCCPort, DCCFilesize, Channel, Nick, IsDataChannel | | SMB | 26 | Timestamp, SrcIP, DstIP, SrcPort, DstPort, Version, Command, CommandName, Status, Flags, Flags2, Username, Domain, ShareName, Filename, FileID, Action, BytesTransferred, FileSize, Offset, SessionID, TreeID, IsEncrypted, DialectRevision, ClientGUID, Capabilities | | IMAP | 23 | Timestamp, SrcIP, DstIP, SrcPort, DstPort, IsResponse, Tag, Command, Arguments, Response, ResponseText, Username, AuthMethod, Mailbox, MessageCount, RecentCount, UIDNext, UIDValidity, MessageID, UID, Flags, STARTTLSRequested, STARTTLSSuccess | | Diameter | 13 | Timestamp, Version, Flags, MessageLen, CommandCode, ApplicationID, HopByHopID, EndToEndID, AVPs, SrcIP, DstIP, SrcPort, DstPort | # Specification The Netcap audit record format *Netcap* files have the file extension **.ncap** or **.ncap.gz** if compressed with gzip and contain serialized protocol buffers of one type. Naming of each file happens according to the naming in the [gopacket](https://godoc.org/github.com/google/gopacket) library: a short uppercase letter representation for common protocols, and a camel case version full word version for less common protocols. Audit records are modeled as protocol buffers. Each file contains a header that specifies which type of audit records is inside the file, what version of *Netcap* was used to generate it, what input source was used and what time it was created. Each audit record should be tagged with the timestamp the packet was seen, in the format *seconds.microseconds*. Output is written to a file that represents each data structure from the protocol buffers definition, i.e. *TCP.ncap*, *UDP.ncap*. For this purpose, the audit records are written as length delimited records into the file. ## Delimited Protocol Buffer Records The data format on disk consists of gzipped length-delimited byte records. Each delimited Protocol Buffer record is preceded by a variable-length encoded integer (varint) that specifies the length of the serialized protocol buffer record in bytes. A stream consists of a sequence of such records packed consecutively without additional padding. There are no checksums or compression involved in this processing step. ![Delimited protocol buffers](https://github.com/dreadl0ck/netcap/tree/767852a00d76fcf7c921a4f3830ae6cec0162481/docs/.gitbook/assets/netcap-delimited%20%281%29.svg) ## Data Compression Encoding the output as protocol buffers does not help much with reducing the size, compared to the CSV format. To further reduce the disk size required for storage, the data is gzipped prior to writing it into the file. This makes the resulting files around 70% smaller. Gzip is a common and well supported format, support for decoding it exists in almost every programming language. If this is not desired for e.g. direct access to the stored data, this can be toggled with the **-comp** command-line flag. ## Audit Records A piece of information produced by Netcap is called an audit record. Audit records are type safe structured data, encoded as protocol buffers. An audit record can describe a specific protocol, or other abstractions built on top of observations from the analyzed traffic. Netcap does currently not enforce the presence of any special fields for each audit record, however by convention each audit record should have a timestamp with microsecond precision. A record file contains a header followed by a list of length-delimited serialized audit records. Naming of the audit record file happens according to the decoder name and should signal whether the file contents are compressed by adding the .gz extension. ![](/files/ooN7iLRvYnyuBEULnmJ9) # Installation Setup instructions ## Binary Distributions Compiled versions for macOS, Linux and Windows are available on GitHub: {% embed url="" %} ## Go Get Installation via go get: ``` $ go get -u github.com/dreadl0ck/netcap/... ``` ## Manual Build ``` $ go build -ldflags "-s -w" -o /usr/local/bin/net github.com/dreadl0ck/netcap/cmd ``` ## Reproducible Builds via Go Modules In order to provide stable and reproducible builds, Go modules are used to pin the versions of source code dependencies to specific versions. Go has included support for versioned modules as proposed [here](https://golang.org/design/24301-versioned-go) since `1.11`. The initial prototype `vgo` was [announced](https://research.swtch.com/vgo) in February 2018. In July 2018, versioned modules [landed](https://groups.google.com/d/msg/golang-dev/a5PqQuBljF4/61QK4JdtBgAJ) in the main Go repository. They are used by default by the go toolchain starting from version `1.13` . You can read about Go modules here: {% embed url="" %} {% embed url="" %} ## Development Build To install the command-line tool: ``` $ go build -o /usr/local/bin/net github.com/dreadl0ck/netcap/cmd ``` ## Cross Compilation To cross compile for other architectures, set the *GOARCH* and *GOOS* environment variables. For example to cross compile a binary for *linux amd64*: ``` $ GOARCH=amd64 GOOS=linux go build -o bin/net github.com/dreadl0ck/netcap/cmd ``` ## Homebrew On macOS, you can install the *netcap* command-line tool with Homebrew: ``` $ brew tap dreadl0ck/formulas $ brew install netcap ``` ## Buildsystem *Netcap* uses the [zeus](https://github.com/dreadl0ck/zeus) build system, it can be found on GitHub along with installation instructions: {% embed url="" %} To install the *Netcap* and *Netlabel* command-line tool and the library with zeus, run: ``` $ zeus install ``` # Quickstart For those who can't wait to get their hands dirty. ## Capture traffic to create audit records Read traffic live from interface, stop with *Ctrl-C* (*SIGINT*): ``` $ net capture -iface eth0 ``` Read traffic from a dump file (supports PCAP or PCAPNG): ``` $ net capture -read traffic.pcap ``` ## Read audit records Read a netcap dumpfile and print to stdout as CSV: ``` $ net dump -read TCP.ncap.gz ``` Show the available fields for a specific Netcap dump file: ``` $ net dump -fields -read TCP.ncap.gz ``` Print only selected fields and output as CSV: ``` $ net dump -read TCP.ncap.gz -select Timestamp,SrcPort,DstPort ``` Save CSV output to file: ``` $ net dump -read TCP.ncap.gz -select Timestamp,SrcPort,DstPort > tcp.csv ``` Print output separated with tabs: ``` $ net dump -read TPC.ncap.gz -tsv ``` Run with 24 workers and disable gzip compression and buffering: ``` $ net capture -workers 24 -buf false -comp false -read traffic.pcapng ``` Parse pcap and write all data to output directory (will be created if it does not exist): ``` $ net capture -read traffic.pcap -out traffic_ncap ``` Convert timestamps to UTC: ``` $ net dump -read TCP.ncap.gz -select Timestamp,SrcPort,Dstport -utc ``` ## Show Audit Record File Header To display the header of the supplied audit record file, the -header flag can be used: ``` $ net capture -read TCP.ncap.gz -header +----------+---------------------------------------+ | Field | Value | +----------+---------------------------------------+ | Created | 2018-11-15 04:42:22.411785 +0000 UTC | | Source | Wednesday-WorkingHours.pcap | | Version | v0.3.3 | | Type | NC_TCP | +----------+---------------------------------------+ ``` ## Print Structured Audit Records Audit records can be printed structured, this makes use of the *proto.MarshalTextString()* function. This is sometimes useful for debugging, but very verbose. ``` $ net dump -read TCP.ncap.gz -struc ... NC_TCP Timestamp: "1499255023.848884" SrcPort: 80 DstPort: 49472 SeqNum: 1959843981 AckNum: 3666268230 DataOffset: 5 ACK: true Window: 1025 Checksum: 2348 PayloadEntropy: 7.836586993143013 PayloadSize: 1460 ... ``` ## Print as CSV This is the default behavior. First line contains all field names. ``` $ net dump -read TCP.ncap.gz Timestamp,SrcPort,DstPort,SeqNum,AckNum,DataOffset,FIN,SYN,RST,PSH,ACK,URG,... 1499254962.234259,443,49461,1185870107,2940396492,5,false,false,false,true,true,false,... 1499254962.282063,49461,443,2940396492,1185870976,5,false,false,false,false,true,false,... ... ``` ## Print as Tab Separated Values To use a tab as separator, the *-tsv* flag can be supplied: ``` $ net dump -read TCP.ncap.gz -tsv Timestamp SrcPort DstPort Length Checksum PayloadEntropy PayloadSize 1499254962.084372 49792 1900 145 34831 5.19616448 137 1499254962.084377 49792 1900 145 34831 5.19616448 137 1499254962.084378 49792 1900 145 34831 5.19616448 137 1499254962.084379 49792 1900 145 34831 5.19616448 137 ... ``` ## Print as Table The *-table* flag can be used to print output as a table. Every 100 entries the table is printed to stdout. ``` $ net dump -read UDP.ncap.gz -table -select Timestamp,SrcPort,DstPort,Length,Checksum +--------------------+----------+----------+---------+-----------+ | Timestamp | SrcPort | DstPort | Length | Checksum | +--------------------+----------+----------+---------+-----------+ | 1499255691.722212 | 62109 | 53 | 43 | 38025 | | 1499255691.722216 | 62109 | 53 | 43 | 38025 | | 1499255691.722363 | 53 | 62109 | 59 | 37492 | | 1499255691.722366 | 53 | 62109 | 59 | 37492 | | 1499255691.723146 | 56977 | 53 | 43 | 7337 | | 1499255691.723149 | 56977 | 53 | 43 | 7337 | | 1499255691.723283 | 53 | 56977 | 59 | 6804 | | 1499255691.723286 | 53 | 56977 | 59 | 6804 | | 1499255691.723531 | 63427 | 53 | 43 | 17441 | | 1499255691.723534 | 63427 | 53 | 43 | 17441 | | 1499255691.723682 | 53 | 63427 | 87 | 14671 | ... ``` ## Print with Custom Separator Output can also be generated with a custom separator: ``` $ net dump -read TCP.ncap.gz -sep ";" Timestamp;SrcPort;DstPort;Length;Checksum;PayloadEntropy;PayloadSize 1499254962.084372;49792;1900;145;34831;5.19616448;137 1499254962.084377;49792;1900;145;34831;5.19616448;137 1499254962.084378;49792;1900;145;34831;5.19616448;137 ... ``` ## Validate generated CSV output To ensure values in the generated CSV would not contain the separator string, the *-check* flag can be used. This will determine the expected number of separators for the audit record type, and print all lines to stdout that do not have the expected number of separator symbols. The separator symbol will be colored red with ansi escape sequences and each line is followed by the number of separators in red color. The *-sep* flag can be used to specify a custom separator. ``` $ net util -read TCP.ncap.gz -check $ net util -read TCP.ncap.gz -check -sep=";" ``` # Configuration Adjusting framework parameters ## Command-line Flags Each subcommand has a dedicated set of flags for configuration. List the flag names, a short description and their default values with: ``` $ net -h ``` ## Environment All default values for flags can be overriden via environment variables, by using the flag name and prefixing it with "NC\_", for example lets overwrite the **-read** flag from net capture: ``` $ NC_READ=/home/user/traffic.pcap net capture ``` Since the provide the value via the environment, passing it via flag is no longer necessary. This is generally useful to enable or disable features globally on your system. ## Configuration File Additionally, the configuration can be provided as a config file via the **-config** flag. To retrieve a sane default configuration for the subcommand you want to execute, use the **-gen-config** flag and redirect the output into a file: ``` $ net capture -gen-config > capture.conf ``` The config file will look something like this, using the **name value** syntax to set values: ```bash ... # toggle promiscous mode for live capture promisc true # don't print infos to stdout quiet false # reassemble TCP connections reassemble-connections true # resolve ips to domains via the operating systems default dns resolver reverse-dns false # use serviceDB for device profiling serviceDB false # configure snaplen for live capture from interface snaplen 1514 # print netcap package version and exit version false # wait for all connections to finish processing before cleanup wait-conns true # number of workers workers 12 # write incomplete response writeincomplete false ... ``` > Lines starting with # are treated as comments, blank lines are being ignored. Adjust the parameters of interest and pass the config file: ``` $ net capture -config capture.conf ``` ## Resolver Database The environment variable **NC\_CONFIG\_ROOT** can be used to overwrite the default path for the resolver databases **\~/.config/netcap/dbs**. Read more about the resolvers package here: {% content-ref url="/pages/-MZYVj3IwWYjp6yCQjOs" %} [Resolvers](/master/resolvers) {% endcontent-ref %} # Bash Completion Tab completion for the shell ## Installation Completions for the command-line is provided via the bash-completion package which is available for most linux distros and macOS. On macOS you can install it with brew: ``` $ brew install bash-completion ``` on linux use the package manager of your distro. Then add the completion file **cmd/net** to: * macOS: /usr/local/etc/bash\_completion.d/ * Linux: /etc/bash\_completion.d/ and source it with: * macOS: . /usr/local/etc/bash\_completion.d/net * Linux: . /etc/bash\_completion.d/net If you use zeus, simply execute the following in the project root to install the completion script: ``` $ zeus install-completions ``` or move and source the file manually from the project root: ``` $ cp cmd/net /usr/local/etc/bash_completion.d/net && . /usr/local/etc/bash_completion.d/net ``` Afterwards you should receive predictions when hitting tab in the shell, for subcommands and flags. For flags that expect a path on the filesystem, path completion is available and will only display files with the expected datatype (based on the file extension). To use completion with **zsh** run the following: ``` autoload -U +X compinit && compinit autoload -U +X bashcompinit && bashcompinit cp cmd/net /usr/local/etc/bash_completion.d/net && . /usr/local/etc/bash_completion.d/net ``` # Packet Collection # Audit Record Labeling # HTTP Proxy Inspect traffic to web applications with a HTTP reverse proxy ## Motivation The **proxy** tool allows to quickly spin up monitoring of web applications and retrieving netcap audit records. Since currently, TCP stream reassembly is only supported for IPv4, netcap misses HTTP traffic over IPv6 when decoding traffic from raw packets. Also there is currently no support implemented for decoding HTTP2 over TCP or QUIC. By using a simple reverse proxy for HTTP traffic, the operating system handles the stream reassembly and we can make sure no IPv6 and / or HTTP2 traffic is missed. ## Usage Spin up a single proxy instance from the commandline: `$ net proxy -local 127.0.0.1:4000 -remote http://google.com` Specifiy a custom config file for proxying multiple services with the **-proxy-config** flag: ``` $ net proxy -proxy-config example_config.yml ``` The default config path is **net.proxy-config.yml**, so if this file exists in the folder where you execute the proxy, you do not need to specify it on the commandline. ## Configuration For proxying several services, you need to provide a config file, here is a simple example: ```yaml # Proxies map holds all reverse proxies proxies: service1: local: 127.0.0.1:443 remote: http://127.0.0.1:8080 tls: true service2: local: 127.0.0.1:9999 remote: http://192.168.1.20 service3: local: 127.0.0.1:7000 remote: https://google.com # CertFile for TLS secured connections certFile: "certs/cert.crt" # KeyFile for TLS secured connections keyFile: "certs/cert.key" # Logdir is used as destination for the logfile logdir: "logs" ``` ## Help ```erlang Usage of net proxy: -version bool print netcap package version and exit -config string set config file path (default "net.proxy-config.yml") -debug set debug mode -dialTimeout int seconds until dialing to the backend times out (default 30) -idleConnTimeout int seconds until a connection times out (default 90) -local string set local endpoint -maxIdle int maximum number of idle connections (default 120) -remote string set remote endpoint -skipTlsVerify skip TLS verification -tlsTimeout int seconds until a TLS handshake times out (default 15) ``` # USB Capture Capture traffic sent via Universal Serial Bus (USB) protocol ## Live Capture USB live capture is now possible, currently the following Audit Records exist: USB and USBRequestBlockSetup. To capture USB traffic live on macOS, install wireshark and bring up the USB interface: ``` $ sudo ifconfig XHC20 up ``` Now attach netcap and set baselayer to USB: ``` $ net.cap -iface XHC20 -base usb ``` ## Offline from dumpfile To read offline USB traffic from a PCAP file use: ``` $ net.cap -r usb.pcap -base usb ``` Don't forget to set the **-payload** flag if you want to preserve the data being transmitted! ## Audit Records The **USB** and **USBRequestBlockSetup** audit records contain the following fields: ```erlang message USB { string Timestamp = 1; uint64 ID = 2; int32 EventType = 3; int32 TransferType = 4; int32 Direction = 5; int32 EndpointNumber = 6; int32 DeviceAddress = 7; int32 BusID = 8; int64 TimestampSec = 9; int32 TimestampUsec = 10; bool Setup = 11; bool Data = 12; int32 Status = 13; uint32 UrbLength = 14; uint32 UrbDataLength = 15; uint32 UrbInterval = 16; uint32 UrbStartFrame = 17; uint32 UrbCopyOfTransferFlags = 18; uint32 IsoNumDesc = 19; bytes Payload = 20; } message USBRequestBlockSetup { string Timestamp = 1; int32 RequestType = 2; int32 Request = 3; int32 Value = 4; int32 Index = 5; int32 Length = 6; } ``` � # Payload Capture Capture full packet payloads It is now possible to capture payload data for the following protocols: **TCP, UDP, ModbusTCP, USB** This can be enabled with the **-payload** flag: ``` $ net capture -read traffic.pcap -payload ``` Setting the flag works for both live and offlline capture, afterwards the raw payload bytes are stored in the **Payload** field of the audit records. You can use the **-struc** flag with the **dump** tool to see the payload in the command-line: ``` $ net dump -read TCP.ncap.gz -struc ``` # Distributed Collection Sensors and Collection Server ## Collection Server Using Netcap as a data collection mechanism, sensor agents can be deployed to export the traffic they see to a central collection server. This is especially interesting for internet of things (IoT) applications, since these devices are placed inside isolated networks and thus the operator does not have any information about the traffic the device sees. Although Go was not specifically designed for this application, it is an interesting language for embedded systems. Each binary contains the complete runtime, which increases the binary size but requires no installation of dependencies on the device itself. Data exporting currently takes place in batches over UDP sockets. Transferred data is compressed in transit and encrypted with the public key of the collection server. Asymmetric encryption was chosen, to avoid empowering an attacker who compromised a sensor, to decrypt traffic of all sensors communicating with the collection server. To increase the performance, in the future this could be replaced with using a symmetric cipher, together with a solid concept for key rotation and distribution. Sensor agents do not write any data to disk and instead keep it in memory before exporting it. ![](/files/cWCssq15tSA0qkrnm9FJ) As described in the concept chapter, sensors and the collection server use UDP datagrams for communication. Network communication was implemented using the go standard library. This section will focus on the procedure of encrypting the communication between sensor and collector. For encryption and decryption, cryptographic primitives from the [golang.org/x/crypto/nacl/box](https://godoc.org/golang.org/x/crypto/nacl/box) package are used. The NaCl (pronounced 'Salt') toolkit was developed by the reowned cryptographer Daniel J. Bernstein. The box package uses *Curve25519*, *XSalsa20* and *Poly1305* to encrypt and authenticate messages. It is important to note that the length of messages is not hidden. Netcap uses a thin wrapper around the functionality provided by the nacl package, the wrapper has been published here: [github.com/dreadl0ck/cryptoutils](https://www.github.com/dreadl0ck/cryptoutils). ## Batch Encryption The collection server generates a keypair, consisting of two 32 byte (256bit) keys, hex encodes them and writes the keys to disk. The created files are named *pub.key* and *priv.key*. Now, the servers public key can be shared with sensors. Each sensor also needs to generate a keypair, in order to encrypt messages to the collection server with their private key and the public key of the server. To allow the server to decrypt and authenticate the message, the sensor prepends its own public key to each message. ![NETCAP batch encryption](/files/LxA1Asgk58N9Gcr9Uxjt) ## Batch Decryption When receiving an encrypted batch from a sensor, the server needs to trim off the first 32 bytes, to get the public key of the sensor. Now the message can be decrypted, and decompressed. The resulting bytes are serialized data for a batch protocol buffer. After unmarshalling them into the batch structure, the server can append the serialized audit records carried by the batch, into the corresponding audit record file for the provided client identifier. ![](/files/A38Uk34fvnNsOU68dica) ## Usage Both sensor and client can be configured by using the *-addr* flag to specify an IP address and port. To generate a keypair for the server, the *-gen-keypair* flag must be used: ``` $ net collect -gen-keypair wrote keys $ ls priv.key pub.key ``` Now, the server can be started, the location of the file containing the private key must be supplied: ```bash $ net collect -privkey priv.key -addr 127.0.0.1:4200 ``` The server will now be listening for incoming messages. Next, the sensor must be configured. The keypair for the sensor will be generated on startup, but the public key of the server must be provided: ``` $ net agent -pubkey pub.key -addr 127.0.0.1:4200 got 126 bytes of type NC_ICMPv6RouterAdvertisement expected [126] got size [73] for type NC_Ethernet got 73 bytes of type NC_Ethernet expected [73] got size [27] for type NC_ICMPv6 got size [126] for type NC_ICMPv6RouterAdvertisement got 126 bytes of type NC_ICMPv6RouterAdvertisement expected [126] got size [75] for type NC_IPv6 got 75 bytes of type NC_IPv6 expected [75] got 27 bytes of type NC_ICMPv6 expected [27] ``` The client will now collect the traffic live from the specified interface, and send it to the configured server, once a batch for an audit record type is complete. The server will log all received messages: ``` $ net collect -privkey priv.key -addr 127.0.0.1:4200 packet-received: bytes=2412 from=127.0.0.1:57368 decoded batch NC_Ethernet from client xyz new file xyz/Ethernet.ncap packet-received: bytes=2701 from=127.0.0.1:65050 decoded batch NC_IPv4 from client xyz new file xyz/IPv4.ncap ... ``` When stopping the server with a *SIGINT* (Ctrl-C), all audit record file handles will be flushed and closed properly. The agent uses the **$USER** environment variable to identify the workstation where the audit records are created. This will be replaced with a unique identifier in a future release. # Workers This is where the magic happens ## Introduction To make use of multi-core processors, processing of packets should happen in an asynchronous way. Since Netcap should be usable on a stream of packets, fetching of packets has to happen sequentially, but decoding them can be parallelized. The packets read from the input data source (PCAP file or network interface) are assigned to a configurable number of workers routines via round-robin. Each of those worker routines operates independently, and has all selected decoders loaded. It decodes all desired layers of the packet, and writes the encoded data into a buffer that will be flushed to disk after reaching its capacity. ## Worker [Workers](https://github.com/dreadl0ck/netcap/blob/master/collector/worker.go) are a core concept of *Netcap*, as they handle the actual task of decoding each packet. *Netcap* can be configured to run with the desired amount of workers, the default is 1000, since this configuration has shown the best results on the development machine. Increasing the number of workers also increases the number of runtime operations for goroutine scheduling, thus performance might decrease with a huge amount of workers. It is recommended to experiment with different configurations on the target system, and choose the one that performs best. Packet data fetched from the input source is distributed to a worker pool for decoding in round robin style. Each worker decodes all layers of a packet and calls all available custom decoders. After decoding of each layer, the generated protocol buffer instance is written into the *Netcap* data pipe. Packets that produced an error in the decoding phase or carry an unknown protocol are being written in the corresponding logs and dumpfiles. > Note: by default the number of workers is set to the numbers of cores of your machine! You can use the **-workers** flag to overwrite this value. ![NETCAP worker](https://github.com/dreadl0ck/netcap/tree/767852a00d76fcf7c921a4f3830ae6cec0162481/docs/.gitbook/assets/netcap-worker%20%281%29.svg) ## Buffering Each worker receives its data from an input channel. This channel can be buffered, by default the buffer size is 100, also because this configuration has shown the best results on the development machine. When the buffer size is set to zero, the operation of writing a packet into the channel blocks, until the goroutine behind it is ready for consumption. That means, the goroutine must finish the currently processed packet, until a new packet can be accepted. By configuring the buffer size for all routines to a specific number of packets, distributing packets among workers can continue even if a worker is not finished yet when new data arrives. New packets will be queued in the channel buffer, and writing in the channels will only block if the buffer is full. ## Data Pipe The Netcap data pipe describes the way from a network packet that has been processed in a worker routine, to a serialized, delimited and compressed record into a file on disk. ![](/files/hYiBgw62RgwjX4XphTm6) # Filtering and Export Process Netcap audit records and extract the data you are interested in ## Exporting Data with net dump Netcap offers a simple interface to filter for specific fields and select only those of interest. Filtering and exporting specific fields can be performed with all available audit record types, over a uniform command-line interface. By default, output is generated as CSV with the field names added as first line. It is also possible to use a custom separator string. Fields are exported in the order they are named in the select statement. Sub structures of audit records (for example IPv4Options from an IPv4 packet), are converted to a human readable string representation. More examples for using this feature on the command-line can be found in the usage section. ![](/files/-M4zf-fqn5vKXXSv1SAY) Netcap offers a simple command-line interface to select fields of interest from the gathered audit records. ## Examples Show available header fields: ``` $ net dump -read UDP.ncap.gz -fields Timestamp,SrcPort,DstPort,Length,Checksum,PayloadEntropy,PayloadSize ``` Print all fields for the supplied audit record: ``` $ net dump -read UDP.ncap.gz 1331904607.100000,53,42665,120,41265,4.863994469989251,112 1331904607.100000,42665,53,53,1764,4.0625550894074385,45 1331904607.290000,51190,53,39,22601,3.1861758166070766,31 1331904607.290000,56434,53,39,37381,3.290856864924384,31 1331904607.330000,137,137,58,64220,3.0267194361875682,50 ... ``` Selecting fields will also define their order: ``` $ net dump -read UDP.ncap.gz -select Length,SrcPort,DstPort,Timestamp Length,SrcPort,DstPort,Timestamp 145,49792,1900,1499254962.084372 145,49792,1900,1499254962.084377 145,49792,1900,1499254962.084378 145,49792,1900,1499254962.084379 145,49792,1900,1499254962.084380 ... ``` Print selection in the supplied order and convert timestamps to UTC time: ``` $ net dump -read UDP.ncap.gz -select Timestamp,SrcPort,DstPort,Length -utc 2012-03-16 13:30:07.1 +0000 UTC,53,42665,120 2012-03-16 13:30:07.1 +0000 UTC,42665,53,53 2012-03-16 13:30:07.29 +0000 UTC,51190,53,39 2012-03-16 13:30:07.29 +0000 UTC,56434,53,39 2012-03-16 13:30:07.33 +0000 UTC,137,137,58 ... ``` To save the output into a new file, simply redirect the standard output: ``` $ net dump -read UDP.ncap.gz -select Timestamp,SrcPort,DstPort,Length -utc > UDP.csv ``` # Data Compression Save storage space To reduce the amount of disk space used for storing the audit records, netcap compresses them by default with **gzip**. Compressed files have the extension **.ncap.gz.** For this purpose Netcap currently uses the following gzip implementation: {% embed url="" %} This implementation will split the data into blocks that are compressed in parallel, which can be useful for compressing big amounts of data. The output is a standard gzip file. The gzip decompression is modified so it decompresses ahead of the current reader. This means that reads will be non-blocking and CRC calculation also takes place in a separate goroutine. This design implements input buffering to the compressor which has a nice performance effect: writes to the compressor only block if the compressor is already compressing the number of blocks specified. This reduces waiting time for the workers which they can instead use to decode packets. To get any performance gains, you should at least be compressing more than 1 megabyte of data at the time. You should at least have a block size of 100k and at least a number of blocks that match the number of cores you would like to utilize, but about twice the number of blocks would be the best. The default configuration uses 1MB block size and 2x NumCPUs as the number of blocks. Netcap only uses the parallel gzip implementation for reading and writing audit records, as only there the required amounts of data are reached to allow a speedup. For tasks where the data size can vary heavily, such as decompressing HTTP requests and responses, the standard library **compress/gzip** is used instead. # Internals Framework inner workings and Implementation details ## Packages You can browse the source and sub packages on GoDev: ### cmd The cmd package contains the command-line application. It receives configuration parameters from command-line flags, creates and configures a collector instance, and then starts collecting data from the desired source. #### label [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/l](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector)abel The label package contains the code for creating labeled datasets. For now, the suricata IDS / IPS engine is used to scan the input PCAP and generate alerts. In the future, support could also be added for using YARA. Alerts are then parsed with regular expressions and transformed into the **label.SuricataAlert** type. This could also be replaced by parsing suricatas eve.json event logs in upcoming versions. A suricata alert contains the following information: ```go // SuricataAlert is a summary structure of an alerts contents type SuricataAlert struct { Timestamp string Proto string SrcIP string SrcPort int DstIP string DstPort int Classification string Description string } ``` In the next iteration, the gathered alerts are mapped onto the collected data. For layer types which are not handled separately, this is currently by using solely the timestamp of the packet, since this is the only field required by Netcap, however multiple alerts might exist for the same timestamp. To detect this and throw an error, the **-strict** flag can be used. The default is to ignore duplicate alerts for the same timestamp, use the first encountered label and ignore the rest. Another option is to collect all labels that match the timestamp, and append them to the final label with the **-collect** flag. To allow filtering out classifications that shall be excluded, the **-excluded** flag can be used. Alerts matching the excluded classi- fication will then be ignored when collecting the generated alerts. Flow, Connection, HTTP and TLS records mapping logic also takes source and destination information into consider- ation. The created output files follow the naming convention: **\\_labeled.csv**. ### types [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/t](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector)ypes The types package contains types.AuditRecord interface implementations for each supported protocol, to enable converting data to the CSV format. For this purpose, each protocol must provide a CSVRecord() \[]string and a CSVHeader() \[]string function. Additionally, a NetcapTimestamp() string function that returns the Netcap timestamp must be implemented. ### decoder [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/e](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector)ncoder The decoder package implements conversion of decoded network protocols to protocol buffers. This has to be defined for each supported protocol. Two types of decoders exist: The LayerEncoder and the CustomEncoder. #### GoPacket Decoder A GoPacketDecoder operates on a gopacket.Layer and has to provide the gopacket.LayerType constant, as well a handler function to receive the layer and the timestamp and convert it into a protocol buffer. #### Custom Decoder A CustomDecoder operates on a gopacket.Packet and is used to decode traffic into abstractions such as Flows or Connections. To create it a name has to be supplied among three different handler functions to control initialization, decoding and deinitialization. Its handler function receives a gopacket.Packet interface type and returns a proto.Message. The postinit function is called after the initial initialization has taken place, the deinit function is used to teardown any additionally created structures for a clean exit. Both functions are optional and can be omitted by supplying nil as value. ### resolvers [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/resolvers](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/resolvers) Resolvers for lookup of various external information, such as geolocation, domain names, hardware addresses, port numbers etc ### dpi [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/dpi](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/dpi) Deep Packet Inspection integration, using a fork of **mushorg/go-dpi** that was extended to identify the full range of protocols offered by **nDPI** and **libprotoident**. Both libraries are loaded dynamically at runtime and is invoked via C bindings. The fork can be found here: {% embed url="" %} ### delimited [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/delimited](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/delimited) Primitives for reading and writing length delimited binary data ### utils [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/utils](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/utils) The utils package contains shared utility functions used by several other packages. ### collector [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/collector](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/collector) The collector package provides an interface for fetching packets from a data source, this can either be a PCAP / PCAPNG file or directly from a named network interface. It is used to implement the command-line interface for Netcap. {% hint style="info" %} Warning: Do not use multiple instances of a collector in parallel! This is not supported yet. Once it is possible, this warning will be removed. {% endhint %} ### io [https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.5/io](https://pkg.go.dev/github.com/dreadl0ck/netcap@v0.4.7/io) Primitives for atomic maps and write operations ## Caveats Protocol buffers have a few caveats that developers and researchers should be aware of. First, there are no types for 16 bit signed (int16) and unsigned (uint16) integers in protobuf, also there is no type for unsigned 8 bit integers (uint8). This data type is seen a lot in network protocols, so the question arises how to represent it in protocol buffers. The non-fixed integer types use variable length encoding, so int32 is used instead. The variable-length encoding will take care of not sending the bytes that are not being used. Unfortunately, the mu type is too short for this purpose. Second, protocol buffers require all strings to be encoded as valid UTF-8, otherwise encoding to proto will fail. This means all input data that will be encoded as a string in protobuf must be checked to contain valid UTF-8, or they will create an error upon serialization and end up in the errors.pcap file. If this behavior is not desired strings must be filtered prior to setting them on the protocol buffer instances. Another thing that has to be kept in mind is that Netcap processes packets in parallel, thus the order in which packets are written to the dump file is not guaranteed. In experiments, no mixup was detected, and records were tracked in the correct order. However, under heavy load conditions or with a high number of workers, this might be different. Because of this caveat, the Netcap specification requires each record to preserve the timestamp, in order to allow sorting the packets afterwards, if required. ## Data Race Detection Builds In concurrent programming, shared resources need to be synchronized, in order to guarantee their state when modifying or reading them. If access is not synchronized, race conditions occur, which will lead to faulty program behavior. To avoid this and detect race conditions early in the development cycle, the go toolchain offers compiling the program with the race detector enabled. This will let the application crash with stack traces to assist the developer in debugging, if a data race occurs. Programs with active race detection are slower by the factor of 10 to 100. To compile a Go program with the race detection enabled the **-race** flag must be added to the compilation command. To compile a netcap binary with the race detection enabled use: ``` $ zeus install-race ``` # Metrics Prometheus Metrics ## Introduction Netcap now supports exporting prometheus metrics about its go runtime, the collection process and the audit records itself. These data points can be used to gain insights about the collection performance or discover security related events. [Prometheus](https://github.com/prometheus) is an open-source systems monitoring and alerting toolkit originally built at [SoundCloud](https://soundcloud.com/). Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user [community](https://prometheus.io/community). It is now a standalone open source project and maintained independently of any company. {% embed url="" %} To visualize the captured data I recomment the open source analytics and monitoring solution Grafana: {% embed url="" %} This feature can be used with the **export** tool, which behaves similar to **capture** but is able to operate on pcaps, audit records and network interfaces. ## Configuration Metrics are served by default on [**127.0.0.1:7777/metrics**](http://127.0.0.1:7777/metrics). Configure a prometheus instance to scrape it: ```yaml # reference: https://prometheus.io/docs/prometheus/latest/configuration/configuration/ global: scrape_interval: 15s scrape_timeout: 15s #evaluation_interval: 15s scrape_configs: # process_ metrics - job_name: netcap metrics_path: /metrics scheme: http static_configs: - targets: - 127.0.0.1:7777 ``` {% hint style="info" %} Tip: The latest prometheus config documentation can be found at: [https://prometheus.io/docs/prometheus/latest/configuration/configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) {% endhint %} Run the export tool to capture live from an interface: ``` net export -iface en0 ``` {% hint style="info" %} Tip: Use `$ net capture -interfaces` to get a list of available interfaces to choose from {% endhint %} Go to or to the port you configured alternatively, to check if your prometheus instance is scraping data correctly. Now that we have some data at hands, lets use Grafana to visualize it! You can setup Grafana on macOS via brew: ``` $ brew install grafana ``` {% hint style="info" %} Tip: On macOS, Grafanas default config is at **/usr/local/etc/grafana/grafana.ini** and installed plugins are stored at **/usr/local/opt/grafana/share/grafana/data/plugins**. {% endhint %} Start the prometheus server and pass the previously created config: ``` $ prometheus --config.file prometheus/prometheus.yml ``` You need to install the pie chart plugin for grafana: ``` $ cd /usr/local/opt/grafana/share/grafana/data/plugins $ git clone https://github.com/grafana/piechart-panel.git --branch release-1.4.0 ``` Start the grafana server: ``` $ grafana-server --homepath /usr/local/opt/grafana/share/grafana ``` Now download the NETCAP Dashboard and import it into Grafana: {% file src="/files/zDNZhxlSJXmE3iBZcjL6" %} Go to **Settings > Datasources** and a prometheus datasource, either with the default port 9090 or the one you choose in the config. You should be good to go! ## Usage Export a PCAP dumpfile and serve metrics . ``` $ net export -read 2017-09-19-traffic-analysis-exercise.pcap ``` Capture and export traffic live from the named interface: ``` $ net export -iface en0 ``` Export a specific audit record file: ``` $ net export -read HTTP.ncap.gz ``` Export all audit record files in the current directory: ``` $ net export . ``` ## Overview Dashboard Preview ![Grafana Dashboard Overview](/files/-Le3wPN6-cckxppkzBkT) ## TCP Dashboard Preview ![Grafana Dashboard TCP](https://github.com/dreadl0ck/netcap/tree/767852a00d76fcf7c921a4f3830ae6cec0162481/docs/.gitbook/assets/screenshot-2019-05-04-at-23.39.41%20%281%29.png) ## HTTP Dashboard Preview ![Grafana Dashboard HTTP](https://github.com/dreadl0ck/netcap/tree/767852a00d76fcf7c921a4f3830ae6cec0162481/docs/.gitbook/assets/screenshot-2019-05-04-at-23.40.05%20%281%29.png) # Resolvers Lookup everything! ## Motivation Lots of information is not available on first sight, and we need to combine our data with knowledge from other data sources to make it easier to understand for humans. Think of resolving ip addresses to geolocations, hardware addreses to manufacturers, domains to ip addresses and vice versa, or simply identifying the service name associated with a given port number. Or consider filtering ip addresses or domain names against a whitelist, to eliminate known legitimate traffic. The resolvers package provides primitives for such tasks, and if possible, caches results in memory for better performance. ## Design External data sources are stored in a central directory on the system, which defaults to **\~/.config/netcap/dbs** but can be overridden using the **NC\_CONFIG\_ROOT** environment variable. Database files: * *domain-whitelist.csv* * *GeoLite2-City.mmdb* * *GeoLite2-ASN.mmdb* * *ja3fingerprint.json* * *macaddress.io-db.json* * *service-names-port-numbers.csv* * *ja3UserAgents.json* * *ja3erDB.json* ## Configuration By default, all resolvers are disabled. You need to use the **-reverse-dns**, **-local-dns**, **-macDB**, **-ja3DB**, **-serviceDB** and **-geoDB** to enable what you want to use, or configure it via environment variables or config file, as described in: {% content-ref url="/pages/-MZYVj35MsVGORqvARs5" %} [Configuration](/master/configuration) {% endcontent-ref %} ## Quickstart You can download a bundled version of all databases except for the MaxMind GeoLite, here: {% file src="/files/ca1IfaHX3lFYz9sO5Sk6" %} ## DNS Reverse DNS lookups can be used to identify the domains associated with an address. By default the standard system resolver will be contacted for this. ### Passive / Local DNS Passive DNS will read the hosts mapping from a file and load it into memory, instead of looking up encountered adresses by contacting a resolver. This can be used to provide names for known hosts in your network for example. To avoid producing lookups that leave the network, you can generate a hosts mapping based on the DNS traffic in your dumpfile using tshark: ``` $ tshark -r traffic.pcap -q -z hosts ``` And provide it to netcaps resolver via a **hosts** file in the database directory. ## Domain Whitelisting To filter known legitimate domains away, the alexa top 1 million can be used for example. {% embed url="" %} You can download the CSV file here: {% embed url="" %} Rename it to **domain-whitelist.csv** and move it into the database path: ``` $ mv top-1m.csv ~/.config/netcap/dbs/domain-whitelist.csv ``` ## Geolocation To determine the geolocation for a given host, the MaxMind GeoLite database is used. The lite database is freely available, but you have to register on their website to download it. {% embed url="" %} Geolocation lookups can provide the Country, City and ASN for a given IP address. Download the databases and move them into the database path. ## Vendor Identification To identify the vendor for a given MAC address, the **macaddress.io** JSON database is used. At the time of this writing it contains 39,041 tracked address blocks and 28,961 unique vendors. {% embed url="" %} ## Service Identification Resolving port numbers to service names is done according to the CSV mapping from IANA, which contains 6104 records for TCP and UDP services at the time of this writing: {% embed url="" %} ## TLS Fingerprints To identify hosts that use TLS connections, the Ja3 fingerprint database from **Trisul** is used: {% embed url="" %} For more fingerprints, you can load other databases additionally. For example from **ja3erDB**: {% embed url="" %} # TLS Fingerprinting Identify client and server that are using encrypted connections ## TLS Audit Records Watch a quick demo of creating and exploring the **TLSClientHello** audit records on the command-line {% embed url="" %} ## JA4 Fingerprinting JA4 is the successor to JA3, developed by FoxIO-LLC. It addresses JA3's limitations with modern browsers that randomize TLS extension order (like Chrome), and adds QUIC protocol support. Unlike JA3's MD5 hash, JA4 fingerprints are human-readable. ### JA4 Format JA4 consists of three parts separated by underscores: ``` {ja4_a}_{ja4_b}_{ja4_c} ``` **JA4\_a** (10 characters): `{protocol}{version}{sni}{cipher_count:2d}{ext_count:2d}{alpn_first}` * `protocol`: `t` for TCP/TLS, `q` for QUIC * `version`: `13` (TLS 1.3), `12` (TLS 1.2), `11` (TLS 1.1), `10` (TLS 1.0), `s3` (SSL 3.0) * `sni`: `d` (domain present) or `i` (IP/missing) * `cipher_count`: Number of cipher suites (excluding GREASE), 2 digits * `ext_count`: Number of extensions (excluding GREASE), 2 digits * `alpn_first`: First two characters of first ALPN, or `00` if none **JA4\_b** (12 characters): Truncated SHA256 of sorted cipher suites (GREASE filtered, comma-separated hex) **JA4\_c** (12 characters): Truncated SHA256 of sorted extensions (GREASE + SNI + ALPN filtered, comma-separated hex), followed by signature algorithms (if present) separated by underscore **Example:** ``` t13d1516h2_8daaf6152771_e5627efa2ab1 ``` This fingerprint indicates: * `t`: TCP/TLS connection * `13`: TLS 1.3 * `d`: Domain present in SNI * `15`: 15 cipher suites * `16`: 16 extensions * `h2`: HTTP/2 (first ALPN) ### JA4S Format (Server Hello) JA4S fingerprints the server's response: ``` {ja4s_a}_{ja4s_b}_{ja4s_c} ``` **JA4S\_a** (7 characters): `{protocol}{version}{ext_count:2d}{alpn}` * `protocol`: `t` for TCP/TLS, `q` for QUIC * `version`: `13` (TLS 1.3), `12` (TLS 1.2), etc. * `ext_count`: Number of extensions (excluding GREASE), 2 digits * `alpn`: First and last character of selected ALPN, or `00` if none **JA4S\_b** (4 characters): Cipher suite in hex **JA4S\_c** (12 characters): Truncated SHA256 of extensions (SNI and ALPN filtered, NOT sorted per spec) ### Advantages over JA3 1. **Resistant to TLS extension randomization**: JA4 sorts extensions, so Chrome's randomization doesn't affect the fingerprint 2. **Human-readable**: The JA4\_a component is immediately interpretable 3. **QUIC support**: Works with QUIC protocol connections 4. **Includes ALPN**: Captures application protocol negotiation details ### References * [JA4+ GitHub Repository](https://github.com/FoxIO-LLC/ja4) * [JA4+ Technical Specification](https://blog.foxio.io/ja4%2B-network-fingerprinting) ### License JA4 (TLS Client Fingerprinting) is licensed under BSD 3-Clause. JA4S, JA4H, JA4X, JA4T, JA4SSH and other JA4+ methods are licensed under FoxIO License 1.1. See `internal/ja4/LICENSE-JA4` for full license text. *** ### Client Hello Audit Record ```protobuf message TLSClientHello { int64 Timestamp = 1; int32 Type = 2; int32 Version = 3; int32 MessageLen = 4; int32 HandshakeType = 5; uint32 HandshakeLen = 6; int32 HandshakeVersion = 7; bytes Random = 8; uint32 SessionIDLen = 9; bytes SessionID = 10; int32 CipherSuiteLen = 11; int32 ExtensionLen = 12; string SNI = 13; bool OSCP = 14; repeated int32 CipherSuites = 15; repeated int32 CompressMethods = 16; repeated int32 SignatureAlgs = 17; repeated int32 SupportedGroups = 18; repeated int32 SupportedPoints = 19; repeated string ALPNs = 20; string SrcIP = 22; string DstIP = 23; string SrcMAC = 24; string DstMAC = 25; int32 SrcPort = 26; int32 DstPort = 27; repeated int32 Extensions = 28; // Threat intelligence fields bool IsKnownMalware = 30; string ThreatCategory = 31; bool HasGreaseExtensions = 32; bool IsSuspiciousCipherOrder = 34; int32 ExtensionCount = 35; // JA4 fingerprint string Ja4 = 36; } ``` ### Server Hello Audit Record ```protobuf message TLSServerHello { int64 Timestamp = 1; int32 Version = 2; bytes Random = 3; bytes SessionID = 4; int32 CipherSuite = 5; int32 CompressionMethod = 6; bool NextProtoNeg = 7; repeated string NextProtos = 8; bool OCSPStapling = 9; bool TicketSupported = 10; bool SecureRenegotiationSupported = 11; bytes SecureRenegotiation = 12; string AlpnProtocol = 13; bool Ems = 14; repeated bytes Scts = 15; int32 SupportedVersion = 16; bool SelectedIdentityPresent = 18; int32 SelectedIdentity = 19; bytes Cookie = 20; int32 SelectedGroup = 21; repeated int32 Extensions = 22; string SrcIP = 23; string DstIP = 24; string SrcMAC = 25; string DstMAC = 26; int32 SrcPort = 27; int32 DstPort = 28; // JA4S fingerprint string Ja4s = 30; } ``` *** ## Migration from JA3 JA3 support has been removed from netcap in favor of JA4. If you have existing JA3-based workflows: 1. **Threat Intelligence**: JA4 fingerprint databases are being developed. In the meantime, you can use the human-readable JA4\_a component for basic client identification. 2. **Filtering/Detection Rules**: Update rules to match on `Ja4` field instead of the former `Ja3` field. 3. **Historical Data**: Existing audit records with JA3 fields will remain readable, but new captures will only contain JA4 fingerprints. ### Why JA3 Was Removed Modern browsers (Chrome, Firefox, Edge) randomize TLS extension order to prevent fingerprinting-based tracking. This makes JA3 fingerprints inconsistent and unreliable for the same browser version. JA4's sorted extension approach provides stable fingerprints regardless of extension ordering. # Reassembly TCP stream reassembly ## Implementation For reassembling TCP streams the gopacket/reassembly implementation is used. This allows to parse application layer protocols such as HTTP and POP3. The reassembly package currrently only implements reassembling stream over IPv4. To overcome this limitation for HTTP capture, you can use the **proxy** tool. {% content-ref url="/pages/-M0KJmfCZgxxxCiomVIa" %} [HTTP Proxy](/master/http-proxy) {% endcontent-ref %} ## Architecture The gopacket reassembly implementation leaves several options for using it. Netcap currently uses one dedicated assembler for each worker and a single shared connection pool for all streams. Another option would be using a dedicated assembler for each worker for each L7 protocol with a shared stream pool for that specific protocol. This would potentially decrease lock contention for the reassembly, and might be implemented to improve performance in future versions. ## Configuration The following fields of the **decoder.Config** affect the TCP stream reassembly: ```go // Interval to apply connection flushes FlushEvery int // Do not use IPv4 defragger NoDefrag bool // Dont verify the packet checksums Checksum bool // Dont check TCP options NoOptCheck bool // Ignore TCP state machine errors IgnoreFSMerr bool // TCP state machine allow missing init in three way handshake AllowMissingInit bool // Toggle debug mode Debug bool // Dump packet contents as hex for debugging HexDump bool // Wait until all connections finished processing when receiving shutdown signal WaitForConnections bool // Write incomplete HTTP responses to disk when extracting files WriteIncomplete bool ``` ## Debugging To see debug output for the reassembly, run with the **-debug** flag and check the **reassembly.log** file. For more general troubleshooting advice, please refer to the Troubleshooting page: {% content-ref url="/pages/-MZYVj3WL58t53raFi4n" %} [Troubleshooting](/master/troubleshooting) {% endcontent-ref %} # Deep Packet Inspection Identify applications and categories ## Libprotoident NETCAP has support for using **libprotoident** (v[2.0.14](https://github.com/wanduow/libprotoident/releases/tag/2.0.14-1)), to identify 45 application categories and 500+ applications and protocols! The full list of supported protocols can be found here: {% embed url="" %} **libprotoident** is maintained by the WAND group, you can download and install the library here: {% embed url="" %} ## nDPI Furthermore **nDPI** (v4.14 Stable) can be used to identify 244+ applications, they are listed here: {% embed url="" %} **nDPI** is mainted by **ntop**, and can be downloaded here: {% embed url="" %} The results from all heuristic engines (lPI, nDPI and go heuristics) get dedpulicated automatically. Future versions could create a certainity score based on the number of votes from different heuristics. ## Audit Records with Applications Field DPI detected applications are stored in the **Applications** field, which is available in the following audit records: * **Connection**: DPI applications detected for bidirectional flows * **Service**: DPI applications detected for services running on specific IP:Port combinations * **DeviceProfile**: Aggregated DPI applications seen from/to a specific MAC address * **IPProfile**: Aggregated DPI applications seen from/to a specific IP address The Applications field is a repeated string field (array) that contains the names of all detected applications for that audit record. ### DPI Classification Strategy NETCAP invokes DPI classification for each packet up to `MaxPacketsPerFlow` (10 packets): * **Invocation**: DPI is called for each of the first 10 packets per flow * **Internal Management**: godpi internally checks `MinPacketsForClassification` (default: 1) before actually performing classification * **Efficiency**: Results are cached after first successful classification * **Performance**: Automatically stops after 10 packets per flow This ensures short HTTP/REST API connections (typically 3-8 packets) are properly classified while maintaining performance on long-lived flows. **Note:** The `ApplicationProto` field in Connection records may show "Payload" for HTTP connections. This is expected because gopacket does not decode HTTP at the individual packet level (only at the TCP stream reassembly level). The DPI `Applications` field contains the actual protocol identification. Read more about DeviceProfiles here: {% content-ref url="/pages/-MZYVj3TwccuZAN3k4z4" %} [Device Profiles](/master/device-profiles) {% endcontent-ref %} ## Platform Support NETCAPs DPI integration is currently only available on linux and macOS. # Live Capture Capture from a network interface To capture packets live, simple use the **-iface** flag: ``` $ net capture -iface en0 ``` Use the **-interfaces** flag to list all available intefaces and their MTUs: ``` $ net capture -interfaces ┌───────┬─────────┬───────────────────────────┬───────────────────┬───────┐ │ Index │ Name │ Flags │ HardwareAddr │ MTU │ ├───────┼─────────┼───────────────────────────┼───────────────────┼───────┤ │ 1 │ lo0 │ up|loopback|multicast │ │ 16384 │ │ 2 │ gif0 │ pointtopoint|multicast │ │ 1280 │ │ 3 │ stf0 │ 0 │ │ 1280 │ │ 4 │ en5 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 5 │ ap1 │ broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 6 │ en0 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 7 │ en4 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 8 │ en1 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 9 │ en2 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 10 │ en3 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 11 │ bridge0 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 12 │ p2p0 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 2304 │ │ 13 │ awdl0 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1484 │ │ 14 │ llw0 │ up|broadcast|multicast │ XX:XX:XX:XX:XX:XX │ 1500 │ │ 15 │ utun0 │ up|pointtopoint|multicast │ │ 1380 │ │ 16 │ utun1 │ up|pointtopoint|multicast │ │ 2000 │ │ 17 │ utun2 │ up|pointtopoint|multicast │ │ 1380 │ │ 18 │ utun3 │ up|pointtopoint|multicast │ │ 1380 │ │ 19 │ utun4 │ up|pointtopoint|multicast │ │ 1380 │ │ 20 │ utun5 │ up|pointtopoint|multicast │ │ 1380 │ │ 21 │ utun6 │ up|pointtopoint|multicast │ │ 1380 │ │ 22 │ utun7 │ up|pointtopoint|multicast │ │ 1380 │ └───────┴─────────┴───────────────────────────┴───────────────────┴───────┘ ``` ## Promiscous Mode Netcap uses promiscous mode by default, which requires root permissions. You can toggle this behavior with the **-promisc** flag: ``` $ net capture -iface en0 -promisc=false ``` ## Windows For windows, things work a little bit different. First, download & install the latest version of **WinPcap**: {% embed url="" %} Next, open a CMD prompt and run: ```aspnet C:\>getmac /fo csv /v "Connection Name","Network Adapter","Physical Address","Transport Name" "Ethernet0","Intel(R) PRO/1000 MT Network Connection","00-0C-29-BB-EC-9B","\Device\Tcpip_{B1B1E59F-FA8F-4A7B-B28C-7A26F6E00F5A}" ``` Note down the Identifier for your adapter of interest (here: **Ethernet0**), in this example the identifier is: ```aspnet \Device\Tcpip_{B1B1E59F-FA8F-4A7B-B28C-7A26F6E00F5A} ``` To capture traffic on the interface, you must prefix the interface **ID ({B1B1E59F-FA8F-4A7B-B28C-7A26F6E00F5A})** with **\Device\NPF\_** This leaves us with the final command: ```aspnet net.exe capture -iface \Device\NPF_{B1B1E59F-FA8F-4A7B-B28C-7A26F6E00F5A} ``` {% hint style="info" %} Info: When you stop the packet capture on windows with ctrl-C you will see several errors of the format: `failed to remove file remove XXXXX.ncap.gz: The process cannot access the file because it is being used by another process.` This happens because NETCAP creates and opens files for all supported audit records types on startup, and closes them when packet capture is finished or interrupted. Since it often happens that not all supported protocols appeared in the data stream, NETCAP opens the audit record files after closing again, to check if they are empty (=only contain the NETCAP header), and if so, removes the empty audit record files. Similarly, NETCAP automatically removes log files (netcap.log, collector.log, decoder.log, io.log, resolvers.log, reassembly.log, db.log, errors.log) that are empty after flushing and closing their file handles. Unfortunately, windows does not allow closing and opening a file from the same process within such small time interval, which leads to the shown error. As a consequence, empty audit record files and empty log files may not be removed automatically on windows. If you know a workaround for this, please let me know. {% endhint %} # Maltego Integration Graphical link analysis to the rescue! ## Introduction **Maltego** is an open source intelligence (OSINT) and graphical link analysis tool for gathering and connecting information for investigative tasks. {% embed url="" %} It allows to transform data using external knowledge and visualize the results in a graph topology. Transforms are small pieces of code that automatically fetch data from different sources and return the results as visual entities in the desktop client. Transforms are the central elements of Maltego which enable its users to unleash the full potential of the software whilst using a point-and-click logic to run analyses. Netcap provides a set of entities and transformations to analyze packet capture dump files in Maltego! The current implementation focuses on behavorial analysis of entities within the traffic dump. ## Installation Ensure netcap **>= v0.5** is installed and can be found in **$PATH**: ``` $ net -version v0.5 ``` Ensure the **net** binary is placed in **/usr/local/bin**: ``` $ which net /usr/local/bin/net ``` {% hint style="info" %} Transformations in maltego have to specify a working directory. Currently /**usr/local** is used for this so make sure the current user has sufficient right to enter the directory. No data will be written there, all logs from transformations that invoke the netcap core are written into dedicated directories for each processed pcap file. {% endhint %} Next, download install the maltego transformations and enities for netcap: Currently there are **20 entities** and **42 transformations** implemented. You can download them here: {% file src="/files/PDx2Br9L9zKE96e8oy4T" %} Import them into Maltego in the "**Import / Export Config**" tab under "**Import Config**". ## Loading PCAP files into Maltego To load a pcap file into maltego you have two options: 1\) Drag and Drop the file into a maltego graph. The files type will be **maltego.File** by default, and the path to the file on disk is set as a note on the entity. Now, change the entities type to **netcap.PCAP**, and double click it to open the detail view. Copy the filesystem path from the **Notes** tab into the **path** property of the PCAP entity. 2\) Create a new **netcap.PCAP** entity and set the **path** property to the path of your pcap file on disk ## Running Transformations Right click an entity and start typing **Get** into the search bar to see all available transformations for the selected type. Alternatively you can also use the **Run View** in the **Windows** tab to see and launch available transformations with a single click. Transformations are usually bound to specific entities. For example, the **netcap.DeviceProfiles** entity, which represents the device profiles that have been derived from an input PCAP file, currently only offers the following two transformations: ![](/files/-M5Xfon6c4TAO_Kbh1IN) **GetDeviceProfilesWithDPI** enables deep packet inspection, which requires to install dependencies and can slow down the processing drastically. When not using DPI the transformations making use of this data will simply return no results. To add the actual devices to the graph, use the **GetDevices** transformation. ![](/files/-M5XenFiN06my_CIY5Mu) This bring the first usable entities to the graph, of type **netcap.Device**. A device has contacts and addresses it has been using to access network services by itself. When selecting a device entity, you will see the following transformations: ![](/files/-M5XfSCvVACoq0Ki1VfO) The generated entities will be of type **netcap.IPAddr** and contain information about the host, as well as a set of transforms to further drill down and investigate. When selecting an entity of type **netcap.IPAddr**, the following transformations are offered: ![netcap.IPAddr transformations](/files/-M5XdRgKRJe1lBrsszv-) ## Configuration Netcap offers an **OpenFile** maltego transform, which will pass filetypes except for executables to the default system application for the corresponding file format. On macOS the open utility will be used for this and on the linux the default is gio open. You can override the application used for this by setting **NC\_MALTEGO\_OPEN\_FILE**. ## Examples Search for DHCP information from the selected hosts: ![](/files/-M5XhHXTxQ-07HJGtQLH) Add Server Names provided as SNI on the TLS handshake: ![](/files/-M5Xhaq2Un74mbFGA4tD) Use Deep Packet Inspection to list all identified application categories: ![](/files/-M5XhpGDioQY16Px9mQd) Extraction of a POP3 authentication token: ![](/files/aCRynv2iJ5fmYNvg4Jzt) ## Gallery When working with larger amount of nodes, the organic topology can be useful: ![Graph during an investigation (organic topology)](/files/w8o4jrzzQ6JDo5pbT7Vn) Example of interaction with a PHP webshell: ![PHP webshell interaction](/files/BXnunTZ58mw2oBoCxtOc) Example of an exploit abusing a HTTP parameter command injection vulnerability: ![HTTP parameter command injection](/files/OLiPZzMh663ao37YTgZh) Graph during an investigation where the attacker has been identified and further information is gathered: ![Dataset investigation](/files/-M4zlcbKtUlsLz79kL2e) ![Dataset investigation](/files/Efci9PgMCOED31fL5P8G) ![Flow Graph](/files/-M4zloIdCPwzDrcwoYtu) ## Detail Views ![](/files/-M5Xh0dpS4dTdt0bzCF-) ![](/files/-M5XiU2R3LIJMsfM4GOX) ![](/files/-M5XimlxAjqbP6NdGMv0) ![](/files/-M5XipirHhG9fK26tfBE) ![](/files/-M5XiuwQt33Bk69C3g6r) ![](/files/-M5Xj0MFh3cX41uaEuLY) # Logging Logging options ## Quiet mode Netcap writes a general summary to stdout, if you wish to disable output entirely use the **-quiet** flag: ``` $ net capture -quiet ``` When the quiet mode is used, the output is instead written into the **netcap.log** file in the directory where netcap is executed from. ## Decoding errors Errors when parsing packets are logged by default into the **errors.log** file in the current directory. Each log entry contains a hex dump of the entire packet and the error message or stack trace. ## Log files in debug mode The following log file are produced when running with the **-debug** flag: * debug.log: general debug messages * reassembly.log: tcp stream reassembly debug logs ## Automatic removal of empty log files Similar to how empty audit record files are handled, NETCAP automatically removes log files that are empty after flushing and closing their file handles at the end of packet capture or processing. This applies to all log files created during execution: * netcap.log * collector.log * decoder.log * io.log * resolvers.log * reassembly.log * db.log * errors.log When a log file contains no data (size = 0 bytes) after being flushed and closed, it is automatically deleted to avoid cluttering the output directory with empty files. This behavior ensures that only log files containing actual log entries are kept. **Note**: On Windows systems, there may be timing issues that prevent empty log file removal in some cases, similar to the behavior with empty audit record files. # Packet Contexts Preserve information about other layers on audit records Netcap v0.4.3 added PacketContexts, a new feature to preserve additional information on audit records. This need originates from a core concept of Netcap: separating the results based on the different protocols, which results in audit records like TCP not providing any IP address information, since they only provide information about the TCP protocol which operates at the Transport Layer. A packet context looks as follows: ```erlang message PacketContext { string SrcIP = 1; string DstIP = 2; string SrcPort = 3; string DstPort = 4; } ``` Many audit record types (e.g: IPv4, IPv6, TCP, UDP, ICMPv4, ICMPv6, SCTP, DNS, DHCP, SIP etc) now have an addional field called context, which will contain a PacketContext that describes the flow where the packet originated from, if context capture is enabled. When generating a CSV representation the fields from the PacketContext are flattened, which means they will be shown as if they are direct member of the dumped audit record, e.g: ``` $ net dump -read UDP.ncap.gz -fields Timestamp,SrcPort,DstPort,Length,Checksum,PayloadEntropy,PayloadSize,Payload,SrcIP,DstIP ``` If the audit record already has information that would be duplicated by the PacketContext (for example Port information for UDP), this information is cleared on the context to avoid repetition. Context capture is enabled by default and can be controlled using the **-context** flag. # Industrial Control Systems ICS / SCADA threat hunting ## Protocol Support Netcap offers audit records for the following protocols seen in industrial control systems: * Ethernet/IP * CIP - Common Industrial Protocol * Modbus / ModbusTCP The decoders are enabled by default. ## Modbus ```erlang message Modbus { string Timestamp = 1; int32 TransactionID = 2; // Identification of a MODBUS Request/Response transaction int32 ProtocolID = 3; // It is used for intra-system multiplexing int32 Length = 4; // Number of following bytes (includes 1 byte for UnitIdentifier + Modbus data length int32 UnitID = 5; // Identification of a remote slave connected on a serial line or on other buses bytes Payload = 6; bool Exception = 7; int32 FunctionCode = 8; PacketContext Context = 9; } ``` ## CIP ```erlang message CIP { string Timestamp = 1; bool Response = 2; // false if request, true if response int32 ServiceID = 3; // The service specified for the request uint32 ClassID = 4; // request only uint32 InstanceID = 5; // request only int32 Status = 6; // Response only repeated uint32 AdditionalStatus = 7; // Response only bytes Data = 8; // Command data for request, reply data for response PacketContext Context = 9; } ``` ## ENIP ```erlang message ENIP { string Timestamp = 1; uint32 Command = 2; uint32 Length = 3; uint32 SessionHandle = 4; uint32 Status = 5; bytes SenderContext = 6; uint32 Options = 7; ENIPCommandSpecificData CommandSpecific = 8; PacketContext Context = 9; } ``` # File Extraction Extract transferred files and save them to disk ## Introduction Various protocols allow transferring files (e.g: HTTP, POP3) and some are made for the sole purpose of transferring files (FTP, SMB etc). From a network security monitoring perspective, transferred files are interesting because they can contain malicious software or prohibited content. Netcap extracts files from HTTP and saves them to disk, for both HTTP responses and HTTP requests. It uses the **File** audit record type to model the extracted information. > Future versions will add file extraction support for other protocols as well. ## File Audit Records The audit record definition for a file looks like this: ```erlang message File { string Timestamp = 1; string Name = 2; int64 Length = 3; string Hash = 4; string Location = 5; string Ident = 6; string Source = 7; string ContentType = 8; PacketContext Context = 9; string Host = 10; string ContentTypeDetected = 11; } ``` As can be seen, the content type indicated by the HTTP header is included, as well as the content type that was detected. In addition, the source of the File is specified (e.g: from HTTP, Mail attachment etc), as well the identifier of the connection where it originated from. The Hash field currently holds an MD5 hash of the file, Location points to the path on disk where the file is stored. > This will likely be replaced with a stronger hash function in the future. ## Usage File capture is enabled by default and will store extracted files in the **files** subdirectory within your output directory. The **-fileStorage** flag allows you to customize this path (relative to the output directory): ``` $ net capture -read traffic.pcap -fileStorage files ``` To disable file extraction, set an empty string: ``` $ net capture -read traffic.pcap -fileStorage "" ``` After capturing, lets inspect the directory contents: ``` $ tree files files ├── application │ └── x-gzip │ └── unknown-193.24.227.12->216.66.80.30-80->60075.gz ├── image │ └── x-icon │ └── favicon.ico-193.24.227.12->216.66.80.30-80->60076.ico └── text └── html ├── unknown-193.24.227.12->216.66.80.30-80->55031.html ├── unknown-193.24.227.12->216.66.80.30-80->55032.html ├── unknown-193.24.227.12->216.66.80.30-80->55033.html └── unknown-80.237.133.136->192.168.110.10-80->1152.html 6 directories, 6 files ``` As you can see, files are sorted by their MIME types retrieved from classifying them using the go standard library and named after the TCP connection they originated from. By default, only complete requests and responses are captured, if you also want to extract incomplete data, use the **-writeincomplete** flag: ``` $ net capture -read traffic.pcap -fileStorage files -writeincomplete ``` Dumping a File on the commandline looks like this: ``` $ net dump -read File.ncap.gz -struc NC_File Timestamp: "2015-03-08 14:05:29.664213 +0000 UTC" Name: "ads.bmp" Length: 126 Hash: "2d5a035011854b04a456b244b15a583b" Location: "files/image/bmp/ads.bmp-80.239.178.178->192.168.0.51-80->41214.bmp" Ident: "80.239.178.178->192.168.0.51-80->41214" Source: "HTTP RESPONSE from /ads.bmp" Context: < SrcIP: "192.168.0.51" DstIP: "80.239.178.178" SrcPort: "41214" DstPort: "80" > ContentTypeDetected: "image/bmp" ... ``` For properly exploring files for each host I recommend using the Maltego Integration: {% content-ref url="/pages/-MZYVj3NaKroiawq0yoD" %} [Maltego Integration](/master/maltego-integration) {% endcontent-ref %} ![](/files/Efci9PgMCOED31fL5P8G) # Email Extraction Extract transferred emails ## Motivation Emails are a key communication mechanism that holds plenty of digital evidence, starting from Mail header information about the sender and route, to transferred files via attachments. Netcap currently extracts Email fetched over POP3 and support IMAP is in the making. ## POP3 A POP3 audit record contains information about the addresses involved, as well as authentication information. The fetched mails are provided as an array of **Mail** instances. ```erlang message POP3 { string Timestamp = 1; string ClientIP = 2; string ServerIP = 3; string AuthToken = 4; string User = 5; string Pass = 6; repeated Mail Mails = 7; } ``` A Mail instance has the following fields: ```erlang message Mail { string ReturnPath = 1; string DeliveryDate = 2; string From = 3; string To = 4; string CC = 5; string Subject = 6; string Date = 7; string MessageID = 8; string References = 9; string InReplyTo = 10; string ContentLanguage = 11; bool HasAttachments = 12; string XOriginatingIP = 13; string ContentType = 14; string EnvelopeTo = 15; repeated MailPart Body = 16; } ``` For exploring captured emails, the Maltego Integration can be used: {% content-ref url="/pages/-MZYVj3NaKroiawq0yoD" %} [Maltego Integration](/master/maltego-integration) {% endcontent-ref %} ![](/files/-M5kyIbH_wnm4b1scsMm) ![](/files/-M5kyNom5uMyxBCOnl-H) ## SMTP For SMTP an audit record is also available, though mail extraction has not been implemented yet: ```erlang message SMTP { string Timestamp = 1; bool IsEncrypted = 2; bool IsResponse = 3; repeated SMTPResponse ResponseLines = 4; SMTPCommand Command = 5; PacketContext Context = 6; } ``` # Device Profiles Behavorial Profiling with Netcap ## Motivation Which device on the network uses which IP address? Which addresses / devices did it contact? How are devices related to each other, how does communication flow? Identifying devices within a network is a good starting point for any investigation and helps to understand complex situations and relations quickly. The **DeviceProfile** custom decoder implements exactly this, and is enabled from v0.5 on by default. > Note: DeviceProfile currently get written when processing all traffic is done - that means when using live capture, the profiles will be available when processing stopped. Future versions will implement a flushing mechanism similar to the one for Flows / Connections. DeviceProfiles rely heavily on local resolvers to be set up and configured. You can use the **DeviceProfile** audit records without resolvers, but you will get less information. Read more about the resolvers here: {% content-ref url="/pages/-MZYVj3IwWYjp6yCQjOs" %} [Resolvers](/master/resolvers) {% endcontent-ref %} Analyzing DeviceProfiles can be done using Maltego, for example: {% content-ref url="/pages/-MZYVj3NaKroiawq0yoD" %} [Maltego Integration](/master/maltego-integration) {% endcontent-ref %} ![DeviceProfiles and their used IP addresses from an industrial automation system](/files/-M5lEVm_iZaECOSQ137P) ## DeviceProfile Audit Records Lets look at the protocol buffer definition for a device profile: ```erlang message DeviceProfile { string MacAddr = 1; string DeviceManufacturer = 2; repeated string DeviceIPs = 3; repeated string Contacts = 4; int64 NumPackets = 5; int64 Timestamp = 6; // first seen uint64 Bytes = 7; repeated string Applications = 8; // DPI detected applications } ``` As you can see, a DeviceProfile is a summary structure built around the hardware address of a physical device. It captures the addresses that have been used, as well as the contacted addresses in form of IPProfiles, among other meta information, like the number of packets, the hardware manufacturer, and DPI detected applications. Lets take a closer look at an IPProfile: ```erlang message IPProfile { string Addr = 1; int64 NumPackets = 2; string Geolocation = 3; repeated string DNSNames = 4; string TimestampFirst = 5; string TimestampLast = 6; repeated string Applications = 7; map Ja3 = 8; // ja3 to lookup map Protocols = 9; uint64 Bytes = 10; map DstPorts = 11; // Ports to bytes map SrcPorts = 12; // Ports to bytes map SNIs = 13; } ``` This is the information associated with a single ip address. Note how in addition to general meta data like the number of packets, bytes and timestamps, there is also information retrieved from the new resolvers API, namely the geolocation and dns names. Additionally, the results from Deep Packet Inspection for all flows seen from or towards this IP are added as well, in addition to the Server Name Indicators seen and flow statistics for each seen port number from this address. To enhance encrypted telemetry, Ja3 fingerprints seen for this host are mapped to lookup results from the Ja3 database. # Rules Engine The NETCAP Rules Engine allows you to define detection rules that automatically generate alerts when specific network patterns are observed. Rules use expr-lang expressions to match audit records and can be configured to detect various attack patterns, anomalies, and policy violations. ## Table of Contents * [Overview](#overview) * [Rule Configuration](#rule-configuration) * [Writing Rules](#writing-rules) * [Alert Structure](#alert-structure) * [MITRE ATT\&CK Integration](#mitre-attck-integration) * [Example Rules](#example-rules) * [Best Practices](#best-practices) ## Overview ### Features * **Expression-based matching**: Use powerful expr-lang expressions to define detection logic * **MITRE ATT\&CK mapping**: Associate rules with MITRE ATT\&CK techniques * **Alert deduplication**: Automatic deduplication within configurable time windows * **Rate limiting**: Prevent alert flooding with per-rule rate limits * **Multiple severity levels**: Categorize alerts as low, medium, high, or critical * **Flexible tagging**: Organize rules with custom tags * **Audit record output**: Alerts are written as NETCAP audit records for analysis ### Architecture ``` ┌──────────────┐ │ Audit Records│ └──────┬───────┘ │ ▼ ┌──────────────┐ │ Rules Engine │ └──────┬───────┘ │ ├─► Alert (if match) │ ▼ ┌──────────────┐ │ Alert.ncap.gz│ └──────────────┘ ``` ## Rule Configuration Rules are defined in YAML files with the following structure: ```yaml rules: - name: Rule_Name description: Human-readable description type: AuditRecordType # e.g., TCP, HTTP, DNS expression: expr-lang expression severity: low|medium|high|critical mitre: ["T1XXX.YYY"] # MITRE ATT&CK IDs tags: ["tag1", "tag2"] enabled: true|false ``` ### Field Reference #### `name` (required) Unique identifier for the rule. Used in alert generation and logging. **Example**: `SSH_Bruteforce_Attempt` #### `description` (required) Human-readable explanation of what the rule detects. **Example**: `Detect high frequency SSH connection attempts indicating possible brute force attack` #### `type` (required) The audit record type this rule applies to. Can be specified with or without the `NC_` prefix. **Valid values**: `TCP`, `UDP`, `HTTP`, `DNS`, `TLS`, `IPv4`, `IPv6`, `ICMP`, etc. **Examples**: * `TCP` or `NC_TCP` * `HTTP` or `NC_HTTP` #### `expression` (required) An expr-lang expression that evaluates to true when the rule matches. Has access to all fields of the audit record type. **Example**: `DstPort == 22 && SYN && !ACK` #### `severity` (required) Alert severity level. **Valid values**: `low`, `medium`, `high`, `critical` (case-insensitive) #### `mitre` (optional) Array of MITRE ATT\&CK technique IDs associated with this detection. **Format**: `["T####.###", ...]` **Example**: `["T1110.001", "T1021.004"]` Find technique IDs at: #### `tags` (optional) Custom tags for organizing and categorizing rules. **Example**: `["ssh", "bruteforce", "authentication"]` #### `enabled` (required) Whether the rule is active. Disabled rules are not evaluated. **Valid values**: `true`, `false` ## Writing Rules ### Basic Rule Structure ```yaml rules: - name: HTTPS_Traffic description: Detect HTTPS connections type: TCP expression: DstPort == 443 severity: low tags: ["https", "encrypted"] enabled: true ``` ### Using Helper Functions Rules have access to all helper functions available in filtering: ```yaml rules: - name: Private_to_Public description: Detect outbound traffic from private networks type: IPv4 expression: IsPrivateIP(SrcIP) && IsPublicIP(DstIP) severity: low tags: ["network", "outbound"] enabled: true ``` ### Complex Expressions Combine multiple conditions for sophisticated detections: ```yaml rules: - name: Suspicious_HTTP_Upload description: Detect large POST requests with suspicious user agents type: HTTP expression: | Method == "POST" && ReqContentLength > 10000000 && UserAgent in ["curl", "wget", "python"] severity: high mitre: ["T1041"] tags: ["exfiltration", "http"] enabled: true ``` ### Field Access Access nested fields using dot notation: ```yaml rules: - name: DNS_Large_Query description: Detect DNS queries with long domain names type: DNS expression: len(Questions) > 0 && len(Questions[0].Name) > 100 severity: medium mitre: ["T1071.004"] tags: ["dns", "tunneling"] enabled: true ``` ## Alert Structure When a rule matches, an alert is generated with the following information: ```go type Alert struct { Timestamp int64 // When the alert was generated Name string // Rule name Description string // Rule description SrcIP string // Source IP from matched record DstIP string // Destination IP from matched record SrcPort string // Source port from matched record DstPort string // Destination port from matched record MITRE string // MITRE ATT&CK IDs (comma-separated) RuleName string // Rule name (duplicate of Name) RecordType string // Type of audit record that matched Severity string // Alert severity level Tags []string // Rule tags MatchedRecord string // JSON representation of matched record } ``` Alerts are written to `Alert.ncap.gz` in the output directory and can be analyzed like any other audit record: ```bash # View all alerts net dump -read Alert.ncap.gz # Filter critical alerts net dump -read Alert.ncap.gz -filter "Severity == 'critical'" # Find SSH-related alerts net dump -read Alert.ncap.gz -filter "RuleName in ['SSH_Bruteforce_Attempt', 'SSH_Tunnel_Detection']" ``` ## MITRE ATT\&CK Integration NETCAP rules can be mapped to MITRE ATT\&CK tactics and techniques to provide context about detected threats. ### Common Technique Mappings | Technique ID | Name | Use Case | | ------------ | -------------------------------------- | --------------------------------- | | T1046 | Network Service Scanning | Port scans, service enumeration | | T1071.001 | Web Protocols | HTTP/HTTPS C2 communication | | T1071.004 | DNS | DNS tunneling, exfiltration | | T1110 | Brute Force | Password guessing attacks | | T1190 | Exploit Public-Facing Application | Web application attacks | | T1021.004 | SSH | Remote access via SSH | | T1041 | Exfiltration Over C2 Channel | Data exfiltration | | T1048 | Exfiltration Over Alternative Protocol | Non-standard exfiltration methods | | T1572 | Protocol Tunneling | Encapsulation for evasion | ### Example with MITRE Mapping ```yaml rules: - name: SQL_Injection_Attempt description: Detect SQL injection patterns in HTTP requests type: HTTP expression: MatchesPattern(URL, "(?i)(union.*select|insert.*into|delete.*from)") severity: critical mitre: ["T1190"] # Exploit Public-Facing Application tags: ["web", "sql-injection", "injection"] enabled: true ``` ## Example Rules ### Network Reconnaissance ```yaml rules: # Port Scanning - name: SYN_Scan description: Detect TCP SYN scan attempts type: TCP expression: SYN && !ACK && !RST severity: medium mitre: ["T1046"] tags: ["reconnaissance", "port-scan"] enabled: true # ICMP Reconnaissance - name: ICMP_Ping_Sweep description: Detect ICMP echo requests (ping sweep) type: ICMPv4 expression: TypeCode == 8 severity: low mitre: ["T1018"] tags: ["reconnaissance", "icmp"] enabled: true ``` ### Data Exfiltration ```yaml rules: # DNS Exfiltration - name: DNS_Tunneling description: Detect suspiciously large DNS queries type: DNS expression: len(Questions) > 0 && len(Questions[0].Name) > 100 severity: high mitre: ["T1048.003"] tags: ["exfiltration", "dns", "tunneling"] enabled: true # Large Upload - name: Large_HTTP_Upload description: Detect large HTTP uploads type: HTTP expression: Method == "POST" && ReqContentLength > 50000000 severity: high mitre: ["T1041"] tags: ["exfiltration", "http"] enabled: true ``` ### Malware Communication ```yaml rules: # IRC C2 - name: IRC_Communication description: Detect IRC traffic (possible botnet C2) type: TCP expression: DstPort >= 6667 && DstPort <= 6669 severity: high mitre: ["T1219"] tags: ["malware", "c2", "irc"] enabled: true # Suspicious Port - name: Backdoor_Port_31337 description: Detect connections to common backdoor port type: TCP expression: DstPort == 31337 || SrcPort == 31337 severity: critical mitre: ["T1571"] tags: ["malware", "backdoor"] enabled: true ``` ### Web Attacks ```yaml rules: # SQL Injection - name: SQL_Injection description: Detect SQL injection attempts type: HTTP expression: MatchesPattern(URL, "(?i)(union.*select|insert.*into)") severity: critical mitre: ["T1190"] tags: ["web", "sql-injection"] enabled: true # Directory Traversal - name: Directory_Traversal description: Detect path traversal attempts type: HTTP expression: MatchesPattern(URL, "\\.\\./") severity: high mitre: ["T1083"] tags: ["web", "path-traversal"] enabled: true ``` ## Best Practices ### Rule Design 1. **Be Specific**: Target specific behaviors rather than broad patterns ```yaml # Good - specific expression: Method == "POST" && DstPort == 80 && ReqContentLength > 10000000 # Bad - too broad expression: ReqContentLength > 0 ``` 2. **Balance False Positives**: Adjust thresholds to minimize false alerts ```yaml # May generate false positives expression: DstPort == 22 # Better - more specific expression: DstPort == 22 && SYN && !ACK && IsPublicIP(SrcIP) ``` 3. **Use Appropriate Severity**: Match severity to actual threat level * `low`: Informational, minor anomalies * `medium`: Suspicious activity, requires investigation * `high`: Likely malicious activity * `critical`: Active attacks, immediate response needed ### Rule Organization 1. **Group Related Rules**: Create separate files for different categories ``` rules/ ├── reconnaissance.yml ├── exfiltration.yml ├── malware.yml └── web_attacks.yml ``` 2. **Use Descriptive Names**: Make rule purpose clear from name ```yaml # Good name: SSH_Bruteforce_External_Source # Bad name: Rule_1 ``` 3. **Add Comprehensive Tags**: Enable filtering and analysis ```yaml tags: ["protocol:ssh", "attack:bruteforce", "source:external"] ``` ### Testing Rules 1. **Test Against Known Traffic**: Verify rules work as expected ```bash # Test rule file net capture -read traffic.pcap -rules test_rules.yml -out test_output # Check generated alerts net dump -read test_output/Alert.ncap.gz ``` 2. **Monitor Alert Volume**: Ensure rules don't generate excessive alerts ```bash # Count alerts per rule net dump -read Alert.ncap.gz -csv | cut -d';' -f2 | sort | uniq -c ``` 3. **Review False Positives**: Refine rules based on real-world results ```bash # Review specific rule alerts net dump -read Alert.ncap.gz -filter "RuleName == 'SSH_Bruteforce'" ``` ### Performance Optimization 1. **Limit Regex Complexity**: Simple patterns are faster ```yaml # Fast expression: Method == "POST" # Slower expression: MatchesPattern(Method, "^(POST|PUT|DELETE)$") ``` 2. **Order Conditions**: Place fast checks first ```yaml # Good - fast check first expression: DstPort == 80 && MatchesPattern(URL, "complex_regex") # Suboptimal - slow check first expression: MatchesPattern(URL, "complex_regex") && DstPort == 80 ``` 3. **Use Helper Functions**: Optimized native implementations ```yaml # Prefer expression: IsPrivateIP(SrcIP) # Over expression: InSubnet(SrcIP, "10.0.0.0/8") || InSubnet(SrcIP, "172.16.0.0/12") ``` ## Troubleshooting ### Rule Not Triggering 1. **Check rule is enabled**: `enabled: true` 2. **Verify record type**: Ensure rule type matches audit records 3. **Test expression**: Use filter on dump command to test expression 4. **Check field names**: Use `-fields` flag to see available fields ### Too Many Alerts 1. **Increase specificity**: Add more conditions to reduce false positives 2. **Adjust thresholds**: Increase numeric thresholds 3. **Check deduplication**: Ensure deduplication window is appropriate ### Expression Errors ```bash # Common errors and solutions # Error: undefined identifier "InvalidField" # Solution: Check available fields with -fields flag # Error: type mismatch # Solution: Ensure field types match comparison (string vs int) # Error: invalid regex # Solution: Test regex pattern separately, escape special characters ``` ## Next Steps * Review [example rules](https://github.com/dreadl0ck/netcap/blob/master/rules/examples/README.md) for more patterns * See [FILTERING.md](https://github.com/dreadl0ck/netcap/blob/master/docs/FILTERING.md) for expression syntax details * Check the [expr-lang documentation](https://expr-lang.org/docs/language-definition) * Explore MITRE ATT\&CK framework at # Firewall Response Actions NETCAP can automatically execute firewall actions (like blocking IPs) when detection rules match. This enables automated incident response by integrating with the Linux iptables firewall subsystem. ## Table of Contents * [Overview](#overview) * [Requirements](#requirements) * [Architecture](#architecture) * [Configuration](#configuration) * [Response Action Types](#response-action-types) * [Rule Examples](#rule-examples) * [Firewall Manager API](#firewall-manager-api) * [Safety Features](#safety-features) * [Monitoring & Statistics](#monitoring--statistics) * [Best Practices](#best-practices) * [Troubleshooting](#troubleshooting) ## Overview ### Features * **Automated Blocking**: Automatically block IPs based on detection rules * **Time-based Expiration**: Blocks automatically expire after configurable durations * **Whitelist Protection**: Prevent blocking of critical infrastructure * **Custom Chain**: Uses dedicated `NETCAP` iptables chain for easy management * **Dual-Stack Support**: Works with both IPv4 and IPv6 * **Dry-Run Mode**: Test configurations without modifying firewall * **Statistics Tracking**: Monitor block counts, expirations, and errors * **Graceful Cleanup**: All rules removed on shutdown ### Architecture ``` ┌─────────────────────────────────────────────────────────────────────┐ │ NETCAP │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────┐ ┌──────────────────────────────────┐ │ │ │ Rules Engine │─────────│ Detection Rules │ │ │ │ (rules/) │ │ (YAML with expressions) │ │ │ └────────┬─────────┘ └──────────────────────────────────┘ │ │ │ │ │ │ on match + alert generated │ │ ▼ │ │ ┌──────────────────┐ ┌──────────────────────────────────┐ │ │ │ Action Executor │─────────│ Response Actions │ │ │ │ │ │ - iptables_block │ │ │ └────────┬─────────┘ │ - iptables_reject │ │ │ │ │ - iptables_rate_limit │ │ │ ▼ │ - iptables_log │ │ │ ┌──────────────────┐ └──────────────────────────────────┘ │ │ │ Firewall Manager │◄──── github.com/coreos/go-iptables │ │ │ (firewall/) │ │ │ └────────┬─────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Linux iptables │ │ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ │ │ NETCAP Chain (custom) │ │ │ │ │ │ -s 192.168.1.100 -j DROP -m comment "NETCAP: ..." │ │ │ │ │ │ -s 10.0.0.50 -j REJECT -m comment "NETCAP: ..." │ │ │ │ │ └──────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ INPUT ──► -j NETCAP │ │ │ │ FORWARD ──► -j NETCAP │ │ │ │ OUTPUT ──► -j NETCAP │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` ## Requirements | Requirement | Details | | -------------------- | ------------------------------------- | | **Operating System** | Linux (iptables required) | | **Privileges** | Root or `CAP_NET_ADMIN` capability | | **iptables** | iptables and/or ip6tables installed | | **Kernel Modules** | `xt_comment` module for rule comments | ### Non-Linux Platforms On non-Linux platforms (macOS, Windows), the firewall manager returns an error: ``` firewall management is only supported on Linux ``` Rules with firewall response actions will be skipped with a warning. ## Configuration ### Firewall Manager Configuration ```go config := &firewall.ManagerConfig{ // Custom chain name (default: "NETCAP") ChainName: "NETCAP", // Enable IPv4 iptables (default: true) EnableIPv4: true, // Enable IPv6 ip6tables (default: true) EnableIPv6: true, // How often to check for expired blocks (default: 1 minute) CleanupInterval: 1 * time.Minute, // Default block duration if not specified (default: 30 minutes) DefaultDuration: 30 * time.Minute, // IPs/CIDRs that should never be blocked Whitelist: []string{ "127.0.0.0/8", "::1/128", "192.168.1.1", // Gateway }, // Preview mode - log actions without executing DryRun: false, // Enable verbose logging Verbose: true, } manager, err := firewall.NewManager(config) ``` ### Rule Configuration with Response Actions Add an `actions` array to detection rules: ```yaml rules: - name: Block SSH Brute Force description: Block hosts after multiple SSH attempts type: SSH expression: DstPort == 22 severity: high threshold: 10 threshold_window: 300 actions: - type: iptables_block config: target: source # "source" or "destination" duration: 2h # Block duration enabled: true tags: ["ssh", "brute-force", "block"] ``` ## Response Action Types ### `iptables_block` Blocks traffic by adding a DROP rule to iptables. ```yaml actions: - type: iptables_block config: target: source # "source" or "destination" duration: 30m # Duration: 30m, 1h, 24h, etc. ``` | Parameter | Type | Default | Description | | ---------- | ---------- | ---------- | --------------------------------------------------------------------- | | `target` | string | `"source"` | Block source (`source`/`src`) or destination (`destination`/`dst`) IP | | `duration` | string/int | `30m` | Block duration. String (`"30m"`, `"2h"`) or int (minutes) | ### `iptables_reject` Rejects traffic with an ICMP response (more informative than DROP). ```yaml actions: - type: iptables_reject config: target: source duration: 1h ``` | Parameter | Type | Default | Description | | ---------- | ---------- | ---------- | ------------------------------ | | `target` | string | `"source"` | Block source or destination IP | | `duration` | string/int | `30m` | Block duration | ### `iptables_log` Logs matching traffic (currently logs to stdout, iptables LOG target planned). ```yaml actions: - type: iptables_log config: prefix: "SUSPICIOUS: " ``` | Parameter | Type | Default | Description | | --------- | ------ | ------------ | ------------------ | | `prefix` | string | `"NETCAP: "` | Log message prefix | ### `iptables_rate_limit` Rate-limits traffic from/to an IP (placeholder - full implementation pending). ```yaml actions: - type: iptables_rate_limit config: rate: "10/minute" burst: 5 ``` | Parameter | Type | Default | Description | | --------- | ------ | ------------- | ------------------------ | | `rate` | string | `"10/minute"` | Rate limit specification | | `burst` | int | `5` | Initial burst allowance | > **Note**: Rate limiting requires the `hashlimit` iptables module and is not fully implemented. ## Rule Examples ### Port Scanning Detection & Block ```yaml - name: Block Port Scanner description: Block hosts performing port scans type: TCP expression: | SYN && !ACK && PayloadSize == 0 severity: high threshold: 50 threshold_window: 10 actions: - type: iptables_block config: target: source duration: 1h enabled: true mitre: ["T1046"] tags: ["scanning", "recon", "block"] ``` ### SSH Brute Force Protection ```yaml - name: Block SSH Brute Force description: Block hosts after 10 SSH connection attempts in 5 minutes type: TCP expression: DstPort == 22 && SYN && !ACK severity: high threshold: 10 threshold_window: 300 actions: - type: iptables_block config: target: source duration: 2h enabled: true mitre: ["T1110"] tags: ["ssh", "brute-force", "block"] ``` ### DNS Tunneling Detection ```yaml - name: Block DNS Tunneling description: Block hosts using DNS tunneling for data exfiltration type: DNS expression: | len(Questions) > 0 && len(Questions[0].Name) > 100 severity: high threshold: 20 threshold_window: 60 actions: - type: iptables_block config: target: source duration: 4h - type: iptables_log config: prefix: "DNS_TUNNEL: " enabled: true mitre: ["T1071.004", "T1048"] tags: ["dns", "tunneling", "exfiltration", "block"] ``` ### Web Attack Detection ```yaml - name: Block SQL Injection Attempts description: Block hosts attempting SQL injection type: HTTP expression: | MatchesPattern(URL, "(?i)(union.*select|insert.*into|drop.*table)") severity: critical threshold: 3 threshold_window: 60 actions: - type: iptables_block config: target: source duration: 24h enabled: true mitre: ["T1190"] tags: ["sql-injection", "web-attack", "block"] ``` ### SYN Flood Protection ```yaml - name: Block SYN Flood Source description: Block hosts generating SYN flood attacks type: TCP expression: SYN && !ACK && PayloadSize == 0 severity: critical threshold: 100 threshold_window: 5 actions: - type: iptables_block config: target: source duration: 6h - type: iptables_log config: prefix: "SYN_FLOOD: " enabled: true mitre: ["T1498"] tags: ["dos", "syn-flood", "block"] ``` ## Firewall Manager API ### Creating a Manager ```go import "github.com/dreadl0ck/netcap/firewall" // Create with default config manager, err := firewall.NewManager(nil) // Create with custom config config := firewall.DefaultManagerConfig() config.Whitelist = append(config.Whitelist, "10.0.0.1") config.Verbose = true manager, err := firewall.NewManager(config) ``` ### Blocking IPs ```go // Block an IP err := manager.BlockIP("192.168.1.100", &firewall.BlockConfig{ Target: "source", Duration: 30 * time.Minute, Action: "DROP", RuleName: "manual-block", Reason: "Port scanning detected", }) // Block a CIDR range err := manager.BlockCIDR("10.0.0.0/24", &firewall.BlockConfig{ Target: "source", Duration: 1 * time.Hour, Action: "DROP", RuleName: "network-block", }) ``` ### Unblocking IPs ```go // Unblock an IP err := manager.UnblockIP("192.168.1.100") // Unblock a CIDR err := manager.UnblockCIDR("10.0.0.0/24") ``` ### Querying State ```go // Check if IP is blocked if manager.IsBlocked("192.168.1.100") { fmt.Println("IP is blocked") } // Get all active blocks blocks := manager.GetActiveBlocks() for _, block := range blocks { fmt.Printf("Blocked: %s (expires: %v)\n", block.IP, block.ExpiresAt) } // Get statistics stats := manager.GetStats() fmt.Printf("Blocks created: %d\n", stats["blocks_created"]) fmt.Printf("Active blocks: %d\n", stats["active_blocks"]) ``` ### Whitelist Management ```go // Add to whitelist manager.AddToWhitelist("192.168.1.1") manager.AddToWhitelist("10.0.0.0/8") // Remove from whitelist manager.RemoveFromWhitelist("192.168.1.1") ``` ### Cleanup ```go // Flush all rules (but keep chain) err := manager.Flush() // Close manager (flushes rules, removes chain, stops goroutines) err := manager.Close() ``` ### Integration with Rules Engine ```go import ( "github.com/dreadl0ck/netcap/firewall" "github.com/dreadl0ck/netcap/rules" ) // Create firewall manager fwManager, err := firewall.NewManager(nil) if err != nil { log.Fatal(err) } defer fwManager.Close() // Create rules engine engine, err := rules.NewEngine("rules/", alertWriter) if err != nil { log.Fatal(err) } // Connect firewall manager to rules engine engine.SetFirewallManager(fwManager) // Now response actions in rules will automatically execute ``` ## Safety Features ### Whitelist Protection IPs/CIDRs in the whitelist are never blocked: ```go config := firewall.DefaultManagerConfig() config.Whitelist = []string{ "127.0.0.0/8", // Localhost "::1/128", // IPv6 localhost "192.168.1.1", // Gateway "10.0.0.0/8", // Internal network } ``` ### Automatic Expiration All blocks expire automatically: * Configurable per-rule duration * Default: 30 minutes * Background cleanup every minute * Zero duration = permanent until restart ### Custom Chain Isolation All rules are placed in a dedicated `NETCAP` chain: ```bash # View NETCAP chain rules sudo iptables -L NETCAP -n -v # Manually flush if needed sudo iptables -F NETCAP ``` ### Dry-Run Mode Test configurations without modifying firewall: ```go config := firewall.DefaultManagerConfig() config.DryRun = true // Actions are logged but not executed ``` ### Graceful Shutdown On `Close()`: 1. Cleanup goroutine stopped 2. All rules flushed 3. Jump rules removed 4. Custom chain deleted ## Monitoring & Statistics ### Available Statistics ```go stats := manager.GetStats() ``` | Metric | Description | | ----------------- | --------------------------------------- | | `blocks_created` | Total blocks created since start | | `blocks_removed` | Blocks manually removed | | `blocks_expired` | Blocks removed due to expiration | | `duplicates_skip` | Duplicate block requests skipped | | `whitelist_skip` | Block requests skipped due to whitelist | | `errors` | Total errors encountered | | `active_blocks` | Currently active blocks | ### Action Statistics (Rules Engine) ```go actionStats := engine.GetActionStats() ``` | Metric | Description | | ------------------ | ------------------------------- | | `actions_executed` | Total response actions executed | | `actions_success` | Successful actions | | `actions_failed` | Failed actions | | `ips_blocked` | Total IPs blocked | ### Viewing Active Blocks ```go blocks := manager.GetActiveBlocks() for _, b := range blocks { fmt.Printf("IP: %s\n", b.IP) fmt.Printf(" Rule: %s\n", b.RuleName) fmt.Printf(" Reason: %s\n", b.Reason) fmt.Printf(" Created: %v\n", b.CreatedAt) fmt.Printf(" Expires: %v\n", b.ExpiresAt) fmt.Printf(" Action: %s\n", b.Action) } ``` ## Best Practices ### Rule Design 1. **Use Thresholds**: Avoid blocking on single events ```yaml # Good - requires multiple matches threshold: 10 threshold_window: 60 # Bad - blocks on first match threshold: 1 ``` 2. **Set Appropriate Durations**: Match severity to block duration ```yaml # Low severity: short blocks duration: 15m # High severity: longer blocks duration: 24h ``` 3. **Combine Actions**: Use logging with blocking ```yaml actions: - type: iptables_block config: duration: 1h - type: iptables_log config: prefix: "BLOCKED: " ``` ### Whitelist Management 1. **Always whitelist critical infrastructure**: * Gateways and routers * DNS servers * Management IPs * Monitoring systems 2. **Include your own IPs**: * SSH access IPs * Admin workstations * CI/CD systems ### Testing 1. **Start with dry-run mode**: ```go config.DryRun = true ``` 2. **Monitor logs during initial deployment**: ```go config.Verbose = true ``` 3. **Test with known traffic**: ```bash # Generate test traffic nmap -sS target_ip # Verify block was created sudo iptables -L NETCAP -n -v ``` ### Production Deployment 1. **Monitor statistics regularly** 2. **Review blocked IPs periodically** 3. **Keep whitelists up to date** 4. **Set up alerting for high block rates** 5. **Log all firewall actions for audit** ## Troubleshooting ### Manager Creation Fails ``` Error: failed to initialize iptables (IPv4): ... ``` **Solutions**: * Verify iptables is installed: `which iptables` * Check permissions: Run as root or with `CAP_NET_ADMIN` * Verify kernel modules: `lsmod | grep xt_` ### Rules Not Blocking 1. **Check firewall manager is set**: ```go if engine.GetFirewallManager() == nil { log.Warn("Firewall manager not configured") } ``` 2. **Verify action configuration**: ```yaml actions: - type: iptables_block # Not "block" or "iptables-block" config: target: source # Not "src" alone at top level ``` 3. **Check whitelist**: ```go // IP might be whitelisted config.Whitelist ``` ### Blocks Not Expiring 1. **Verify cleanup is running**: ```go // Check verbose logs for cleanup messages config.Verbose = true ``` 2. **Check expiration time**: ```go blocks := manager.GetActiveBlocks() for _, b := range blocks { fmt.Printf("Expires: %v (in %v)\n", b.ExpiresAt, time.Until(b.ExpiresAt)) } ``` ### Rules Persist After Shutdown If rules remain after unclean shutdown: ```bash # Manually flush NETCAP chain sudo iptables -F NETCAP sudo ip6tables -F NETCAP # Remove jump rules sudo iptables -D INPUT -j NETCAP sudo iptables -D FORWARD -j NETCAP sudo iptables -D OUTPUT -j NETCAP # Delete chain sudo iptables -X NETCAP ``` ### Viewing iptables Rules ```bash # List NETCAP chain with details sudo iptables -L NETCAP -n -v --line-numbers # List all chains showing NETCAP jumps sudo iptables -L -n -v | grep -A5 NETCAP # Watch for changes watch -n1 'sudo iptables -L NETCAP -n' ``` ## Next Steps * Review [RULES\_ENGINE.md](/master/rules_engine) for detection rule syntax * See [FILTERING.md](https://github.com/dreadl0ck/netcap/blob/master/docs/FILTERING.md) for expression syntax * Check example rules in [configs/firewall-rules.yml](https://github.com/dreadl0ck/netcap/blob/master/configs/firewall-rules.yml) * Explore [injection rules](https://github.com/dreadl0ck/netcap/blob/master/configs/injection-rules.yml) for real-time packet actions # Python Integration Read Netcap Audit records from Python ## Source Code The Python library for interacting with netcap audit records has been published here: {% embed url="" %} ## Usage ### Read into python dictionary Currently it is possible to retrieve the audit records as python dictionary: ```python #!/usr/bin/python import pynetcap as nc reader = nc.NCReader('pcaps/HTTP.ncap.gz') reader.read(dataframe=False) print("RECORDS:") print(reader.records) ``` ### Read into pandas dataframe Retrieving the audit records as pandas dataframe: ```python #!/usr/bin/python import pynetcap as nc reader = nc.NCReader('pcaps/HTTP.ncap.gz') reader.read(dataframe=True) print("[INFO] completed reading the audit record file:", reader.filepath) print("DATAFRAME:") print(reader.df) ``` # Changelog Detailed Version History Information ## v0.5 - April 2020 ### Fixed * multiple bugs in the stream reassembly * several panics during parsing in gopacket ### Changed * CLI interface refactored: single binary app with subcommands, stripped size \~**17MB** * Updated units tests * Documentation updates * Updated Docker containers for **Ubuntu** and **Alpine** * Compiled with **Go 1.14.2** * removed custom audit records Link-, Network- and TransportFlow ### New Features * **Maltego** integration * **File** audit records * **Diameter** protocol audit records * **SMTP** audit records * **POP3** support for extracting Mails * **JA3S** support and separate audit record for **TLSServerHello** * New configuration options: via **environment variables** or **configuration** file * Resolvers package for **Geolocation**, **DNS** and **Service** lookups and **whitelisting** * Deep Packet Inspection via **nDPI** and **libprotoident** * **DeviceProfile** Audit records, to capture the behavior of a single device within a traffic dump * Added an integration for **bash-completion** support # Troubleshooting Look behind the curtain ## First Aid * are all files in place? * does the current user have sufficient rights to access them? * does the current user have sufficient rights to access the current working directory? ## Pitfalls * the go flag package implementation only allows to set boolean value using the **-name=value** syntax (e.g: **-debug=true**), but strings can be set also using a space instead with the **-name value** syntax (e.g: **-read traffic.pcap**) ## Debug Mode Use the **-debug** flag to generate debug logs. The files will be created in the directory from which the netcap process has been started. The reassembly engine logs data into **reassembly.log** and all other debug messages go into **debug.log.** > Remember that netcap logs packet decoding errors into **errors.log!** ## Advanced Debugging In order to use advanced debugging features, you will need to recompile the code and make a few changes. Some primitives in the core library have a second implementation that spawns an additional goroutine for each invocation to time out the call. This is useful to debug hangs in the reassembly or due to invocation of external code, for example the DPI integration of **nDPI** and **libprotoident**. List of \*Timeout primitives: * handlePacketTimeout(p \*packet) in collector.go * AssembleWithContextTimeout(...) in tcpConnection.go * GetProtocolsTimeout(packet gopacket.Packet) in dpi.go Simply replace the calls to the original versions with the \*Timeout primitives, and if one of those will block for longer than the configured thresholds, the call will be interrupted and an error logged. ## Race Detection Builds To debug synchronization problems and data races you can compile a version with the **-race** flag set for the go compiler and see if the program crashes due to a race condition. There is a command implemented for that in the build scripts: ``` $ zeus install-race ``` # Unit Tests Netcap has tests for its core functionality ## Prerequisites Some of the tests operate on a dump file that is not in the repository. You can download it with: ``` $ zeus download-test-pcap ``` which will basically just invoke: ``` wget https://weberblog.net/wp-content/uploads/2020/02/The-Ultimate-PCAP.7z ``` Now unpack the file and move it to the tests folder in the project root. ## Unit Tests Unit tests have been implemented for parts of the core functionality. Currently there are basic tests for reading pcap data from files and traffic live from an interface, as well as tests and benchmarks for common utility functions, such progress displaying and time conversions. The tests and benchmarks can be executed from the repository root by executing the following from the project root: ``` $ go test -v ./... === RUN TestCountRecords --- PASS: TestCountRecords (0.18s) === RUN TestReader --- PASS: TestReader (0.01s) === RUN TestWriter --- PASS: TestWriter (0.06s) PASS ok github.com/dreadl0ck/netcap 0.862s ? github.com/dreadl0ck/netcap/cmd [no test files] ? github.com/dreadl0ck/netcap/cmd/agent [no test files] ? github.com/dreadl0ck/netcap/cmd/capture [no test files] ? github.com/dreadl0ck/netcap/cmd/collect [no test files] ? github.com/dreadl0ck/netcap/cmd/dump [no test files] ? github.com/dreadl0ck/netcap/cmd/export [no test files] ? github.com/dreadl0ck/netcap/cmd/label [no test files] ? github.com/dreadl0ck/netcap/cmd/proxy [no test files] ? github.com/dreadl0ck/netcap/cmd/split [no test files] ? github.com/dreadl0ck/netcap/cmd/transform [no test files] ? github.com/dreadl0ck/netcap/cmd/util [no test files] === RUN TestCollectPCA done in 2.595847118s --- PASS: TestCollectPCAP (2.60s) PASS ok github.com/dreadl0ck/netcap/collector 3.242s === RUN TestCorruptedWriter TestCorruptedWriter: delimited_test.go:46: Put record returned expected error: BAD --- PASS: TestCorruptedWriter (0.00s) === RUN TestGoodWriter --- PASS: TestGoodWriter (0.00s) === RUN TestCorruptedReader TestCorruptedReader: delimited_test.go:87: Next record returned expected error: unexpected EOF --- PASS: TestCorruptedReader (0.00s) === RUN TestGoodReader --- PASS: TestGoodReader (0.00s) === RUN TestRoundTrip TestRoundTrip: delimited_test.go:136: After writing: buffer="\x04Some\x02of\x04what\x01a\x04fool\x06thinks\x05often\bremains." len=42 --- PASS: TestRoundTrip (0.00s) PASS ok github.com/dreadl0ck/netcap/delimited 0.526s ? github.com/dreadl0ck/netcap/dpi [no test files] ? github.com/dreadl0ck/netcap/decoder [no test files] ? github.com/dreadl0ck/netcap/io [no test files] ? github.com/dreadl0ck/netcap/label [no test files] ? github.com/dreadl0ck/netcap/maltego [no test files] ? github.com/dreadl0ck/netcap/metrics [no test files] ? github.com/dreadl0ck/netcap/resolvers [no test files] === RUN TestMarshal --- PASS: TestMarshal (0.00s) PASS ok github.com/dreadl0ck/netcap/types 0.668s === RUN TestTimeToString --- PASS: TestTimeToString (0.00s) === RUN TestStringToTime --- PASS: TestStringToTime (0.00s) PASS ok github.com/dreadl0ck/netcap/utils 0.932s ``` ## Benchmarks Run the benchmarks using: ``` $ go test -bench=. ./... | grep -E "Bench|pkg" pkg: github.com/dreadl0ck/netcap/collector BenchmarkReadPcapNG-12 1265539 844 ns/op 1249 B/op 1 allocs/op BenchmarkReadPcapNGZeroCopy-12 2028283 640 ns/op 0 B/op 0 allocs/op BenchmarkReadPcap-12 7557667 137 ns/op 106 B/op 1 allocs/op pkg: github.com/dreadl0ck/netcap/types BenchmarkMarshal-12 9817819 110 ns/op 64 B/op 1 allocs/op BenchmarkUnmarshal-12 8703766 134 ns/op 40 B/op 2 allocs/op pkg: github.com/dreadl0ck/netcap/utils BenchmarkTimeToStringOld-12 5283726 229 ns/op 64 B/op 4 allocs/op BenchmarkTimeToString-12 8273997 136 ns/op 80 B/op 3 allocs/op BenchmarkStringToTime-12 8842005 137 ns/op 32 B/op 1 allocs/op BenchmarkStringToTimeFieldsFunc-12 6809409 185 ns/op 32 B/op 1 allocs/op BenchmarkProgressOld-12 54425902 21.0 ns/op 0 B/op 0 allocs/op BenchmarkProgress-12 23389420 45.9 ns/op 16 B/op 2 allocs/op ``` ## Race Detection Tests Run the tests with race detection enabled: ``` $ go test -race -v ./... ``` # Extension Implementing new audit records and features To add support for a new protocol or custom abstraction the following steps need to be performed. ## Protocol Buffer Definitions First, a type definition of the new audit record type must be added to the AuditRecord protocol buffers definitions, as well as a **Type enumeration** following the naming convention with the **NC prefix**. First, make sure you have code generator plugin(s) that NETCAP is using to accelerate the protocol buffer en- and decoding. Get the plugins with: ```go $ go get github.com/gogo/protobuf/... ``` The framework for this can be found here: {% embed url="" %} Recompile the protocol buffers with: ```go $ zeus gen-proto-dev ``` This will create the type definitions for your new audit record in the **types** package. ## Encoder Implementation After recompiling the protocol buffers, a file for the new decoder named after the protocol must be created in the decoder package. The new file must contain a variable created with **CreateLayerEncoder** or **CreateCustomEncoder** depending on the desired decoder type. Lets take a brief look at a very simple **LayerEncoder**, for example for the ARP protocol: ```go package decoder import ( "github.com/dreadl0ck/gopacket" "github.com/dreadl0ck/gopacket/layers" "github.com/dreadl0ck/netcap/types" "github.com/golang/protobuf/proto" ) var arpEncoder = CreateLayerEncoder( types.Type_NC_ARP, layers.LayerTypeARP, func(layer gopacket.Layer, timestamp string) proto.Message { if arp, ok := layer.(*layers.ARP); ok { return &types.ARP{ Timestamp: timestamp, AddrType: int32(arp.AddrType), Protocol: int32(arp.Protocol), HwAddressSize: int32(arp.HwAddressSize), ProtAddressSize: int32(arp.ProtAddressSize), Operation: int32(arp.Operation), SrcHwAddress: arp.SourceHwAddress, SrcProtAddress: arp.SourceProtAddress, DstHwAddress: arp.DstHwAddress, DstProtAddress: arp.DstProtAddress, } } return nil }, ) ``` �Since **ARP** can be decoded by **gopacket** already, all we have to do is check if the packet has the **ARP** layer, and if yes, convert it to the **types.ARP** audit record and return it. The constructor for a **LayerEncoder** needs the type enumeration for the new audit record, as well as the **gopacket.LayerType**, followed by the actual decoder function. This function will be called for every network packet. As you can see, **LayerEncoders** are tied to **gopacket**. If you want to implement custom decoding logic or support for a new protocol, you essentially have two options: * implement protocol decoding in **gopacket**, then use a **LayerEncoder** in netcap * implement protocol decoding in a **CustomEncoder** A **CustomEncoder** works the same way but offers more flexibility for the implementation, like functions for initialisation and teardown. The CustomEncoder constructor signature looks as follows: ```go func CreateCustomEncoder( t types.Type, name string, postinit func(*CustomEncoder) error, handler CustomEncoderHandler, deinit func(*CustomEncoder) error ) *CustomEncoder ``` The **CustomEncoderHandler** will simply receive the raw **gopacket.Packet** and return a **proto.Message**: ```go CustomEncoderHandler = func(p gopacket.Packet) proto.Message ``` Depending on the choice of the decoder type, the new variable must be added to the customEncoderSlice in **decoder/customEncoder.go** or layerEncoderSlice in **decoder/layerEncoder.go**. ## Audit Record Interface Implementation Next, the interface for conversion to CSV and JSON and exporting metrics must be implemented in the types package, by creating a new file with the protocol name and implementing the **types.AuditRecord** interface: ```go // AuditRecord is the interface for basic operations with NETCAP audit records // this includes dumping as CSV or JSON or prometheus metrics // and provides access to the timestamp of the audit record type AuditRecord interface { // returns CSV values CSVRecord() []string // returns CSV header fields CSVHeader() []string // used to retrieve the timestamp of the audit record for labeling Time() string // Src returns the source of an audit record // for Layer 2 records this shall be the MAC address // for Layer 3+ records this shall be the IP address Src() string // Dst returns the source of an audit record // for Layer 2 records this shall be the MAC address // for Layer 3+ records this shall be the IP address Dst() string // increments the metric for the audit record Inc() // returns the audit record as JSON JSON() (string, error) // can be implemented to set additional information for each audit record // important: // - MUST be implemented on a pointer of an instance // - the passed in packet context MUST be set on the Context field of the current audit record SetPacketContext(ctx *PacketContext) } ``` If the new protocol contains sub-structures, functions to convert them to strings need to be implemented as well. Take a look at other decoders that have lots of substructures, for example **DNS**. ## Add Initializer Finally, the **InitRecord(typ types.Type) (record proto.Message)** function in netcap.go needs to be updated, to initialize the structure for the new type. # Downloads A collection of cheatsheets and useful resources ## Releases You can find the latest release on the releases page on GitHub: {% embed url="" %} ## Publications In this paper, we explore Graph based analysis using Maltego to visualise data from NETCAP during a forensic investigation: ### Thesis {% file src="/files/xPUuQyCk6hUWsRkcOvG0" %} Implementation and Evaluation of secure and scalable anomaly-based Network Intrusion Detection {% endfile %} ### Thesis Presentation ### SecurIT Cup 2018 Presentation ## External Publications The authors used the framework to process their recorded PCAP dumps: {% embed url="" %} ## Cheatsheets ### List of all supported protocols and fields {% file src="/files/8rLoD6hg1Xz98kjUvYy6" %} ### Command Cheatsheet {% file src="/files/J6ITgt9CEUck9rxcyBog" %} --- [Next Page](/llms-full.txt/1)