Utilize Our New Kafka Output and Enrichment Features with 5.2
November 10, 2021
New Features
User-Defined Metadata Enrichment for IP addresses. For more details see: User-Defined Metadata Enrichment
User-Defined Metadata Enrichment can be used to override IP address and hostname values to obfuscate individual IPs and hostnames where such privacy is required. An example is provided HERE (see the note).
Cached enrichment features for DNS/hostnames, Maxmind ASN and GeoIP, RiskIQ ASN and Threats, and User-Defined Metadata have been combined into an all-new combined enrichment module.
Hostname/DNS, RiskIQ Threat/IP Reputation and Maxmind GeoIP enrichment features can be scoped to a subset of IP addresses by specifying specific Autonomous Systems or CIDRs. For more details see: Scoping Enrichment with Include/Exclude
Better performance is achieved by fetching multiple enrichment attributes concurrently.
Cache maintenance tasks are handled asynchronously. This eliminates the throughput impact of cache purges, especially when a high number of IP addresses are cached.
The contents of the cache are now expired using a configurable time-to-live (TTL).
The enrichment features which read external files can reload those files, refreshing values, without having to restart the collector.
Kafka Output: topic name is now configurable by setting the EF_FLOW_OUTPUT_KAFKA_TOPIC option. The default value is elastiflow-flow-codex.
Kafka Output: now allows a partition key to be specified by setting EF_FLOW_OUTPUT_KAFKA_PARTITION_KEY. The default is set to flow.export.ip.addr.
Learn more from the changelog.