From Zero to Flow: Setting Up ElastiFlow in Minutes

January 23, 2024

From Zero to Flow: Setting Up ElastiFlow in Minutes

What’s ElastiFlow?

ElastiFlow offers complete network visibility. ElastiFlow collects unsampled flow data and standardizes data from network devices, including industry-leading vendor-specific field support. As well as being easy to set up, it's also easy to use and requires no special training. ElastiFlow takes an open data approach to network observability, meaning you remain in control of your data and can economically leverage this data. ElastiFlow provides powerful analytics tools that allow you to troubleshoot problems, plan capacity, and investigate security incidents.

Getting Ready

Before diving in, ensure you have:

  • Docker v24.0.7

  • Docker-compose v2.23.0

  • A router that supports NetFlow, IPFIX, or sFlow

About Licenses

This guide uses a community license. If your network is large and you aim to collect more data quickly, visit ElastiFlow’s Subscription Page to select the right license for your needs.

Setting Up with Docker-Compose

Your docker-compose.yml file will serve as the core of this setup. Below is a generic setup that brings together Elasticsearch, Kibana, and the ElastiFlow flow collector:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 version: '3' services: es_master1: image: docker.elastic.co/elasticsearch/elasticsearch:8.10.4 restart: unless-stopped hostname: es_master1 ulimits: memlock: soft: -1 hard: -1 nofile: soft: 131072 hard: 131072 nproc: 8192 fsize: -1 network_mode: host volumes: - /var/lib/elasticsearch:/usr/share/elasticsearch/data environment: ES_JAVA_OPTS: '-Xms2g -Xmx2g' cluster.name: elastiflow node.name: es_master1 bootstrap.memory_lock: 'true' network.bind_host: http.port: 9200 http.publish_port: 9200 transport.port: 9300 transport.publish_port: 9300 cluster.initial_master_nodes: 'es_master1' indices.query.bool.max_clause_count: 8192 search.max_buckets: 250000 action.destructive_requires_name: 'true' xpack.security.enabled: 'false' kibana: image: docker.elastic.co/kibana/kibana:8.10.4 restart: unless-stopped hostname: NODE_NAME network_mode: host ports: - 5601:5601/tcp environment: TELEMETRY_OPTIN: 'false' TELEMETRY_ENABLED: 'false' SERVER_NAME: 'NODE_NAME' SERVER_HOST: '' SERVER_PORT: 5601 SERVER_MAXPAYLOADBYTES: 8388608 ELASTICSEARCH_HOSTS: 'http://localhost:9200' ELASTICSEARCH_REQUESTTIMEOUT: 132000 ELASTICSEARCH_SHARDTIMEOUT: 120000 ELASTICSEARCH_SSL_VERIFICATIONMODE: 'none' KIBANA_AUTOCOMPLETETIMEOUT: 3000 KIBANA_AUTOCOMPLETETERMINATEAFTER: 2500000 VIS_TYPE_VEGA_ENABLEEXTERNALURLS: 'true' XPACK_MAPS_SHOWMAPVISUALIZATIONTYPES: 'true' XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: 'ElastiFlow_0123456789_0123456789_0123456789' flow-collector: image: elastiflow/flow-collector:6.3.6 container_name: flow-collector restart: 'unless-stopped' network_mode: 'host' volumes: - /etc/elastiflow:/etc/elastiflow environment: EF_LICENSE_ACCEPTED: 'true' EF_FLOW_SERVER_UDP_IP: '' EF_FLOW_SERVER_UDP_PORT: 9995 EF_OUTPUT_ELASTICSEARCH_ENABLE: 'true' EF_OUTPUT_ELASTICSEARCH_ECS_ENABLE: 'true' EF_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE: 'start' EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD: 'rollover'

Make sure to tailor this docker-compose file to fit your specific needs. For detailed configuration guidance, you can review the ElastiFlow Configuration Reference.

When running your Elasticsearch container, you might encounter the following bootstrap check failure that prevents the service from starting.

1 2 3 bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name" :"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name" :"es_master1","elasticsearch.cluster.name":"elastiflow

To solve this, increase the vm.max_map_countfield in your sysctl.conf, then restart the Elasticsearch container.

1 echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.conf sudo sysctl -p

Wrapping It Up

With your configurations in place, initiate the services using the command docker-compose up. Remember to configure your supported router to forward flows to port 9995.

Kibana Dashboards: To enhance your monitoring experience, download and upload ElastiFlow’s pre-defined ECS (Elastic Common Schema) Kibana dashboards. These dashboards offer a detailed view of your network flows, aiding in identifying patterns, anomalies, or potential issues more effectively. Access and integrate these ECS dashboards by following the instructions in the official ElastiFlow Kibana Dashboards Guide.

Final Words

As a software engineer on ElastiFlow, I’ve experienced firsthand how this tool has empowered businesses. It provides real-time insights, streamlines network analysis, and simplifies troubleshooting. Whether you’re in banking, healthcare, or any other sector, ElastiFlow can make a big difference. Dive in and explore what your network data can reveal! If you need help, join the ElastiFlow Slack Community https://elastiflowcommunity.slack.com

(Note: This post is an edit of my original which was published on Medium in October, 2023)