Network Observability eBPF Agent

The Network Observability eBPF Agent allows collecting and aggregating all the ingress and egress flows on a Linux host (required a Kernel 5.8+ with eBPF enabled).

• How to build
• How to configure
• How to run
• Development receipts
• Known issues
• Frequently-asked questions
• Troubleshooting

How to build

To build the agent image and push it to your Docker / Quay repository, run:

# compile project
make build

# build the default image (quay.io/netobserv/netobserv-ebpf-agent:main):
make image-build

# push the default image (quay.io/netobserv/netobserv-ebpf-agent:main):
make image-push

# build and push on your own quay.io account (quay.io/myuser/netobserv-ebpf-agent:dev):
IMAGE_ORG=myuser VERSION=dev make images

# build and push on a different registry
IMAGE=dockerhub.io/myuser/plugin:tag make images

How to configure

The eBPF Agent is configured by means of environment variables. Check the configuration documentation for more details.

How to run

The NetObserv eBPF Agent is designed to run as a DaemonSet in OpenShift/K8s. It is triggered and configured by our Network Observability Operator.

Anyway you can run it directly as an executable from your command line:

export TARGET_HOST=...
export TARGET_PORT=...
sudo -E bin/netobserv-ebpf-agent

We don't recommend using the agent's IPFIX exporter mode as it is not actively maintained (if you're interested in maintaining it, let us know!). Note that flowlogs-pipeline can also generate IPFIX exports, so a valid way to get IPFIX data is to export to flowlogs-pipeline (via GRPC, Kafka or direct-flp) and then configure IPFIX within flowlogs-pipeline.

A simple way to try the agent is using the direct-flp export mode, printing directly to stdout:

Given the following file flp-config.json:

{
	"pipeline":[
		{"name": "writer","follows": "preset-ingester"}
	],
	"parameters":[
		{"name": "writer","write": {"type": "stdout"}}
	]
}

Run:

export FLP_CONFIG=$(cat flp-config.json)
export EXPORT="direct-flp"
sudo -E bin/netobserv-ebpf-agent

For more information about configuring flowlogs-pipeline, please refer to its documentation.

To deploy locally, use instructions from flowlogs-dump (like tcpdump).
To deploy it as a Pod, you can check the deployment examples.

The Agent needs to be executed either with:

1. The following Linux capabilities (recommended way): BPF, PERFMON, NET_ADMIN, SYS_RESOURCE. If you deploy it in Kubernetes or OpenShift, the container running the Agent needs to define the following securityContext:

   securityContext:
     runAsUser: 0
     capabilities:
       add:
         - BPF
         - PERFMON
         - NET_ADMIN
         - SYS_RESOURCE

   (Please notice that the runAsUser: 0 is still needed).
2. Administrative privileges. If you deploy it in Kubernetes or OpenShift, the container running the Agent needs to define the following securityContext:

   securityContext:
     privileged: true
     runAsUser: 0

   This option is only recommended if your Kernel does not recognize some of the above capabilities. We found some Kubernetes distributions (e.g. K3s) that do not recognize the BPF and PERFMON capabilities.

Here is a list of distributions where we tested both full privileges and capability approaches, and whether they worked (✅) or did not (❌):

Running on KinD cluster

How to run on kind cluster

Install KinD and the ebpf agent and export KUBECONFIG

make create-and-deploy-kind-cluster
export KUBECONFIG=$(pwd)/scripts/kubeconfig

Deleting the kind cluster

In order to delete the kind cluster:

make destroy-kind-cluster

Development receipts

How to regenerate the eBPF Kernel binaries

The eBPF program is embedded into the pkg/ebpf/bpf_* generated files. This step is generally not needed unless you change the C code in the bpf folder.

If you have Docker installed, you just need to run:

make docker-generate

If you can't install docker, you can install locally the following packages, then run make generate:

dnf install -y kernel-devel make llvm clang glibc-devel.i686
make generate

Regularly tested on Fedora.

Running end-to-end tests

Refer to the specific documentation: e2e readme

Known issues

Extrenal Traffic in Openshift (OVN-Kubernetes CNI)

For egress traffic, you can see the source Pod metadata. For ingress traffic (e.g. an HTTP response), you see the destination Host metadata.

Frequently-asked questions

Where is the collector?

As part of our Network Observability solution, the eBPF Agent is designed to send the traced flows to our Flowlogs Pipeline component.

In addition, we provide a simple GRPC+Protobuf library to allow implementing your own collector. Check the packet counter code for an example of a simple collector using our library.

Troubleshooting

Deployed as a Kubernetes Pod, the agent shows permission errors in the logs and can't start

In your deployment file, make sure that the container runs as the root user (runAsUser: 0) and with the granted capabilities or privileges (see how to run section).

The Agent doesn't work in my Amazon EKS puzzle

Despite Amazon Linux 2 enables eBPF by default in EC2, the EKS images are shipped with disabled eBPF.

You'd need either:

1. Provide your own AMI configured to work with eBPF
2. Use other Linux distributions that are shipped with eBPF enabled by default. We have successfully tested the eBPF Agent in EKS with the Bottlerocket Linux distribution, without requiring any extra configuration.

Licenses

Two licenses are used for the source code in this repository:

• GPL v2 covers the eBPF code in ./bpf directory.
• Apache v2 covers everything else.

Discussions and contributions

Discussions related to NetObserv are welcome on GitHub discussions as well as on the #netobserv-project channel from CNCF slack.

If you'd like to reach out because you've found a security issue, please do not share sensitive details publicly. Please follow the instructions described on the Red Hat Customer Portal.

Refer to the NetObserv projects contribution guide for more details on contributions.