To test the increasing and decreasing of capacity

Rationale

Do we request horizontal scaling from all CNF-s?

Most (data plane, signalling, etc) but not all (eg OSS)

Test if the Helm chart is published

Rationale

Test if the Helm chart is valid

Rationale

Test if the Helm deploys

Rationale

Test if the install script uses Helm v3

Rationale

To test if the CNF can perform a rolling update

Rationale

To check if a CNF version can be downgraded through a rolling_version_change

Rationale

It is not clear what is the difference between a rolling downgrade and a rolling version change.

A: Defined in the external docs in the usage guide. Some these are relevant for a ReplicaSet some of them are for a Deployment.

Maybe when you request an arbitrary version?

To check if a CNF version can be downgraded through a rolling_downgrade

Rationale

To check if a CNF version can be rolled back rollback

Rationale

To check if the CNF is compatible with different CNIs

Rationale

This covers only the default CNI, does not cover the metaplugin part.

Need additional tests for cases with multiple interfaces.

(PoC) To check if a CNF uses Kubernetes alpha APIs

Rationale

Alpha API-s are not recommended by ra2.k8s.012. It fails with alpha

PoC: it might happen that these testcases are removed from the Testsuite and this will be not part of the CNF certification.  Probably will be a bonus case.

To check if the CNF has a reasonable image size

Rationale

It passes if the image size is smaller than 5GB.

A: Whenever it is possible tests are configurable or parameters can be overwritten from the outside. This will be part of the CNF Certification. Valid for each image referred from the Helm chart.

Ok but should be documented or configurable?

issue to clarify name

should read "pod image size"

To check if the CNF have a reasonable startup time

Rationale

It is not clear what reasonable startup time is. It is about the startup time of the microservices inside the CNF.

Should be Check if all the Pods in the CNF have a reasonable startup time.

A: Reasonable time is 60 sec.

Ok but should be documented or configurable?

issue to clarify name

should read "pod startup time"

To check if the CNF has multiple process types within one container

Rationale

Containers in the CNF should have only one process type.

even for exposing an API a separate process is required - should this test if the number of processes is less than a certain number instead?

To check if the CNF exposes any of its containers as a service

Rationale

Service type what?

RA2 mandates that clusters must support Loadbalancer and ClusterIP, and should support Nodeport and ExternalName

Should there be a test for the CNF to use Ingress or Gateway objects as well ?

To check if the CNF has multiple microservices that share a database

Rationale

Clarify rationale? In some cases it is good for multiple Microservices to share a DB, eg when restoring the state of a transaction from a failed service.

Also good to have a shared DB across multiple services for things like HSS etc.

Test if the CNF crashes when node drain and rescheduling occurs. All configuration should be stateless

Rationale

CNF should react gracefully (no loss of context/sessions/data/logs & service continues to run) to eviction and node draining

The statelessness test should be made independent & Should be skipped for stateful pods eg Dns

"crashes" actually means that either the liveness or readiness probe fails - this should be made explicit and the presence of probes should be made mandatory - added issue in RA2

Needed - but replace "crash" with "react gracefully" (no loss of context/sessions/data/logs & service continues to run)

issue: Statelessness test should be separate

To test if the CNF uses a volume host path

Rationale

should pass if the cnf doesn't have a hostPath volume

What's the rationale?

- A: When a cnf uses a volume host path or local storage it makes the application tightly coupled to the node that it is on.

Check this with RA2 - already ok, already in RA2

To test if the CNF uses local storage

Rationale

should fail if local storage configuration found

What's the rationale?

ok, add to RA2 (attach to previous)

To test if the CNF uses elastic volumes

Rationale

should pass if the cnf uses an elastic volume

What's an elastic volume? Does this mean Ephemeral? Or is this an AWS-specific test?

There should be a definition of what an elastic volume is (besides ELASTIC_PROVISIONING_DRIVERS_REGEX)

To test if the CNF uses a database with either statefulsets, elastic volumes, or both

Rationale

A database may use statefulsets along with elastic volumes to achieve a high level of resiliency. Any database in K8s should at least use elastic volumes to achieve a minimum level of resilience regardless of whether a statefulset is used. Statefulsets without elastic volumes is not recommended, especially if it explicitly uses local storage. The least optimal storage configuration for a database managed by K8s is local storage and no statefulsets, as this is not tolerant to node failure.

There should be a definition of what an elastic volume is (besides ELASTIC_PROVISIONING_DRIVERS_REGEX)

Test if the CNF crashes when network latency occurs

Rationale

How is this tested? Where is the test running? Some traffic against a service? Latency should be configurable (default is 2s)?

What should happen if latency is exceeded? Should this be more stringent than "not crashing?"

What is the expectation? (not crashing = not exit with error code or (better) not stopping to process traffic)

A: Explanation added to https://github.com/cncf/cnf-testsuite/blob/main/USAGE.md#heavy_check_mark-test-if-the-cnf-crashes-when-network-latency-occurs

Check this with RA2 - should be ok

Needed but needs clarification

issue on defining "crashing - it's probes

Test if the CNF crashes when disk fill occurs

Rationale

What is the expectation? (not crashing = not exit with error code or (better) not stopping to process traffic)

RM/RA2 should add infra monitoring recommendation for disk usage alerting

Needed

issue on defining "crashing - it's probes

Test if the CNF crashes when pod delete occurs

Rationale

Needed

issue on defining "crashing - it's probes

Test if the CNF crashes when pod memory hog occurs

Rationale

What is the expectation? (not crashing = not exit with error code or (better) not stopping to process traffic)

title should read "CNF pod runs out of memory"?

RA2 should add recommendation to add pod memory reservation: 

A CNF can fail due to running out of memory. This can be mitigated by using two levels of memory policies (pod level and node level) in K8s. If the memory policies for a CNF are not fine grained enough, the CNFs out-of-memory failure blast radius will result in using all of the system memory on the node.

Test if the CNF crashes when pod io stress occurs

Ratoinale

What is the expectation? (not crashing = not exit with error code or (better) not stopping to process traffic)

title should read "pod disk I/O"

Needed

issue on defining "crashing

Test if the CNF crashes when pod network corruption occurs

Rationale

It is not clear what network corruption is in this context. What is the expectation? (not crashing = not exit with error code or (better) not stopping to process traffic)

Rationale explains traffic manipulation: 

A higher quality CNF should be resilient to a lossy/flaky network. This test injects packet corruption on the specified CNF's container by starting a traffic control (tc) process with netem rules to add egress packet corruption.

Test if the CNF crashes when pod network duplication occurs

Rationale

To test if there is a liveness entry in the Helm chart

Rationale

Liveness probe should be mandatory, but RA2 does not mandate Helm at the moment. (it's in the pod definition rather than helm - maybe fix the title)

RA2 now mandates helm3 - it's the pod definition - added issue to recommend probes in RA2 CH4

To test if there is a readiness entry in the Helm chart

Rationale

Readiness probe should be mandatory, but RA2 does not mandate Helm at the moment. (it's in the pod definition rather than helm - maybe fix the title)

RA2 now mandates helm3 - it's the pod definition - added issue to recommend probes in RA2 CH4

To check if logs are being sent to stdout/stderr

Rationale

optional, as there's no way to accurately figure out if we're missing something from stdout/stderr 

title reads "instead of a log file"

RA2 should recommend that the application streams logs out of stdout/stderr

To check if prometheus is installed and configured for the cnf

Rationale

There is a chapter for Additional required components (4.10), but without any content. should ra2 mandate prometheus?

A: All the PaaS components are optionally tested, as bonus tests.

Check this with RA2

RM/RA right now doesn't require specific PaaS tools

To check if logs and data are being routed through fluentd

Rationale

There is a chapter for Additional required components (4.10), but without any content. should ra2 mandate fluent?

A: All the PaaS components are optionally tested, as bonus tests.

Check this with RA2

To check if Open Metrics is being used and or compatible.

Rationale

There is a chapter for Additional required components (4.10), but without any content. should ra2 mandate open metrics?

A: All the PaaS components are optionally tested, as bonus tests.

Check this with RA2

To check if tracing is being used with Jaeger

Rationale

There is a chapter for Additional required components (4.10), but without any content. should ra2 mandate jaeger?

A: All the PaaS components are optionally tested, as bonus tests.

Check this with RA2

To check if a CNF is using container socket mounts

To check if containers are using any tiller images

To check if any containers are running in privileged mode

Rationale

To check if a CNF is running services with external IP's

To check if any containers are running as a root user

Rationale

To check if any containers allow for privilege escalation

Rationale

To check if an attacker can use a symlink for arbitrary host file system access

Rationale

To check if there are service accounts that are automatically mapped

Rationale

what is the expectation?

Application Credentials: Developers store secrets in the Kubernetes configuration files, such as environment variables in the pod configuration. Such behavior is commonly seen in clusters that are monitored by Azure Security Center. Attackers who have access to those configurations, by querying the API server or by accessing those files on the developer’s endpoint, can steal the stored secrets and use them.

Check if the pod has sensitive information in environment variables, by using list of known sensitive key names. Check if there are configmaps with sensitive information.

Remediation: Use Kubernetes secrets or Key Management Systems to store credentials.

See more at ARMO-C0012

To check if there is a host network attached to a pod

Rationale

To check if there are service accounts that are automatically mapped

Rationale

Disable automatic mounting of service account tokens to PODs either at the service account level or at the individual POD level, by specifying the automountServiceAccountToken: false. Note that POD level takes precedence.

See more at ARMO-C0034

To check if there is an ingress and egress policy defined

Rationale

ok - maybe more stringent?

A: There is an answer here: https://github.com/cncf/cnf-testsuite/issues/1282#issuecomment-1081228008

Check this with RA2

To check if there are any privileged containers

Rationale

To check for insecure capabilities

Rationale

what is the expectation?

To check for dangerous capabilities

Rationale

what is the expectation?

To check if namespaces have network policies defined

Rationale

To check if containers are running with non-root user with non-root membership

Rationale

To check if containers are running with hostPID or hostIPC privileges

Rationale

To check if security services are being used to harden containers

Rationale

what services? should be configurable or optional

Linux Hardening: Check if there is AppArmor, Seccomp, SELinux or Capabilities are defined in the securityContext of container and pod. If none of these fields are defined for both the container and pod, alert.

Remediation: In order to reduce the attack surface, it is recommended to harden your application using security services such as SELinux®, AppArmor®, and seccomp. Starting from Kubernetes version 22, SELinux is enabled by default.

Read more at ARMO-C0055

To check if containers have resource limits defined

Rationale

To check if containers have immutable file systems

Rationale

To check if containers have hostPath mounts

Rationale

To check if containers are using labels

To test if there are versioned tags on all images using OPA Gatekeeper

Rationale

To test if there are any (non-declarative) hardcoded IP addresses or subnet masks

Rationale

To test if there are node ports used in the service configuration

Rationale

To test if there are host ports used in the service configuration

Rationale

To test if there are any (non-declarative) hardcoded IP addresses or subnet masks in the K8s runtime configuration

Rationale

To check if a CNF version uses immutable configmaps

Rationale

Test if the CNF crashes when pod dns error occurs