# Local Manual Testing

This directory contains simple examples for deploying MinIO, PostgreSQL, and Redpanda instances in a Kubernetes cluster. These configurations are intended for testing purposes only.

## Contents

- **minio.yaml**: Deploys a MinIO object storage service.
- **postgres.yaml**: Deploys a PostgreSQL database.
- **redpanda.yaml**: Deploys a Redpanda Kafka-compatible streaming platform.
- **datagen.yaml**: Deploys a Datagen service to generate test data for Redpanda.

> **Note**: These deployments are for testing purposes. Do not use these configurations in a production environment without further adjustments.

## Deployment Steps

0. Create a namespace:

    ```bash
    kind create cluster
    kubectl create namespace materialize # or use an existing namespace
    ```

1. Deploy the services to your Kubernetes cluster:

    ```bash
    kubectl apply -f misc/helm-charts/testing/minio.yaml
    kubectl apply -f misc/helm-charts/testing/postgres.yaml
    ```

2. Monitor the deployments to ensure the pods are running:

    ```bash
    kubectl get pods -w -n materialize
    ```

## Node labels for ephemeral storage

When running Materialize locally on Kubernetes (e.g., Docker Desktop, Minikube, Kind), specific node labels need to be added to ensure that pods are scheduled correctly. These labels are required for the pod to satisfy node affinity rules defined in the deployment.

```sh
kubectl get nodes --show-labels
```

If the required labels are missing, add them to the node by running:

```sh
kubectl label node <node-name> materialize.cloud/disk=true
kubectl label node <node-name> workload=materialize-instance
```

After adding the labels, verify that they were successfully applied by running the following command again:

```sh
kubectl get nodes --show-labels
```

## (Optional) Metrics service

Materialize does not require the Kubernetes metrics service to function. However, if `observability.enabled: true` and `observability.podMetrics.enabled: true` are set in the Helm values file, `environmentd` would expect a running metrics service. Without it, and with `observability` disabled, Materialize will still operate, but metrics will not be available in the web console.

For more information, see the [Metrics Server documentation](https://github.com/kubernetes-sigs/metrics-server/blob/master/README.md). To install the metrics server, follow these steps:

First, add the metrics-server Helm repository:

```bash
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm repo update
```

Install the metrics server with TLS disabled and proper address types configured:

```bash
helm install metrics-server metrics-server/metrics-server \
  --namespace kube-system \
  --set args="{--kubelet-insecure-tls,--kubelet-preferred-address-types=InternalIP\,Hostname\,ExternalIP}"
```

As this is a local testing environment, we are disabling TLS for the metrics server to avoid issues with the `environmentd` pod. This is not recommended for production environments.

Get the metrics server pod status:

```bash
kubectl get pods -n kube-system -l app.kubernetes.io/instance=metrics-server
```

Later on, if you need to uninstall the metrics server, run:

```bash
helm uninstall metrics-server -n kube-system
```

## (Optional) Test data ingestion with Redpanda and Datagen

Once everything is set up, you can test data ingestion using Redpanda and Datagen.

```bash
kubectl apply -f redpanda.yaml
kubectl apply -f datagen.yaml
```

After deploying Redpanda and Datagen, you'll need to have a Materialize instance running to consume the data.

Once Materialize is running, connect using `psql` and create a connection to the Redpanda instance:

```sql
-- Create a connection to Redpanda in the 'materialize' namespace
CREATE CONNECTION rp_connection TO KAFKA (
    BROKER 'redpanda.materialize.svc.cluster.local:9092',
    SECURITY PROTOCOL = 'PLAINTEXT'
);
```

Create a source in Materialize that reads data from the topic generated by `datagen`.

```sql
-- Create a source to consume messages from the 'mz_datagen_test' topic
CREATE SOURCE rp_datagen
  FROM KAFKA CONNECTION rp_connection (TOPIC 'mz_datagen_test')
  FORMAT JSON;
```

Create a materialized view to extract and store the data for easy querying.

```sql
-- Create a materialized view to extract 'id' and 'name' fields
CREATE MATERIALIZED VIEW datagen_view AS
SELECT
    (data->>'id')::int AS id,
    data->>'name' AS name
FROM rp_datagen;
```

To check if data is flowing correctly, run:

```sql
-- Query the materialized view to see incoming data
SELECT * FROM datagen_view LIMIT 10;
```

This should display data generated by your `datagen` service.

## Notes

- Adjust resource requests and limits based on your cluster's capacity.
- Ensure your Kubernetes cluster has enough resources to run these services.
- The `minio.yaml`, `postgres.yaml`, and `redpanda.yaml` configurations are minimal examples suitable for local testing environments.