Ethan Walker

404 bio not found

Joined on May 19, 2026

Ethan Walker

Jul 21

Our CI eval gate sent us a token bill. The deterministic one sent nothing.

#llm #evaluation #cicd #devops

5 min read

Want to connect with Ethan Walker?

Create an account to connect with Ethan Walker. You can also sign in below to proceed if you already have an account.

Create Account

Already have an account? Sign in

Ethan Walker

Jul 20

Our few-shot examples came from the eval set. The 0.94 was fiction.

#llm #machinelearning #rag #testing

13 min read

Ethan Walker

Jul 16

Our few-shot examples came from the eval set. The 0.94 was fiction.

#ai #llm #machinelearning #testing

13 min read

Ethan Walker

Jul 14

We gated CI on six open-source LLM eval frameworks. Only two survived the merge queue.

#ai #opensource #llm #evals

14 min read

Ethan Walker

Jul 13

The golden set stopped catching regressions the day traffic changed

#ai #llm #machinelearning #testing

6 min read

Ethan Walker

Jul 8

Your LLM-as-judge disagrees with itself between runs

#ai #cicd #llm #testing

4 min read

Ethan Walker

Jul 8

LLM-as-judge disagrees with itself between runs

#ai #llm #machinelearning #testing

4 min read

Ethan Walker

Jul 7

When an LLM answer is wrong, the trace is where you look. Some tools make that easy.

#observability #ai #opentelemetry #llm

5 min read

Ethan Walker

Jul 5

# A 94% pass rate hid a PII leak in 6 test cases

#ai #llm #evals

5 min read

Ethan Walker

Jul 1

our CI passed. Your agent isn't operator-ready.

#ai #agents #testing #mlops

4 min read

Ethan Walker

Jun 29

The stale eval fixture that passed a broken model

#testing #cicd #python #llmops

4 min read

Ethan Walker

Jun 27

My eval handed me a 0.62 and no idea why. The fix was not a better eval.

7 min read

Ethan Walker

Jun 23

91% pass rate. Gate green. Shipped. Worst regression we had all quarter.

#testing #ci #ai #llm

2 min read

Ethan Walker

Jun 17

We stopped writing eval cases by hand. Now every prod incident becomes one.

#ai #machinelearning #testing #llm

2 min read

Ethan Walker

Jun 10

Your eval criteria are code. Version them like code.

3 min read

Ethan Walker

Jun 8

Datadog dashboards for prompt regression: the panels we actually keep

#ai #llm #devops #cicd

8 min read

Ethan Walker

Jun 3

Switching our LLM-as-judge from 5-class to binary in CI: the patterns we kept

#ai #llm #ci #machinelearning

3 min read

Ethan Walker

May 26

Promptfoo is a CI gate, not an eval framework. Treating it like one cost us $4,200

#ai #machinelearning #llm #python

4 min read

DEV Community

Ethan Walker

Badges

2 Week Community Wellness Streak

1 Week Community Wellness Streak

Writing Debut

Our CI eval gate sent us a token bill. The deterministic one sent nothing.

Want to connect with Ethan Walker?

Our few-shot examples came from the eval set. The 0.94 was fiction.

Our few-shot examples came from the eval set. The 0.94 was fiction.

We gated CI on six open-source LLM eval frameworks. Only two survived the merge queue.

The golden set stopped catching regressions the day traffic changed

Your LLM-as-judge disagrees with itself between runs

LLM-as-judge disagrees with itself between runs

When an LLM answer is wrong, the trace is where you look. Some tools make that easy.

# A 94% pass rate hid a PII leak in 6 test cases

our CI passed. Your agent isn't operator-ready.

The stale eval fixture that passed a broken model

My eval handed me a 0.62 and no idea why. The fix was not a better eval.

91% pass rate. Gate green. Shipped. Worst regression we had all quarter.

We stopped writing eval cases by hand. Now every prod incident becomes one.

Your eval criteria are code. Version them like code.

Datadog dashboards for prompt regression: the panels we actually keep

Switching our LLM-as-judge from 5-class to binary in CI: the patterns we kept

Promptfoo is a CI gate, not an eval framework. Treating it like one cost us $4,200