PromptAtlasRESEARCH / 01

UCL MSc CSML / EVOLUTIONARY LLM OPTIMISATION

Don’t find one prompt.
Map the possibilities.

An evolutionary search for creative problem-solving.

Explore a diverse archive of prompts, inspect how each candidate is evaluated, and see where quality meets diversity. Built from the project’s actual experiment logs.

PythonLLM APIsUMAPMAP-ElitesMulti-objective evaluation
Richard Huang / UCL MSc Computer Science & Machine Learning“Illuminating Prompt Space for Discovering Creativity in LLMs with MAP-Elites”
324evaluated search candidates
3recorded MacGyver problems
15saved archive snapshots
2baseline strategies

02 / LLM EVALUATION PIPELINE

From prompt genes to measured outcomes.

A Python research pipeline connecting generation, LLM APIs, evaluation and evolutionary selection.

SOURCE-BACKED WORKFLOW

PROMPT GENERATION

Search over structured instructions.

A context-free grammar builds prompts from role, thinking, creativity, tool combination and response format. Mutating components changes the search candidate while retaining the problem.

View Python implementation ↗
cfg_generator.pyPYTHON · CORE LOGIC
Select a parent from the archive → mutate instructions → evaluate again → update its niche

QUALITY / CONVERGENT CREATIVITY

Judge the complete solution.

Quality = (Safety × Feasibility × Effectiveness)⅓

The evaluator LLM returns integer ratings from 1 to 5. Their geometric mean becomes the quality objective. These are model judgments, not human-validated outcome labels.

Quality computation ↗

DIVERSITY / DIVERGENT CREATIVITY

Measure semantic variation.

H = −Σ pk log₂ pk

Generate variations of the first solution step, group them through bidirectional NLI entailment, and calculate cluster entropy in bits. It is a proxy for diversity, not proof of creativity.

Semantic entropy calculation ↗

03 / EXPERIMENTAL RESULTS

Compare the evidence.
Keep the tradeoffs visible.

Recorded Baselines and MAP-Elites results, with population sizes and selection scope stated explicitly.

Download comparison CSV ↓

Convergent quality

MEAN SCORE / 5
012345

Geometric mean of safety, feasibility and effectiveness ratings.

Semantic diversity

MEAN ENTROPY / BITS
00.51.01.52.02.53.0

Entropy of semantic clusters in first-step solution variations. Higher is more varied under this evaluator.

The search evaluates 108 candidates per problem; each Baseline has 20 trials. Final archives are selected survivors. These are descriptive comparisons, not equal-budget or matched-population benchmarks.

Quality through the search

SAVED ARCHIVE MEAN

Diversity through the search

SAVED ARCHIVE MEAN
MAP-Elites generation logNo-prompt Baseline meanRandom-prompt Baseline mean

The trend charts preserve recorded generation-log means; the comparison bars recompute means directly from candidate records or final archives. Baseline lines are reference means, not per-generation runs.

Inspect full metrics and original analysis figure
PopulationnQuality meanQuality SDDiversity meanDiversity SD

SD is the sample standard deviation of recorded scores within each population; it is not a confidence interval across independent search seeds.

Original repository Baseline comparison figure for the selected problem.Baseline source records ↗ · Final archive source ↗

A SECOND VIEW OF DIVERSITY

Behavioural space has its own geometry.

UMAP grid coverage measures occupied prompt-style niches. Semantic entropy measures variation among solution-step paraphrases. They answer different questions.

Within-cluster UMAP spread351 · Neutral

Original analysis CSV: average pairwise UMAP distance per cluster. KMeans was configured with 5 clusters; “5 clusters” is not a discovered performance gain.

Download original diversity CSV ↓

04 / ENGINEERING & EVALUATION

The research behind the interface.

Inspect the implementation decisions and the limits of the recorded experiments.

Source provenance ↗
01

Python research tooling

Grammar-based generation, cached style embeddings, UMAP descriptors, JSONL experiment logging and saved archive snapshots.

Search implementation ↗
02

LLM API orchestration

A shared Requests handler coordinates generator, evaluator, embedding and NLI endpoints. It includes timeouts and retry handling for 429/503 responses.

API handler ↗
03

Multi-objective optimisation

Maximise two objectives within each niche. Non-dominated sorting retains tradeoffs; crowding distance trims a cell to its configured capacity.

Pareto selection code ↗

What these experiments establish — and what remains open.

  • Three problem-specific runs, one per category, with stochastic LLM judgments. The records do not establish generalisation or multi-seed significance.
  • Quality ratings are evaluator-dependent. Semantic entropy depends on variation parsing, NLI responses and the entailment threshold.
  • The configuration requests five variations; saved entropy values can exceed log₂(5). The stored outputs are retained as evidence, not revalidated under a fixed five-sample bound.
  • A discrepancy in problem 358’s final logged quality is disclosed above. Hypervolume is omitted because its saved logs are zero and the referenced analysis utility is absent from this repository.
  • This demo replays existing results. It does not run paid API calls, retrain UMAP or reproduce the experiment end to end.

PYTHON / LLM EVALUATION / ALGORITHM OPTIMISATION

Build experiments
you can actually inspect.

Explore my work in machine learning, evaluation and practical AI systems.

Explore Richard’s portfolio ↗