Using Natural Language to Model Strong Gravitational Lenses

James Nightingale, Richard Hayes

Remember This?

  • You spot a tiny issue with a matplotlib figure.

Remember This?

  • You spot a tiny issue with a matplotlib figure.
  • You change the python code, but it doesn't change the figure.

Remember This?

  • You spot a tiny issue with a matplotlib figure.
  • You change the python code, but it doesn't change the figure.
  • You blindly trial-and-error solutions from StackOverflow for the next 4
    hours.

Remember This?

You wish you could just "Tell matplotlib what you want the image to look
like".

Remember This?

Remember This?

We don't have this problem anymore, because AI allows us to use natural
language to interface with matplotlib.

Remember This?

It was of no benefit to any of us as scientists learning the syntax
required to make matplotlib do what we want.

[Is this a controversial statement? Who Disagrees?]

[Demo 1 - CLI Agents]

Demo 1 (Claude)

Claude -> PyAuto:

The folders @PyAutoGalaxy and @PyAutoLens contain the PyAutoLens Source
code. Find the power-law mass model used by PyAutoLens.

Then, compare its parameterization to the Coolest Lens model standard:
https://github.com/aymgal/COOLEST

Then, on the PyAutoLens GitHub repository, put up an issue comparing the
two parameterizations and how they differ.

No More LaTex!

I've lost many evening trying to make a latex table in a paper "look right".

No More LaTex!

Now I just tell codex the results I want to be in the Latex table, what I
want the Latex table to look like and it pretty much does it.

[Demo 2 - Paper Latex]

Demo 2 (Codex)

codex -> PyAuto:

The folder @z_projects/subhalo_los/output contains lens modeling results
for 3 Slacs lenses.

Make a folder @z_projects/subhalo_los/paper, with a main.tex using the
MNRAS template. Then, make a latex table containing the name and Einstein
radii of each lens, including 3 sigma errors, using the model.results file
in each mass_total[1] folder.

Add a table caption describing its contents.

No More LaTex!

It was of no benefit to any of us as scientists learning the syntax
required to make LaTex do what we want.

[Still reckon this is not too controversial? Although some
might be worried more of the paper writing ends up with the AI?]

[Demo 3 - HPC Use]

Demo 3 (Claude)

I modeled 3 SLACS lenses using scripts/imaging.py, which fitted the last
mass model as a power-law plus shear.

Can you refit mass_total[1], calling it mass_total[1]b, by rerunning the
jobs on the HPC?

However, I've just been told the GPUs we used previously are unavailable,
therefore can you submit each job to run on 8 CPUs, using 8 cores per job?

Inspector and monitor the runs to make sure they start correctly.

No More slurm, batch script management, etc!

It was of no benefit to any of us as scientists learning the syntax
required to make rsync, ssh, slurm do what we want.

[Surely we can all agree on this one?]

Using Natural Language For Software Development

Software Development

PyCharm: Describe how CLI agent can see all source code.

GitHub: Show PyAutoLens issues documenting features.

Pull Requests: Show the PR's with the code updates.

PyAutoPrompt: Show how I just describe code feature using natural
language.

[Demo 4 - Live Feature Development]

Demo 4 (Claude)

The following GitHub URL describes the implementation of a new light
profile, the Hernquist light profile:
https://github.com/PyAutoLabs/PyAutoLens/issues/168 . Implement this in
the PyAutoGalaxy light profiles package.

Demo 4 (Claude)

Now go to PyAutoPrompt/autoarray/oversampling.md

Am I Vibe Coding?

This setup alone is not a long term strategy to maintainable software
development.

We need less vibe coding and more natural language assisted software
development.

Software Testing Is Key

Unit Tests: Show how I have over 2000 human-written unit tests run
against the software ecosystem.

Science Examples: show autolens_workspace examples, with over 1000
scientific examples scripts with bespoke environment variables for fast run
times.

Integration Test Workspaces: End-to-end dedicated test files which
confirm high-level functionality works, show example comparing PSF
convolutions.

PyAutoPulse: The heartbeat of the software; a live agent running all of
the above 24:7, providing near immediate feedback when functionality breaks.

Testing Must Be FAST

In AI-assisted development, test runtime = human waiting time

In my early time using Claude, I spent over a week simply making all unit &
integration tests run much faster (e.g. seconds to minutes).

PyAutoPulse monitors test run time and flags slow down.


Implemented a whole new sampling in PyAutoLens in an hour with minimal
human oversight.

[SHOW PROFILING DEMO]

SLACK Development

I literally just copy and paste SLACK conversations in order to develop
features now.

Using Natural Language To Develop Software

It was of no benefit to any of us as scientists learning the syntax
required to make Python, c, Fortran and others do what we want.

[Maybe I'm getting a little controversial now?]

Giving Natural Language Science Context

Scientific Context

https://ai.plainenglish.io/rag-is-dead-karpathys-llm-wiki-is-the-future-project-explained-2ae6541616cb

What does one of these LLM Wiki's look like?

What does it do?

[Go To PyAutoPaper]

Explain: Indexing, concepts, memory.

Using Natural Language To Read Papers

It was of no benefit to any of us as scientists having to read all
those papers to do science.

Using Natural Language To Read Papers

It was of no benefit to any of us as scientists having to read all
those papers to do science.

[I'm Joking.]

But having AI with this context is extremely powerful.

Using Natural Language To Model Strong Lenses

[Go to autolens_assistant]

Why PyAutoLens Suits an AI assistant

Extensive tutorials and examples offer huge amounts of context.

autolens_assistant converts the over 1000 end-to-end science examples
into a wiki, in the same style as PyAutoPaper.

Why PyAutoLens Suits an AI assistant

PyAutoLens stores the full modelling state, Python objects, results,
metadata, priors, samples and diagnostics, as human-readable .json files.

This makes results easy for both humans and AI agents to inspect, query,
modify and reproduce, without needing to reconstruct the original Python
script that generated them.

Why PyAutoLens Suits an AI assistant

PyAutoLens stores the full modelling state, Python objects, results,
metadata, priors, samples and diagnostics, as human-readable .json files.

This makes results easy for both humans and AI agents to inspect, query,
modify and reproduce, without needing to reconstruct the original Python
script that generated them.

❯ I am writing a paper on lens modeling of euclid data in
@z_projects/euclid/paper , I want to produce the images described by this
text: First, we will consider successful lens models. Figure ? shows an
overview of successful lens model fits to three example lenses. This image
shows only 6 panels to show the fit (RGB, Data, Lens Light Model, Lens
Light subtracted, Lensed Source MOdel Image, Source Reconstruction (Mid
Zoom)) to briefly show the result, note however this reduced image was not
used for the actual visual inspection. I want to give you the name of a
lens, and you use @autolens_assistant to produce this image for the paper,
remember that you need to replot the image by loading the results and
whatnot, including loading and plotting the RGB images which are Euclid
specific. Can you do this for the lens called
Tile102008532RA0683495100072DECNEG0642073858939